A colleague had created a new service operation in peoplesoft integration broker, but it wasn’t working. After a lot of investigation, and a call to Oracle, we found the cause was SNI: Service Name Indication. This is a facility which allows several different hostnames to be on the same IP address, but still listen in HTTPS.
It seems the Integration Broker converts the hostname to an IP address before trying to contact the remote site. The problem is then that the remote site doesn’t know which website it wishes to talk to, and this causes a handshake failure. The messages in the logs are not helpful to say the least.