Hi Thomas,
I'm not quite sure, I do understand your question. Are you trying to get X.509 based SSO working without SSL or the connection to the ABAP backend working without SSL.
For the first, this won't work, as X.509 certificates are bound to the SSL protocol. So you would need to use a different auth scheme in this case.
For the second, as far as I do understand you setup, this should be RFC based communication anyway and therefor would require the use of SNC not SSL.
Regards,
Patrick