SSL/TLS Integration between Java and .NET

Through the years, I've done a lot of integration between Java and .NET platforms. Mostly things play well together, but every once and while an interoperability issue is raised that I haven't encountered before.

Recently, we had a service written in C# on the .NET 4.5 platform that exposed a custom length-based framing protocol over TCP. Essentially requests and responses are prefixed with length-bytes which indicate the subsequent message length, and the same is true for any variable-length fields within the message itself. We needed a Java 7 client library for this service, so I wrote one that exposed a contextual service API backed by a connection pool. The pool would eventually serve-up reusable secure connections with the SSL sessions being reused as well to reduce overhead.

I had built in support for SSL throughout, but had turned it off in my local configuration for most of the development testing. At this stage, all of our concurrency and throughput tests were passing. However, when we flipped the switch to turn on secure connections, any tests that reused a connection were failing. Breaking the problem down to the smallest reproducible scope, we could recreate the issue with 2 sequential requests.

1. Create secure socket to the service and complete handshake
2. Send request A to the service
3. Receive response to A from the service, OK
4. Reuse previous secure socket
5. Send request B to the service
6. Service never responds with a response
7. Timeout occurs

Out comes Wireshark on both sides to see what's going on...

It turns out that on the .NET side, the transport layer is throwing away Request B before the application code ever sees it. But why?

A few peculiar things we noticed in Wireshark that surprised me. First, the protocol had been negotiated to TLS 1.0 and not TLS 1.1 or 1.2 as expected by the Java client. Second, Request B was broken apart into two Application Data records within the packet. That second one seemed more relevant to what appeared to be an interoperability issue, so I began focusing on that.

Here's a test where Request B was simplified to have the value [6A:6B:6C:6D:6E:6F] to see how it gets spread between the two Application Data records:

Padded plaintext before ENCRYPTION: len = 32
0000: 6A 30 7C 99 8E 75 41 3A 88 90 BD 94 B2 17 43 34 j0...uA:......C4
0010: 3E 8A 1A 1F B8 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A >...............
. . .

Padded plaintext before ENCRYPTION: len = 32
0000: 6B 6C 6D 6E 6F 67 F5 62 F9 BA D7 A3 32 0B CC 13 klmnog.b....2...
0010: 25 30 C5 EA AD 10 90 A2 0C 06 06 06 06 06 06 06 %0..............

As you can see, the first byte of the message is consistently placed into the first 32-byte-padded Application Data record, and the rest of the message is put into a second. It turns out that JSSE was modified to behave this way in order to protect against the BEAST attack on SSL3 and TLS 1.0.

Quote from Sun:
By default, we counter chosen plaintext issues 
on CBC mode ciphersuites in SSLv3/TLS1.0 by sending 
one byte of application data in the first record of 
every payload, and the rest in subsequent record(s). 
Note that the issues have been solved in TLS 1.1 or later

Apparently, the .NET transport layer does not play well with this. We updated the .NET application to support TLSv1.2 (TLSv1.1 works as well) and the Java client happily negotiated a connection over 1.2. Now, without the odd record splitting, the tests were passing once again.

While it could have been as simple as having the service application force TLS 1.2 when I noticed 1.0 in the traces, I'm glad to have peeled back the covers to find the cause. It was a fun and interesting troubleshooting session, and solved yet another Java and .NET interoperability gotcha for the future.