useCompression=true and potential Memory leak due to connection tracking via CompressionInputStreamMemoryLeak
Hi there,
We came across an issue on one of our systems towards the end of last week, where the application was growing in memory use. Upon analysing the hprof of a couple of replica systems, it was noticed that there seemed to be a consistency across the hprof's with that of a growing number of JDBC4Connection objects in memory. For a proportion of those JDBC4Connections, the "forceClosedReason" was filled out; and as a result the JDBC4Connection shouldn't really be strong referenced anymore.
The only class that seemed to have strong reference to the JDBC4Connection was that of the static map "connectionPhantomRefs" in NonRegisteringDriver and the associated ConnectionPhantomReference.
The date the issue of increasing memory usage (akin to a Memory Leak) also co-incided with our upgrade to 5.1.23 of Connector/J.
----
I've spend some time trying to understand where the issue is occurring, and I beleive I have managed to replicate the issue in a sample piece of code; removing the application code from the equation.
From what seems to be occurring is that when "useCompression=true" is enabled, the NonRegisteringDriver.trackConnection(this) is holding onto the Connection; even though the application no longer has a strong reference to the Connection. With "useCompression=false", the Connections are not held onto and the Leak does not occur.
From what it looks like is that the CompressionInputStream is keeping the JDBC4Connection object alive, for which the ConnectionPhantomReference is monitoring for pre-gc clean up. The CompressionInputSteam itself has a reference to the JDBC4Connection, and the ConnectionPhantomReference has a strong reference to the CompressionInputStream via the NetworkResource object:
ConnectionPhantomReference ---> NetworkResources
NetworkResources ---> io (MysqlIO) ----> CompressedInputStream
CompressedInputStream ---> JDBC4Connection
I think this might be solvable via the use of a WeakReference in CompressedInputStream, but I don't know if this use of a WeakReference may cause regression elsewhere.
I've attempted to document the investigation into the issue here:
https://github.com/tootedom/CompressionInputStreamMemoryLeak
With the sample code to replicate it. I hope the write up/investigation makes sense, and that I've not over looked something, but apologies if I have indeed gone down the wrong track on this.
Is anyone else witnessing this slow growth in the number of connections in the heap? Or is this issue already known about?
Thanks in advance for any assistance. Please let me know if I've mis-represented anything, or if anything requires clarifying.
cheers
/dom