Hanging threads on com.mysql.jdbc.ConnectionImpl.setAutoCommit
Posted by: Daniel Baskakov
Date: May 08, 2018 12:49AM

We have a server system where the following DB layer is used: hibernate 4.3.7, c3p0 0.9.5.2, mysql-connector-java 5.1.46.
MySQL 5.7.21 (and also 5.5.60) is a standalone DB server.
After ~2 days of the system uptime some threads start to hang on the following point:

"Synchronizer: (??????? ?????????? / ??????????)" daemon prio=4 tid=0x00000010ecef9800 nid=0x8148 runnable [0x000000109f97e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
- locked <0x000000093429eaa0> (a com.mysql.jdbc.util.ReadAheadInputStream)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4155)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2832)
- locked <0x0000000c8ab1dbf8> (a com.mysql.jdbc.JDBC4Connection)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5357)
- locked <0x0000000c8ab1dbf8> (a com.mysql.jdbc.JDBC4Connection)
at com.mchange.v2.c3p0.impl.NewProxyConnection.setAutoCommit(NewProxyConnection.java:1085)
- locked <0x000000093509a148> (a com.mchange.v2.c3p0.impl.NewProxyConnection)
at org.hibernate.c3p0.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:94)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:380)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:228)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:171)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:162)
at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1435)
...

Same threads usually operates normally, but after 2 days the number of hanging threads slowly start to increase.
The system became more and more unresponsive and we have to restart it.
Hanging threads are not hanging indeed, they are reading sockets and waiting for the response from the MySQL DB engine.
But reply never happens and we've came to errors like: 'The last packet successfully received from the server was ..... milliseconds ago. Longer than 'wait_timeout' ...'
At the same time other similar ('Synchronizer') threads work perfect.

When the hang happens SHOW FULL PROCESSLIST displays some long-sleeping processes:

Id User Host db Command Time State Info
266707 root localhost:57373 aggregate Sleep 295 NULL <-- like this
266953 root localhost:58471 aggregate Sleep 177 NULL
268071 root localhost:64882 aggregate Sleep 1 NULL
268178 root localhost:65415 aggregate Sleep 1 NULL
268233 root localhost:49234 aggregate Sleep 1 NULL
268236 root localhost:49237 aggregate Sleep 1 NULL
...
268369 root localhost:49919 NULL Query 0 NULL show full processlist

Does anyone has some thoughts how JDBC threads can come to the orphan state hanging on socket read?

Options: ReplyQuote




Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.