Error 1296 :Got error 4350 'Transaction already aborted' on INSERT ... ON DUPLICATE KEY UPDATE

[Fabio Zecchini]

Hi everybody,
we have developed a C multithread application that works in real-time with high volumes of data of about 1.000 calls/second: every thread has its own mysql db connection with NDB engine.

Over these tables we use query like "INSERT ... ON DUPLICATE KEY UPDATE" with transaction check.

All clients API and server node use the following version: 5.5.22-ndb-7.2.6 Source distribution.

Sometimes, during our stress test we get this error after an insert query:

Error 1296 :Got error 4350 'Transaction already aborted' from NDBCLUSTER.

Although the error seems to be a double rollback o something similar, we are sure this is the first error from cluster without a previous rollback operation.

The problems has been encountered with more than 30 thread and with query insert between 600 and 1400 per second.

Replacing "INSERT ... ON DUPLICATE KEY UPDATE" with a normal INSERT the issue and the error disappears.

Following one of the table definition that report this error:
DATA | DATETIME | NOT NULL |
USER | VARCHAR 255 | CHARACTER SET latin1, COLLATE latin1_general_ci | NOT NULL |
APP | VARCHAR 255 | CHARACTER SET latin1, COLLATE latin1_general_ci | NOT NULL |
CAMPAIGN | VARCHAR 255 | CHARACTER SET latin1, COLLATE latin1_general_ci | NOT NULL |
ADV_CLICK | VARCHAR 255 | CHARACTER SET latin1, COLLATE latin1_general_ci | NOT NULL |
OFFLINE | INT | NOT NULL |
COUNT | INT | NOT NULL |

PRIMARY KEY ( DATE, USER, APP, CAMPAIGN, ADV_CLICK, OFFLINE)

Sample query that generate the error:
INSERT INTO table1 VALUES('2012-06-11 08:00:00','VOLUME','VOLUME','VOLUME','VOLUME','0','1') ON DUPLICATE KEY UPDATE COUNT=COUNT+'1'

Hereby the server configuration setting:

[ndb_mgmd]
hostname=appha
datadir=/other/NDB
id=1

[ndbd default]
noofreplicas=2
datadir=/other/NDB
DataMemory=1950M
IndexMemory=450M

# Transaction Parameters #
MaxNoOfConcurrentTransactions: 4096
MaxNoOfConcurrentOperations: 100000
MaxNoOfLocalOperations: 100000

# Transaction Temporary Storage #
MaxNoOfConcurrentIndexOperations: 8192
MaxNoOfFiredTriggers: 4000
TransactionBufferMemory: 1M

# Scans and buffering #
MaxNoOfConcurrentScans: 300
MaxNoOfLocalScans: 32
BatchSizePerLocalScan: 64
LongMessageBuffer: 1M

# Logging and Checkpointing #
NoOfFragmentLogFiles: 300
FragmentLogFileSize: 16M
MaxNoOfOpenFiles: 40
InitialNoOfOpenFiles: 27
MaxNoOfSavedMessages: 25

# Metadata Objects #
MaxNoOfAttributes: 1500
MaxNoOfTables: 400
MaxNoOfOrderedIndexes: 200
MaxNoOfUniqueHashIndexes: 200
MaxNoOfTriggers: 770


[ndbd]
hostname=appclnode1
id=4

[ndbd]
hostname=appclnode2
id=5

[mysqld]
HostName=appel1
id=50

[mysqld]
HostName=appel2
id=51

[mysqld]
HostName=appel3
id=52

[mysqld]
HostName=appel4
id=53

[mysqld]
HostName=appel5
id=55

[mysqld]
HostName=appview1
id=54


Can anyone help us to detect and solve this error?
Is it a matter of query, setting or what?

Thanks in advance for every help you can provide us.

Kind regards,
Fabio