Heterogeneous Latencies with Ndb API

[Bruno Lussan]

Hi,
I developed an application server that uses the C++ ndb API. I did a test with a transaction that inserts data into two tables. The records are a few tens of bytes, everything works fine except that if I do 1000 times the same transaction in a single thread, I have very heterogeneous latencies of 3ms to 732ms, the average is 27ms per transaction. Last year on a previous version of MySQL Cluster I had done a similar test and I did not notice such differences. What do you think ? Is it normal behavior or not? I note that at the two data-nodes, very few resources are used (CPU, memory, disk and network). how do i know if these latencies come from the ndb api or cluster side?

Latency is measured from the response time of the NdbTransaction::execute(COMMIT) method.

Best regards
Bruno

The set_max_adaptive_send_time() is set to 2ms.

MySQL Cluster 7.6.8 config.ini file :
[NDB_MGMD DEFAULT]
ArbitrationRank=1
HeartbeatIntervalMgmdMgmd=1500
Portnumber=1186

[NDB_MGMD]
NodeId=49
HostName=192.168.1.70
DataDir=/home/mysql/cluster/49/
Portnumber=1186

[NDB_MGMD]
NodeId=56
HostName=192.168.1.71
DataDir=/home/mysql/cluster/56/
Portnumber=1186

[TCP DEFAULT]
SendBufferMemory=8M
ReceiveBufferMemory=8M

[NDBD DEFAULT]
NoOfReplicas=2
LateAlloc=1
StartNoNodeGroupTimeout=0
MaxBufferedEpochs=100
MaxBufferedEpochBytes=26214400
MaxDiskWriteSpeed=40M
MinDiskWriteSpeed=10M
MaxDiskWriteSpeedOtherNodeRestart=100M
MaxDiskWriteSpeedOwnRestart=200M
BackupMaxWriteSize=1M
BackupWriteSize=256K
BackupDataBufferSize=1M
BackupLogBufferSize=16M
BackupReportFrequency=10
BackupDiskWriteSpeedPct=50
CompressedBackup=true
CompressedLCP=false
MemReportFrequency=0
LogLevelStartup=1
LogLevelShutdown=0
LogLevelStatistic=0
LogLevelCheckpoint=0
LogLevelNodeRestart=0
LogLevelConnection=0
LogLevelError=0
LogLevelCongestion=0
LogLevelInfo=0
EventLogBufferSize=8K
StartupStatusReportFrequency=0
DataMemory=2G
MinFreePct=5
MaxNoOfAttributes=1000
MaxNoOfTables=128
MaxNoOfOrderedIndexes=128
MaxNoOfTriggers=768
MaxNoOfSubscriptions=0
MaxNoOfSubscribers=0
MaxNoOfConcurrentSubOperations=256
StringMemory=25
MaxAllocate=32M
MaxNoOfSavedMessages=25
LcpScanProgressTimeout=60
Diskless=false
LockPagesInMainMemory=0
Arbitration=Default
DiskPageBufferEntries=10
DiskPageBufferMemory=128M
SharedGlobalMemory=384M
DiskIOThreadPool=2
DiskSyncSize=4M
LongMessageBuffer=64M
MaxParallelCopyInstances=0
MaxNoOfConcurrentTransactions=4096
MaxNoOfConcurrentOperations=131072
MaxDMLOperationsPerTransaction=4294967295
MaxNoOfConcurrentScans=256
MaxParallelScansPerFragment=256
BatchSizePerLocalScan=256
FragmentLogFileSize=256M
NoOfFragmentLogFiles=15
InitFragmentLogFiles=SPARSE
RedoBuffer=64M
RedoOverCommitCounter=3
RedoOverCommitLimit=20
TwoPassInitialNodeRestartCopy=false
BuildIndexThreads=0
MaxNoOfExecutionThreads=4
RealtimeScheduler=false
SchedulerSpinTimer=0
TimeBetweenLocalCheckpoints=20
TimeBetweenGlobalCheckpoints=2000
TimeBetweenGlobalCheckpointsTimeout=120000
TimeBetweenEpochs=100
TimeBetweenEpochsTimeout=32000
TimeBetweenWatchdogCheck=6000
TimeBetweenWatchdogCheckInitial=60000
TransactionInactiveTimeout=4294967039
TransactionDeadlockDetectionTimeout=1200
TimeBetweenInactiveTransactionAbortCheck=1000
HeartbeatIntervalDbDb=1500
HeartbeatIntervalDbApi=60000
ConnectCheckIntervalDelay=0
StartPartialTimeout=30000
StartPartitionedTimeout=4294967039
StartFailureTimeout=0
ArbitrationTimeout=7500
RestartSubscriberConnectTimeout=12000
StopOnError=0
MaxStartFailRetries=3

[NDBD]
NodeId=1
HostName=192.168.1.70
DataDir=/home/mysql/cluster/1/

[NDBD]
NodeId=2
HostName=192.168.1.71
DataDir=/home/mysql/cluster/2/

[MYSQLD DEFAULT]
AutoReconnect=true
BatchByteSize=16K
BatchSize=256
MaxScanBatchSize=256K

[MYSQLD]
NodeId=52
HostName=192.168.1.70

[MYSQLD]
NodeId=55
HostName=192.168.1.71

[API]
NodeId=50

[API]
NodeId=51

[API]
NodeId=53

[API]
NodeId=54



Edited 1 time(s). Last edit at 06/06/2019 11:33AM by Bruno Lussan.