Re: MySQL Cluster data nodes fails with 7200: LCP fragment scan watchdog
Posted by:
Puneet N
Date: January 07, 2018 02:44AM
Latest log of failure again
Data node 1:
2018-01-07 00:01:17 [ndbd] INFO -- part: 0 : time to complete: 2 - Repeated 2 times
LCP Frag watchdog : No progress on table 37, frag 0 for 49 s. 238512 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 29 s. 277280 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 39 s. 277280 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 49 s. 277280 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 59 s. 277280 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 69 s. 277280 rows completed
LCP Frag watchdog : Checkpoint of table 37 fragment 0 too slow (no progress for > 69 s).
m_curr_disk_write_speed: 2048kb m_words_written_this_period: 0kwords m_overflow_disk_write: 0kb
m_reset_delay_used: 101 time since last RESET_DISK_SPEED: 103 millis
m_monitor_words_written : 0, duration : 297 millis, rate : 0 bytes/s : (0 pct of config)
BackupRecord 0: BackupId: 247 MasterRef: f70002 ClientRef: 0
State: 4
noOfByte: 643876 noOfRecords: 4570
noOfLogBytes: 0 noOfLogRecords: 0
errorCode: 0
file 0: type: 3 flags: H'19 tableId: 37 fragmentId: 0
ready: TRUE eof: FALSE
2018-01-07 00:02:33 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100
2018-01-07 00:02:33 [ndbd] INFO -- Watchdog: User time: 349135 System time: 48402
2018-01-07 00:02:33 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=200
2018-01-07 00:02:33 [ndbd] INFO -- Watchdog: User time: 349135 System time: 48402
-----------------------------
Data node 2:
LCP Frag watchdog : No progress on table 37, frag 0 for 29 s. 3092304 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 20 s. 3459488 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 29 s. 3459488 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 39 s. 3459488 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 49 s. 3459488 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 59 s. 3459488 rows completed
LCP Frag watchdog : No progress on table 37, frag 0 for 69 s. 3459488 rows completed
LCP Frag watchdog : Checkpoint of table 37 fragment 0 too slow (no progress for > 69 s).
m_curr_disk_write_speed: 5120kb m_words_written_this_period: 0kwords m_overflow_disk_write: 0kb
m_reset_delay_used: 96 time since last RESET_DISK_SPEED: 20 millis
m_monitor_words_written : 0, duration : 720 millis, rate : 0 bytes/s : (0 pct of config)
BackupRecord 0: BackupId: 247 MasterRef: f70003 ClientRef: 0
State: 4
noOfByte: 643876 noOfRecords: 4570
noOfLogBytes: 0 noOfLogRecords: 0
errorCode: 0
file 0: type: 3 flags: H'19 tableId: 37 fragmentId: 0
ready: TRUE eof: FALSE
2018-01-07 00:04:32 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=100
2018-01-07 00:04:32 [ndbd] INFO -- Watchdog: User time: 286549 System time: 24249
2018-01-07 00:04:32 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=200
2018-01-07 00:04:32 [ndbd] INFO -- Watchdog: User time: 286549 System time: 24249
---------------