ndb crash
Posted by: 陈 jim
Date: April 23, 2021 12:19AM
Date: April 23, 2021 12:19AM
ndb_11_out.log
stack_bottom = 0 thread_stack 0x0
/usr/local/bin/ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x89128e]
/usr/local/bin/ndbd(ndb_print_stacktrace()+0x45) [0x849245]
/usr/local/bin/ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x1f) [0x7fd56f]
/usr/local/bin/ndbd() [0x85e7e1]
/usr/local/bin/ndbd(SimulatedBlock::sendSignal(unsigned int, unsigned short, Signal*, unsigned int, JobBufferLevel) const+0x23d) [0x86646d]
/usr/local/bin/ndbd(Dbtup::sendReadAttrinfo(Signal*, Dbtup::KeyReqStruct*, unsigned int)+0x2d0) [0x7dee10]
/usr/local/bin/ndbd(Dbtup::interpreterStartLab(Signal*, Dbtup::KeyReqStruct*)+0x286) [0x6ca5f6]
/usr/local/bin/ndbd(Dbtup::execTUPKEYREQ(Signal*, void*, void*)+0x336) [0x6cf1a6]
/usr/local/bin/ndbd(Dblqh::next_scanconf_tupkeyreq(Signal*, Dblqh::ScanRecord*, Dblqh::TcConnectionrec*, Dblqh::Fragrecord*, unsigned int)+0x36) [0x6475f6]
/usr/local/bin/ndbd(Dblqh::continue_next_scan_conf(Signal*, Dblqh::ScanRecord::ScanState, Dblqh::ScanRecord*)+0x6b) [0x6483db]
/usr/local/bin/ndbd(Dbtux::continue_scan(Signal*, Ptr<Dbtux::ScanOp>, Dbtux::Frag&, bool)+0x940) [0x70fe50]
/usr/local/bin/ndbd(Dbtux::execNEXT_SCANREQ(Signal*)+0x91) [0x7138e1]
/usr/local/bin/ndbd(Dblqh::send_next_NEXT_SCANREQ(Signal*, SimulatedBlock*, void (SimulatedBlock::*)(Signal*), Dblqh::ScanRecord*, unsigned int)+0xfd) [0x60a6ed]
/usr/local/bin/ndbd(Dblqh::scanNextLoopLab(Signal*, unsigned int, unsigned int, Dblqh::ScanRecord*, Dblqh::Fragrecord*)+0x109) [0x60aac9]
/usr/local/bin/ndbd(Dblqh::continue_next_scan_conf(Signal*, Dblqh::ScanRecord::ScanState, Dblqh::ScanRecord*)+0x6b) [0x6483db]
/usr/local/bin/ndbd(Dbtux::continue_scan(Signal*, Ptr<Dbtux::ScanOp>, Dbtux::Frag&, bool)+0x940) [0x70fe50]
/usr/local/bin/ndbd(Dbtux::execNEXT_SCANREQ(Signal*)+0x91) [0x7138e1]
/usr/local/bin/ndbd(Dblqh::execACC_CHECK_SCAN(Signal*)+0xc8) [0x6076c8]
/usr/local/bin/ndbd(FastScheduler::doJob(unsigned int)+0x1da) [0x85ef1a]
/usr/local/bin/ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6b2) [0x86de12]
/usr/local/bin/ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x758) [0x4fb0e8]
/usr/local/bin/ndbd(real_main(int, char**)+0x2e5) [0x4f9225]
/usr/local/bin/ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xfeb) [0x4f8dab]
/usr/local/bin/ndbd(real_main(int, char**)+0x34c) [0x4f928c]
/usr/local/bin/ndbd(main+0x37) [0x4f5407]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f2709f8e7b3]
/usr/local/bin/ndbd() [0x4f6ae1]
2021-04-23 11:23:17 [ndbd] INFO -- Job Buffer Full
2021-04-23 11:23:17 [ndbd] INFO -- APZJobBuffer.C
2021-04-23 11:23:17 [ndbd] INFO -- Error handler restarting system
2021-04-23 11:23:17 [ndbd] ALERT -- Node 11: Forced node shutdown completed. Caused by error 2334: 'Job buffer congestion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2021-04-23 11:23:17 [ndbd] INFO -- Ndb has terminated (pid 4152230) restarting
2021-04-23 11:23:17 [ndbd] INFO -- Angel reconnected to '172.16.9.21:1186'
2021-04-23 11:23:28 [ndbd] INFO -- Angel reallocated nodeid: 11
2021-04-23 11:23:28 [ndbd] INFO -- Angel pid: 4152229 started child: 44571
2021-04-23 11:23:28 [ndbd] INFO -- Normal start of data node using checkpoint and log info if existing
2021-04-23 11:23:28 [ndbd] INFO -- Configuration fetched from '172.16.9.21:1186', generation: 13
2021-04-23 11:23:28 [ndbd] INFO -- Changing directory to '/usr/local/mysql/data'
2021-04-23 11:23:28 [ndbd] INFO -- SchedulerSpinTimer = 0
2021-04-23 11:23:28 [ndbd] INFO -- AutomaticThreadConfig = 0, NumCPUs = 0
2021-04-23 11:23:28 [ndbd] INFO -- MaxNoOfTriggers set to 16648
NDBMT: non-mt
2021-04-23 11:23:28 [ndbd] INFO -- NDB Cluster -- DB node 11
2021-04-23 11:23:28 [ndbd] INFO -- mysql-8.0.23 ndb-8.0.23 --
2021-04-23 11:23:28 [ndbd] INFO -- Memory Allocation for global memory pools Starting
2021-04-23 11:23:28 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2021-04-23 11:23:28 [ndbd] INFO -- SharedGlobalMemory set to 2048 MB
2021-04-23 11:23:28 [ndbd] INFO -- DataMemory set to 12000 MB
2021-04-23 11:23:28 [ndbd] INFO -- RedoLogBuffer uses 132 MB
2021-04-23 11:23:28 [ndbd] INFO -- DiskPageBuffer uses 70 MB
2021-04-23 11:23:28 [ndbd] INFO -- SchemaTransactionMemory uses 2 MB
2021-04-23 11:23:28 [ndbd] INFO -- TransactionMemory calculated to 1937 MB
2021-04-23 11:23:28 [ndbd] INFO -- Reserved part of TransactionMemory is 363 MB
2021-04-23 11:23:28 [ndbd] INFO -- No Undo log buffer used, will be allocated from TransactionMemory if later defined by command
2021-04-23 11:23:28 [ndbd] INFO -- TransactionMemory can expand and use SharedGlobalMemory if required
2021-04-23 11:23:28 [ndbd] INFO -- QueryMemory can use memory from SharedGlobalMemory until 90% used
2021-04-23 11:23:28 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 14141Mb initial: 16189Mb
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Starting, 4224 pages, page size = 32768
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:28 [ndbd] INFO -- Adding 132Mb to ZONE_19 (1, 4223)
2021-04-23 11:23:28 [ndbd] INFO -- Memory Allocation for global memory pools Completed
Started thread, index = 0, id = 44572, type = NdbfsThread
2021-04-23 11:23:28 [ndbd] INFO -- Loading blocks for data node run-time environment
2021-04-23 11:23:28 [ndbd] INFO -- Starting Sending and Receiving services
Started thread, index = 1, id = 44582, type = SocketClientThread
Started thread, index = 2, id = 44573, type = WatchDogThread
Started thread, index = 3, id = 44583, type = SocketServerThread
2021-04-23 11:23:28 [ndbd] INFO -- Starting the data node run-time environment
2021-04-23 11:23:28 [ndbd] INFO -- First START_ORD executed to connect MGM servers
2021-04-23 11:23:28 [ndbd] INFO -- Received second START_ORD as part of normal start
2021-04-23 11:23:28 [ndbd] INFO -- Disconnect all non-MGM servers
2021-04-23 11:23:28 [ndbd] INFO -- Start excuting the start phases
2021-04-23 11:23:28 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 0, name = CMVMI
2021-04-23 11:23:28 [ndbd] INFO -- Start initiated (mysql-8.0.23 ndb-8.0.23)
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 5, id = 44584, type = NdbfsThread
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Starting, 513834 pages, page size = 32768
2021-04-23 11:23:29 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:29 [ndbd] INFO -- Adding 8060Mb to ZONE_19 (4224, 257919)
2021-04-23 11:23:29 [ndbd] INFO -- Adding 7998Mb to ZONE_19 (262145, 255913)
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 1, name = NDBFS
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 6, id = 44600, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 7, id = 44601, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 8, id = 44602, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 9, id = 44603, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 10, id = 44604, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 11, id = 44605, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 12, id = 44606, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 13, id = 44607, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 14, id = 44608, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 15, id = 44609, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 16, id = 44610, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 17, id = 44611, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 18, id = 44612, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 19, id = 44613, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 20, id = 44614, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 21, id = 44615, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 22, id = 44616, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 23, id = 44617, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 24, id = 44618, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 25, id = 44619, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 26, id = 44620, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 27, id = 44621, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 28, id = 44622, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 29, id = 44623, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 30, id = 44624, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 31, id = 44625, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 32, id = 44626, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 33, id = 44627, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 34, id = 44628, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 35, id = 44629, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 36, id = 44630, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 37, id = 44631, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 38, id = 44632, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 39, id = 44633, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 40, id = 44634, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 41, id = 44635, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 42, id = 44636, type = NdbfsThread
Started thread, index = 43, id = 44637, type = NdbfsThread
Started thread, index = 44, id = 44638, type = NdbfsThread
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 2, name = DBINFO
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 3, name = DBTUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 4, name = DBACC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 5, name = DBTC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 6, name = DBLQH
LCPFragWatchdog : WarnElapsed : 60000(ms) MaxElapsed 180000(ms) : period millis : 1000
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 7, name = DBTUX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 8, name = DBDICT
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 9, name = DBDIH
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 10, name = NDBCNTR
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 11, name = QMGR
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 12, name = TRIX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 13, name = BACKUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 14, name = DBUTIL
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 15, name = SUMA
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 16, name = TSMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 17, name = LGMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 18, name = PGMAN
pgman: page buffer entries = 10
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 19, name = RESTORE
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 20, name = DBSPJ
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 21, name = THRMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 22, name = DBQLQH
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 23, name = DBQACC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 24, name = DBQTUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 25, name = QBACKUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 26, name = DBQTUX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 27, name = QRESTORE
2021-04-23 11:23:29 [ndbd] INFO -- READ_CONFIG_REQ phase completed, this phase is used to read configuration and to calculate various sizes and allocate almost all memory needed by the data node in its lifetime
2021-04-23 11:23:29 [ndbd] INFO -- Not initial start
2021-04-23 11:23:29 [ndbd] INFO -- Local sysfile: Node restorable on its own, gci: 0, version: 70603
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 0 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 0 has made some file system initialisations
2021-04-23 11:23:29 [ndbd] INFO -- We are running with 0 LDM workers and 4 REDO log parts. This means that we can avoid using a mutex to access REDO log parts
2021-04-23 11:23:29 [ndbd] INFO -- Watchdog KillSwitch off.
2021-04-23 11:23:29 [ndbd] INFO -- Starting QMGR phase 1
2021-04-23 11:23:29 [ndbd] INFO -- Starting with m_restart_seq set to 12
2021-04-23 11:23:29 [ndbd] INFO -- DIH reported normal start, now starting the Node Inclusion Protocol
2021-04-23 11:23:29 [ndbd] INFO -- findNeighbours from: 2988 old (left: 65535 right: 65535) new (10 10)
2021-04-23 11:23:29 [ndbd] INFO -- Include node protocol completed, phase 1 in QMGR completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 1 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 1 initialised some variables and included node in cluster, locked memory if configured to do so
2021-04-23 11:23:29 [ndbd] INFO -- Starting with m_restart_seq set to 12
2021-04-23 11:23:29 [ndbd] INFO -- Asking master node to accept our start (nodeId = 10 is master), GCI = 957711
2021-04-23 11:23:29 [ndbd] INFO -- NDBCNTR master accepted us into cluster, start NDB start phase 1
2021-04-23 11:23:29 [ndbd] INFO -- We are performing a node restart
2021-04-23 11:23:29 [ndbd] INFO -- LDM(0): Started LDM restart phase 1 (read REDO log page headers to init REDO log data)
2021-04-23 11:23:29 [ndbd] INFO -- Schema file initialisation Starting
2021-04-23 11:23:29 [ndbd] INFO -- Schema file initialisation Completed
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 1 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 2 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 2 did more initialisations, master accepted our start, we started REDO log initialisations
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 2
2021-04-23 11:23:29 [ndbd] INFO -- Request permission to start our node from master Starting
2021-04-23 11:23:29 [ndbd] INFO -- Request permission to start our node from master Completed
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 2 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 3 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 3 locked the data dictionary, performed local connection setups, we asked for permission to start our node
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 3
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 3 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 4 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 4 continued preparations of the REDO log
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 4
2021-04-23 11:23:29 [ndbd] INFO -- Request copying of distribution and dictionary information from master Starting
2021-04-23 11:23:29 [ndbd] INFO -- Completed setting up multiple transporters to nodes in the same node group
2021-04-23 11:23:29 [ndbd] INFO -- LDM(0):Ready to start execute REDO log phase, prepare REDO log phase completed
2021-04-23 11:23:30 [ndbd] INFO -- Receive arbitrator node 1 [ticket=f05b0002b3b91fc5]
2021-04-23 11:23:48 [ndbd] INFO -- findNeighbours from: 2905 old (left: 10 right: 10) new (10 12)
2021-04-23 11:23:48 [ndbd] INFO -- Node 10 disconnected in state: 0
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
/usr/local/bin/ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x89128e]
/usr/local/bin/ndbd(ndb_print_stacktrace()+0x45) [0x849245]
/usr/local/bin/ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x1f) [0x7fd56f]
/usr/local/bin/ndbd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf2) [0x8629e2]
/usr/local/bin/ndbd(Qmgr::execDISCONNECT_REP(Signal*)+0x1ec) [0x77bd1c]
/usr/local/bin/ndbd(FastScheduler::doJob(unsigned int)+0x1da) [0x85ef1a]
/usr/local/bin/ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6b2) [0x86de12]
/usr/local/bin/ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x758) [0x4fb0e8]
/usr/local/bin/ndbd(real_main(int, char**)+0x2e5) [0x4f9225]
/usr/local/bin/ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xfeb) [0x4f8dab]
/usr/local/bin/ndbd(real_main(int, char**)+0x34c) [0x4f928c]
/usr/local/bin/ndbd(main+0x37) [0x4f5407]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f2709f8e7b3]
/usr/local/bin/ndbd() [0x4f6ae1]
2021-04-23 11:23:48 [ndbd] INFO -- Node 10 disconnected in phase: 3
2021-04-23 11:23:48 [ndbd] INFO -- QMGR (Line: 4245) 0x00000000
2021-04-23 11:23:48 [ndbd] INFO -- Error handler restarting system
2021-04-23 11:23:48 [ndbd] INFO -- Error handler shutdown completed - exiting
2021-04-23 11:23:48 [ndbd] INFO -- Angel detected startup failure, count: 1
2021-04-23 11:23:48 [ndbd] ALERT -- Node 11: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2021-04-23 11:23:48 [ndbd] INFO -- Ndb has terminated (pid 44571) restarting
2021-04-23 11:23:48 [ndbd] INFO -- Angel reconnected to '172.16.9.21:1186'
2021-04-23 11:23:51 [ndbd] INFO -- Angel reallocated nodeid: 11
2021-04-23 11:23:51 [ndbd] INFO -- Angel pid: 4152229 started child: 44736
2021-04-23 11:23:51 [ndbd] INFO -- Normal start of data node using checkpoint and log info if existing
2021-04-23 11:23:51 [ndbd] INFO -- Configuration fetched from '172.16.9.21:1186', generation: 13
2021-04-23 11:23:51 [ndbd] INFO -- Changing directory to '/usr/local/mysql/data'
2021-04-23 11:23:51 [ndbd] INFO -- SchedulerSpinTimer = 0
2021-04-23 11:23:51 [ndbd] INFO -- AutomaticThreadConfig = 0, NumCPUs = 0
2021-04-23 11:23:51 [ndbd] INFO -- MaxNoOfTriggers set to 16648
NDBMT: non-mt
2021-04-23 11:23:51 [ndbd] INFO -- NDB Cluster -- DB node 11
2021-04-23 11:23:51 [ndbd] INFO -- mysql-8.0.23 ndb-8.0.23 --
2021-04-23 11:23:51 [ndbd] INFO -- Memory Allocation for global memory pools Starting
2021-04-23 11:23:51 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2021-04-23 11:23:51 [ndbd] INFO -- SharedGlobalMemory set to 2048 MB
2021-04-23 11:23:51 [ndbd] INFO -- DataMemory set to 12000 MB
2021-04-23 11:23:51 [ndbd] INFO -- RedoLogBuffer uses 132 MB
2021-04-23 11:23:51 [ndbd] INFO -- DiskPageBuffer uses 70 MB
2021-04-23 11:23:51 [ndbd] INFO -- SchemaTransactionMemory uses 2 MB
2021-04-23 11:23:51 [ndbd] INFO -- TransactionMemory calculated to 1937 MB
2021-04-23 11:23:51 [ndbd] INFO -- Reserved part of TransactionMemory is 363 MB
2021-04-23 11:23:51 [ndbd] INFO -- No Undo log buffer used, will be allocated from TransactionMemory if later defined by command
2021-04-23 11:23:51 [ndbd] INFO -- TransactionMemory can expand and use SharedGlobalMemory if required
2021-04-23 11:23:51 [ndbd] INFO -- QueryMemory can use memory from SharedGlobalMemory until 90% used
2021-04-23 11:23:51 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 14141Mb initial: 16189Mb
2021-04-23 11:23:51 [ndbd] INFO -- Touch Memory Starting, 4224 pages, page size = 32768
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:52 [ndbd] INFO -- Adding 132Mb to ZONE_19 (1, 4223)
2021-04-23 11:23:52 [ndbd] INFO -- Memory Allocation for global memory pools Completed
Started thread, index = 0, id = 44737, type = NdbfsThread
2021-04-23 11:23:52 [ndbd] INFO -- Loading blocks for data node run-time environment
2021-04-23 11:23:52 [ndbd] INFO -- Starting Sending and Receiving services
Started thread, index = 1, id = 44747, type = SocketClientThread
Started thread, index = 2, id = 44738, type = WatchDogThread
Started thread, index = 3, id = 44748, type = SocketServerThread
2021-04-23 11:23:52 [ndbd] INFO -- Starting the data node run-time environment
2021-04-23 11:23:52 [ndbd] INFO -- First START_ORD executed to connect MGM servers
2021-04-23 11:23:52 [ndbd] INFO -- Received second START_ORD as part of normal start
2021-04-23 11:23:52 [ndbd] INFO -- Disconnect all non-MGM servers
2021-04-23 11:23:52 [ndbd] INFO -- Start excuting the start phases
2021-04-23 11:23:52 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 0, name = CMVMI
2021-04-23 11:23:52 [ndbd] INFO -- Start initiated (mysql-8.0.23 ndb-8.0.23)
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 5, id = 44749, type = NdbfsThread
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Starting, 513834 pages, page size = 32768
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:52 [ndbd] INFO -- Adding 8060Mb to ZONE_19 (4224, 257919)
2021-04-23 11:23:52 [ndbd] INFO -- Adding 7998Mb to ZONE_19 (262145, 255913)
2021-04-23 11:23:52 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 1, name = NDBFS
stack_bottom = 0 thread_stack 0x0
/usr/local/bin/ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x89128e]
/usr/local/bin/ndbd(ndb_print_stacktrace()+0x45) [0x849245]
/usr/local/bin/ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x1f) [0x7fd56f]
/usr/local/bin/ndbd() [0x85e7e1]
/usr/local/bin/ndbd(SimulatedBlock::sendSignal(unsigned int, unsigned short, Signal*, unsigned int, JobBufferLevel) const+0x23d) [0x86646d]
/usr/local/bin/ndbd(Dbtup::sendReadAttrinfo(Signal*, Dbtup::KeyReqStruct*, unsigned int)+0x2d0) [0x7dee10]
/usr/local/bin/ndbd(Dbtup::interpreterStartLab(Signal*, Dbtup::KeyReqStruct*)+0x286) [0x6ca5f6]
/usr/local/bin/ndbd(Dbtup::execTUPKEYREQ(Signal*, void*, void*)+0x336) [0x6cf1a6]
/usr/local/bin/ndbd(Dblqh::next_scanconf_tupkeyreq(Signal*, Dblqh::ScanRecord*, Dblqh::TcConnectionrec*, Dblqh::Fragrecord*, unsigned int)+0x36) [0x6475f6]
/usr/local/bin/ndbd(Dblqh::continue_next_scan_conf(Signal*, Dblqh::ScanRecord::ScanState, Dblqh::ScanRecord*)+0x6b) [0x6483db]
/usr/local/bin/ndbd(Dbtux::continue_scan(Signal*, Ptr<Dbtux::ScanOp>, Dbtux::Frag&, bool)+0x940) [0x70fe50]
/usr/local/bin/ndbd(Dbtux::execNEXT_SCANREQ(Signal*)+0x91) [0x7138e1]
/usr/local/bin/ndbd(Dblqh::send_next_NEXT_SCANREQ(Signal*, SimulatedBlock*, void (SimulatedBlock::*)(Signal*), Dblqh::ScanRecord*, unsigned int)+0xfd) [0x60a6ed]
/usr/local/bin/ndbd(Dblqh::scanNextLoopLab(Signal*, unsigned int, unsigned int, Dblqh::ScanRecord*, Dblqh::Fragrecord*)+0x109) [0x60aac9]
/usr/local/bin/ndbd(Dblqh::continue_next_scan_conf(Signal*, Dblqh::ScanRecord::ScanState, Dblqh::ScanRecord*)+0x6b) [0x6483db]
/usr/local/bin/ndbd(Dbtux::continue_scan(Signal*, Ptr<Dbtux::ScanOp>, Dbtux::Frag&, bool)+0x940) [0x70fe50]
/usr/local/bin/ndbd(Dbtux::execNEXT_SCANREQ(Signal*)+0x91) [0x7138e1]
/usr/local/bin/ndbd(Dblqh::execACC_CHECK_SCAN(Signal*)+0xc8) [0x6076c8]
/usr/local/bin/ndbd(FastScheduler::doJob(unsigned int)+0x1da) [0x85ef1a]
/usr/local/bin/ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6b2) [0x86de12]
/usr/local/bin/ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x758) [0x4fb0e8]
/usr/local/bin/ndbd(real_main(int, char**)+0x2e5) [0x4f9225]
/usr/local/bin/ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xfeb) [0x4f8dab]
/usr/local/bin/ndbd(real_main(int, char**)+0x34c) [0x4f928c]
/usr/local/bin/ndbd(main+0x37) [0x4f5407]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f2709f8e7b3]
/usr/local/bin/ndbd() [0x4f6ae1]
2021-04-23 11:23:17 [ndbd] INFO -- Job Buffer Full
2021-04-23 11:23:17 [ndbd] INFO -- APZJobBuffer.C
2021-04-23 11:23:17 [ndbd] INFO -- Error handler restarting system
2021-04-23 11:23:17 [ndbd] ALERT -- Node 11: Forced node shutdown completed. Caused by error 2334: 'Job buffer congestion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2021-04-23 11:23:17 [ndbd] INFO -- Ndb has terminated (pid 4152230) restarting
2021-04-23 11:23:17 [ndbd] INFO -- Angel reconnected to '172.16.9.21:1186'
2021-04-23 11:23:28 [ndbd] INFO -- Angel reallocated nodeid: 11
2021-04-23 11:23:28 [ndbd] INFO -- Angel pid: 4152229 started child: 44571
2021-04-23 11:23:28 [ndbd] INFO -- Normal start of data node using checkpoint and log info if existing
2021-04-23 11:23:28 [ndbd] INFO -- Configuration fetched from '172.16.9.21:1186', generation: 13
2021-04-23 11:23:28 [ndbd] INFO -- Changing directory to '/usr/local/mysql/data'
2021-04-23 11:23:28 [ndbd] INFO -- SchedulerSpinTimer = 0
2021-04-23 11:23:28 [ndbd] INFO -- AutomaticThreadConfig = 0, NumCPUs = 0
2021-04-23 11:23:28 [ndbd] INFO -- MaxNoOfTriggers set to 16648
NDBMT: non-mt
2021-04-23 11:23:28 [ndbd] INFO -- NDB Cluster -- DB node 11
2021-04-23 11:23:28 [ndbd] INFO -- mysql-8.0.23 ndb-8.0.23 --
2021-04-23 11:23:28 [ndbd] INFO -- Memory Allocation for global memory pools Starting
2021-04-23 11:23:28 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2021-04-23 11:23:28 [ndbd] INFO -- SharedGlobalMemory set to 2048 MB
2021-04-23 11:23:28 [ndbd] INFO -- DataMemory set to 12000 MB
2021-04-23 11:23:28 [ndbd] INFO -- RedoLogBuffer uses 132 MB
2021-04-23 11:23:28 [ndbd] INFO -- DiskPageBuffer uses 70 MB
2021-04-23 11:23:28 [ndbd] INFO -- SchemaTransactionMemory uses 2 MB
2021-04-23 11:23:28 [ndbd] INFO -- TransactionMemory calculated to 1937 MB
2021-04-23 11:23:28 [ndbd] INFO -- Reserved part of TransactionMemory is 363 MB
2021-04-23 11:23:28 [ndbd] INFO -- No Undo log buffer used, will be allocated from TransactionMemory if later defined by command
2021-04-23 11:23:28 [ndbd] INFO -- TransactionMemory can expand and use SharedGlobalMemory if required
2021-04-23 11:23:28 [ndbd] INFO -- QueryMemory can use memory from SharedGlobalMemory until 90% used
2021-04-23 11:23:28 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 14141Mb initial: 16189Mb
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Starting, 4224 pages, page size = 32768
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:28 [ndbd] INFO -- Adding 132Mb to ZONE_19 (1, 4223)
2021-04-23 11:23:28 [ndbd] INFO -- Memory Allocation for global memory pools Completed
Started thread, index = 0, id = 44572, type = NdbfsThread
2021-04-23 11:23:28 [ndbd] INFO -- Loading blocks for data node run-time environment
2021-04-23 11:23:28 [ndbd] INFO -- Starting Sending and Receiving services
Started thread, index = 1, id = 44582, type = SocketClientThread
Started thread, index = 2, id = 44573, type = WatchDogThread
Started thread, index = 3, id = 44583, type = SocketServerThread
2021-04-23 11:23:28 [ndbd] INFO -- Starting the data node run-time environment
2021-04-23 11:23:28 [ndbd] INFO -- First START_ORD executed to connect MGM servers
2021-04-23 11:23:28 [ndbd] INFO -- Received second START_ORD as part of normal start
2021-04-23 11:23:28 [ndbd] INFO -- Disconnect all non-MGM servers
2021-04-23 11:23:28 [ndbd] INFO -- Start excuting the start phases
2021-04-23 11:23:28 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 0, name = CMVMI
2021-04-23 11:23:28 [ndbd] INFO -- Start initiated (mysql-8.0.23 ndb-8.0.23)
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 5, id = 44584, type = NdbfsThread
2021-04-23 11:23:28 [ndbd] INFO -- Touch Memory Starting, 513834 pages, page size = 32768
2021-04-23 11:23:29 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:29 [ndbd] INFO -- Adding 8060Mb to ZONE_19 (4224, 257919)
2021-04-23 11:23:29 [ndbd] INFO -- Adding 7998Mb to ZONE_19 (262145, 255913)
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 1, name = NDBFS
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 6, id = 44600, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 7, id = 44601, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 8, id = 44602, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 9, id = 44603, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 10, id = 44604, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 11, id = 44605, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 12, id = 44606, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 13, id = 44607, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 14, id = 44608, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 15, id = 44609, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 16, id = 44610, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 17, id = 44611, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 18, id = 44612, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 19, id = 44613, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 20, id = 44614, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 21, id = 44615, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 22, id = 44616, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 23, id = 44617, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 24, id = 44618, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 25, id = 44619, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 26, id = 44620, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 27, id = 44621, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 28, id = 44622, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 29, id = 44623, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 30, id = 44624, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 31, id = 44625, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 32, id = 44626, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 33, id = 44627, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 34, id = 44628, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 35, id = 44629, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 36, id = 44630, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 37, id = 44631, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 38, id = 44632, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 39, id = 44633, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 40, id = 44634, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 41, id = 44635, type = NdbfsThread
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 42, id = 44636, type = NdbfsThread
Started thread, index = 43, id = 44637, type = NdbfsThread
Started thread, index = 44, id = 44638, type = NdbfsThread
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 2, name = DBINFO
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 3, name = DBTUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 4, name = DBACC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 5, name = DBTC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 6, name = DBLQH
LCPFragWatchdog : WarnElapsed : 60000(ms) MaxElapsed 180000(ms) : period millis : 1000
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 7, name = DBTUX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 8, name = DBDICT
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 9, name = DBDIH
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 10, name = NDBCNTR
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 11, name = QMGR
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 12, name = TRIX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 13, name = BACKUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 14, name = DBUTIL
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 15, name = SUMA
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 16, name = TSMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 17, name = LGMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 18, name = PGMAN
pgman: page buffer entries = 10
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 19, name = RESTORE
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 20, name = DBSPJ
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 21, name = THRMAN
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 22, name = DBQLQH
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 23, name = DBQACC
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 24, name = DBQTUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 25, name = QBACKUP
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 26, name = DBQTUX
2021-04-23 11:23:29 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 27, name = QRESTORE
2021-04-23 11:23:29 [ndbd] INFO -- READ_CONFIG_REQ phase completed, this phase is used to read configuration and to calculate various sizes and allocate almost all memory needed by the data node in its lifetime
2021-04-23 11:23:29 [ndbd] INFO -- Not initial start
2021-04-23 11:23:29 [ndbd] INFO -- Local sysfile: Node restorable on its own, gci: 0, version: 70603
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 0 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 0 has made some file system initialisations
2021-04-23 11:23:29 [ndbd] INFO -- We are running with 0 LDM workers and 4 REDO log parts. This means that we can avoid using a mutex to access REDO log parts
2021-04-23 11:23:29 [ndbd] INFO -- Watchdog KillSwitch off.
2021-04-23 11:23:29 [ndbd] INFO -- Starting QMGR phase 1
2021-04-23 11:23:29 [ndbd] INFO -- Starting with m_restart_seq set to 12
2021-04-23 11:23:29 [ndbd] INFO -- DIH reported normal start, now starting the Node Inclusion Protocol
2021-04-23 11:23:29 [ndbd] INFO -- findNeighbours from: 2988 old (left: 65535 right: 65535) new (10 10)
2021-04-23 11:23:29 [ndbd] INFO -- Include node protocol completed, phase 1 in QMGR completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 1 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 1 initialised some variables and included node in cluster, locked memory if configured to do so
2021-04-23 11:23:29 [ndbd] INFO -- Starting with m_restart_seq set to 12
2021-04-23 11:23:29 [ndbd] INFO -- Asking master node to accept our start (nodeId = 10 is master), GCI = 957711
2021-04-23 11:23:29 [ndbd] INFO -- NDBCNTR master accepted us into cluster, start NDB start phase 1
2021-04-23 11:23:29 [ndbd] INFO -- We are performing a node restart
2021-04-23 11:23:29 [ndbd] INFO -- LDM(0): Started LDM restart phase 1 (read REDO log page headers to init REDO log data)
2021-04-23 11:23:29 [ndbd] INFO -- Schema file initialisation Starting
2021-04-23 11:23:29 [ndbd] INFO -- Schema file initialisation Completed
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 1 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 2 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 2 did more initialisations, master accepted our start, we started REDO log initialisations
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 2
2021-04-23 11:23:29 [ndbd] INFO -- Request permission to start our node from master Starting
2021-04-23 11:23:29 [ndbd] INFO -- Request permission to start our node from master Completed
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 2 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 3 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 3 locked the data dictionary, performed local connection setups, we asked for permission to start our node
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 3
2021-04-23 11:23:29 [ndbd] INFO -- NDB start phase 3 completed
2021-04-23 11:23:29 [ndbd] INFO -- Start phase 4 completed
2021-04-23 11:23:29 [ndbd] INFO -- Phase 4 continued preparations of the REDO log
2021-04-23 11:23:29 [ndbd] INFO -- Start NDB start phase 4
2021-04-23 11:23:29 [ndbd] INFO -- Request copying of distribution and dictionary information from master Starting
2021-04-23 11:23:29 [ndbd] INFO -- Completed setting up multiple transporters to nodes in the same node group
2021-04-23 11:23:29 [ndbd] INFO -- LDM(0):Ready to start execute REDO log phase, prepare REDO log phase completed
2021-04-23 11:23:30 [ndbd] INFO -- Receive arbitrator node 1 [ticket=f05b0002b3b91fc5]
2021-04-23 11:23:48 [ndbd] INFO -- findNeighbours from: 2905 old (left: 10 right: 10) new (10 12)
2021-04-23 11:23:48 [ndbd] INFO -- Node 10 disconnected in state: 0
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bottom and thread_stack will always show up as zero.
stack_bottom = 0 thread_stack 0x0
/usr/local/bin/ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x89128e]
/usr/local/bin/ndbd(ndb_print_stacktrace()+0x45) [0x849245]
/usr/local/bin/ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x1f) [0x7fd56f]
/usr/local/bin/ndbd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xf2) [0x8629e2]
/usr/local/bin/ndbd(Qmgr::execDISCONNECT_REP(Signal*)+0x1ec) [0x77bd1c]
/usr/local/bin/ndbd(FastScheduler::doJob(unsigned int)+0x1da) [0x85ef1a]
/usr/local/bin/ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6b2) [0x86de12]
/usr/local/bin/ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x758) [0x4fb0e8]
/usr/local/bin/ndbd(real_main(int, char**)+0x2e5) [0x4f9225]
/usr/local/bin/ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xfeb) [0x4f8dab]
/usr/local/bin/ndbd(real_main(int, char**)+0x34c) [0x4f928c]
/usr/local/bin/ndbd(main+0x37) [0x4f5407]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f2709f8e7b3]
/usr/local/bin/ndbd() [0x4f6ae1]
2021-04-23 11:23:48 [ndbd] INFO -- Node 10 disconnected in phase: 3
2021-04-23 11:23:48 [ndbd] INFO -- QMGR (Line: 4245) 0x00000000
2021-04-23 11:23:48 [ndbd] INFO -- Error handler restarting system
2021-04-23 11:23:48 [ndbd] INFO -- Error handler shutdown completed - exiting
2021-04-23 11:23:48 [ndbd] INFO -- Angel detected startup failure, count: 1
2021-04-23 11:23:48 [ndbd] ALERT -- Node 11: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2021-04-23 11:23:48 [ndbd] INFO -- Ndb has terminated (pid 44571) restarting
2021-04-23 11:23:48 [ndbd] INFO -- Angel reconnected to '172.16.9.21:1186'
2021-04-23 11:23:51 [ndbd] INFO -- Angel reallocated nodeid: 11
2021-04-23 11:23:51 [ndbd] INFO -- Angel pid: 4152229 started child: 44736
2021-04-23 11:23:51 [ndbd] INFO -- Normal start of data node using checkpoint and log info if existing
2021-04-23 11:23:51 [ndbd] INFO -- Configuration fetched from '172.16.9.21:1186', generation: 13
2021-04-23 11:23:51 [ndbd] INFO -- Changing directory to '/usr/local/mysql/data'
2021-04-23 11:23:51 [ndbd] INFO -- SchedulerSpinTimer = 0
2021-04-23 11:23:51 [ndbd] INFO -- AutomaticThreadConfig = 0, NumCPUs = 0
2021-04-23 11:23:51 [ndbd] INFO -- MaxNoOfTriggers set to 16648
NDBMT: non-mt
2021-04-23 11:23:51 [ndbd] INFO -- NDB Cluster -- DB node 11
2021-04-23 11:23:51 [ndbd] INFO -- mysql-8.0.23 ndb-8.0.23 --
2021-04-23 11:23:51 [ndbd] INFO -- Memory Allocation for global memory pools Starting
2021-04-23 11:23:51 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2021-04-23 11:23:51 [ndbd] INFO -- SharedGlobalMemory set to 2048 MB
2021-04-23 11:23:51 [ndbd] INFO -- DataMemory set to 12000 MB
2021-04-23 11:23:51 [ndbd] INFO -- RedoLogBuffer uses 132 MB
2021-04-23 11:23:51 [ndbd] INFO -- DiskPageBuffer uses 70 MB
2021-04-23 11:23:51 [ndbd] INFO -- SchemaTransactionMemory uses 2 MB
2021-04-23 11:23:51 [ndbd] INFO -- TransactionMemory calculated to 1937 MB
2021-04-23 11:23:51 [ndbd] INFO -- Reserved part of TransactionMemory is 363 MB
2021-04-23 11:23:51 [ndbd] INFO -- No Undo log buffer used, will be allocated from TransactionMemory if later defined by command
2021-04-23 11:23:51 [ndbd] INFO -- TransactionMemory can expand and use SharedGlobalMemory if required
2021-04-23 11:23:51 [ndbd] INFO -- QueryMemory can use memory from SharedGlobalMemory until 90% used
2021-04-23 11:23:51 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 14141Mb initial: 16189Mb
2021-04-23 11:23:51 [ndbd] INFO -- Touch Memory Starting, 4224 pages, page size = 32768
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:52 [ndbd] INFO -- Adding 132Mb to ZONE_19 (1, 4223)
2021-04-23 11:23:52 [ndbd] INFO -- Memory Allocation for global memory pools Completed
Started thread, index = 0, id = 44737, type = NdbfsThread
2021-04-23 11:23:52 [ndbd] INFO -- Loading blocks for data node run-time environment
2021-04-23 11:23:52 [ndbd] INFO -- Starting Sending and Receiving services
Started thread, index = 1, id = 44747, type = SocketClientThread
Started thread, index = 2, id = 44738, type = WatchDogThread
Started thread, index = 3, id = 44748, type = SocketServerThread
2021-04-23 11:23:52 [ndbd] INFO -- Starting the data node run-time environment
2021-04-23 11:23:52 [ndbd] INFO -- First START_ORD executed to connect MGM servers
2021-04-23 11:23:52 [ndbd] INFO -- Received second START_ORD as part of normal start
2021-04-23 11:23:52 [ndbd] INFO -- Disconnect all non-MGM servers
2021-04-23 11:23:52 [ndbd] INFO -- Start excuting the start phases
2021-04-23 11:23:52 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 0, name = CMVMI
2021-04-23 11:23:52 [ndbd] INFO -- Start initiated (mysql-8.0.23 ndb-8.0.23)
NDBFS/AsyncFile: Allocating 275256 for In/Deflate buffer
Started thread, index = 5, id = 44749, type = NdbfsThread
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Starting, 513834 pages, page size = 32768
2021-04-23 11:23:52 [ndbd] INFO -- Touch Memory Completed
2021-04-23 11:23:52 [ndbd] INFO -- Adding 8060Mb to ZONE_19 (4224, 257919)
2021-04-23 11:23:52 [ndbd] INFO -- Adding 7998Mb to ZONE_19 (262145, 255913)
2021-04-23 11:23:52 [ndbd] INFO -- Sending READ_CONFIG_REQ to index = 1, name = NDBFS
Subject
Views
Written By
Posted
Sorry, you can't reply to this topic. It has been closed.
Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.