Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
Error log:
========== >
Time: Friday 5 March 2021 - 21:18:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1493
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.3
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.3 [t1..t1]
***EOM***
output log
==========
021-03-05 21:17:57 [ndbd] INFO -- Request copying of distribution and dictionary information from main Completed
2021-03-05 21:17:57 [ndbd] INFO -- NDB start phase 4 completed
2021-03-05 21:17:57 [ndbd] INFO -- Start NDB start phase 5 (only to DBDIH)
2021-03-05 21:17:57 [ndbd] INFO -- Restore Database Off-line Starting
2021-03-05 21:17:57 [ndbd] INFO -- GSN_START_RECREQ keepGci: 0 (5249)
2021-03-05 21:17:57 [ndbd] INFO -- All start fragments sent, requesting LDM to restore all fragments and to execute the REDO log to bring the database to an off-line but consistent state
execSTART_RECREQ changing srnodes from 0000000000000000000000000000000000000070 to 0000000000000000000000000000000000000040
2021-03-05 21:17:57 [ndbd] INFO -- LDM instance 0: Restored LCP : 0 fragments, 0 rows, 0 millis, 0 rows/s
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Completed fuzzy restore 0 fragments (0 from LCP, 0 by non-trans copy)
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Starting DD Undo log application
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Completed DD Undo log application
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Starting to rebuild ordered indexes
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 9 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 12 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 14 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): We have completed restoring our fragments and executed REDO log and rebuilt ordered indexes
2021-03-05 21:17:57 [ndbd] INFO -- Restore Database Off-line Completed
2021-03-05 21:17:57 [ndbd] INFO -- Bring Database On-line Starting
2021-03-05 21:17:57 [ndbd] INFO -- Bring Database On-line Completed
2021-03-05 21:17:57 [ndbd] INFO -- Starting REDO logging
2021-03-05 21:17:57 [ndbd] INFO -- Make On-line Database recoverable by waiting for LCP Starting, all parallel threads have now ceased their activity and we have a single wait state here
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bot and thread_stack will always show up as zero.
Base address/slide: 0x55dfb86b3000
With use of addr2line, llvm-symbolizer, or, atos, subtract the addresses in
stacktrace with the base address before passing them to tool.
For tools that have options for slide use that, e.g.:
llvm-symbolizer --adjust-vma=0x55dfb86b3000 ...
atos -s 0x55dfb86b3000 ...
stack_bot = 0 thread_stack 0x0
ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55dfb8f421c1]
ndbd(ndb_print_stacktrace()+0x56) [0x55dfb8f41606]
ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x33) [0x55dfb8edd023]
ndbd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xfd) [0x55dfb8f6954d]
ndbd(Tsman::execEND_LCPREQ(Signal*)+0x156) [0x55dfb8ed6066]
ndbd(FastScheduler::doJob(unsigned int)+0x1d5) [0x55dfb8f656e5]
ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6a8) [0x55dfb8f75a28]
ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x769) [0x55dfb8ba2ef9]
ndbd(real_main(int, char**)+0x37f) [0x55dfb8ba0f2f]
ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xf48) [0x55dfb8ba0958]
ndbd(real_main(int, char**)+0x2e3) [0x55dfb8ba0e93]
ndbd(main+0x3e) [0x55dfb8ba0a5e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f865f1720b3]
ndbd(_start+0x2e) [0x55dfb8b9e38e]
2021-03-05 21:18:01 [ndbd] INFO -- /var/lib/pb2/sb_1-1373275-1607727651.83/release/mysql-cluster-gpl-8.0.23/storage/ndb/src/kernel/blocks/tsman.cpp
2021-03-05 21:18:01 [ndbd] INFO -- TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
2021-03-05 21:18:01 [ndbd] INFO -- Error handler shutting down system
2021-03-05 21:18:01 [ndbd] INFO -- Error handler shutdown completed - exiting
2021-03-05 21:18:01 [ndbd] ALERT -- Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming >
Edited 2 time(s). Last edit at 03/05/2021 04:45PM by Pradeep Manickam.