MySQL Forums
Forum List  »  NDB clusters

Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
Posted by: Pradeep Manickam
Date: March 05, 2021 04:01PM

Error log:
========== >
Time: Friday 5 March 2021 - 21:18:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: tsman.cpp
Error object: TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
Program: ndbd
Pid: 1493
Version: mysql-8.0.23 ndb-8.0.23
Trace file name: ndb_6_trace.log.3
Trace file path: /var/lib/mysql-cluster/ndb_6_trace.log.3 [t1..t1]
***EOM***


output log
==========

021-03-05 21:17:57 [ndbd] INFO -- Request copying of distribution and dictionary information from main Completed
2021-03-05 21:17:57 [ndbd] INFO -- NDB start phase 4 completed
2021-03-05 21:17:57 [ndbd] INFO -- Start NDB start phase 5 (only to DBDIH)
2021-03-05 21:17:57 [ndbd] INFO -- Restore Database Off-line Starting
2021-03-05 21:17:57 [ndbd] INFO -- GSN_START_RECREQ keepGci: 0 (5249)
2021-03-05 21:17:57 [ndbd] INFO -- All start fragments sent, requesting LDM to restore all fragments and to execute the REDO log to bring the database to an off-line but consistent state
execSTART_RECREQ changing srnodes from 0000000000000000000000000000000000000070 to 0000000000000000000000000000000000000040
2021-03-05 21:17:57 [ndbd] INFO -- LDM instance 0: Restored LCP : 0 fragments, 0 rows, 0 millis, 0 rows/s
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Completed fuzzy restore 0 fragments (0 from LCP, 0 by non-trans copy)
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Starting DD Undo log application
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Completed DD Undo log application
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): Starting to rebuild ordered indexes
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 9 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 12 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): index id 14 rebuild done
2021-03-05 21:17:57 [ndbd] INFO -- LDM(0): We have completed restoring our fragments and executed REDO log and rebuilt ordered indexes
2021-03-05 21:17:57 [ndbd] INFO -- Restore Database Off-line Completed
2021-03-05 21:17:57 [ndbd] INFO -- Bring Database On-line Starting
2021-03-05 21:17:57 [ndbd] INFO -- Bring Database On-line Completed
2021-03-05 21:17:57 [ndbd] INFO -- Starting REDO logging
2021-03-05 21:17:57 [ndbd] INFO -- Make On-line Database recoverable by waiting for LCP Starting, all parallel threads have now ceased their activity and we have a single wait state here
For help with below stacktrace consult:
https://dev.mysql.com/doc/refman/en/using-stack-trace.html
Also note that stack_bot and thread_stack will always show up as zero.
Base address/slide: 0x55dfb86b3000
With use of addr2line, llvm-symbolizer, or, atos, subtract the addresses in
stacktrace with the base address before passing them to tool.
For tools that have options for slide use that, e.g.:
llvm-symbolizer --adjust-vma=0x55dfb86b3000 ...
atos -s 0x55dfb86b3000 ...
stack_bot = 0 thread_stack 0x0
ndbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55dfb8f421c1]
ndbd(ndb_print_stacktrace()+0x56) [0x55dfb8f41606]
ndbd(ErrorReporter::handleError(int, char const*, char const*, NdbShutdownType)+0x33) [0x55dfb8edd023]
ndbd(SimulatedBlock::progError(int, int, char const*, char const*) const+0xfd) [0x55dfb8f6954d]

ndbd(Tsman::execEND_LCPREQ(Signal*)+0x156) [0x55dfb8ed6066]
ndbd(FastScheduler::doJob(unsigned int)+0x1d5) [0x55dfb8f656e5]
ndbd(ThreadConfig::ipControlLoop(NdbThread*)+0x6a8) [0x55dfb8f75a28]
ndbd(ndbd_run(bool, int, char const*, int, char const*, bool, bool, bool, unsigned int, int, int, unsigned long)+0x769) [0x55dfb8ba2ef9]
ndbd(real_main(int, char**)+0x37f) [0x55dfb8ba0f2f]
ndbd(angel_run(char const*, Vector<BaseString> const&, char const*, int, char const*, bool, bool, bool, int, int)+0xf48) [0x55dfb8ba0958]
ndbd(real_main(int, char**)+0x2e3) [0x55dfb8ba0e93]
ndbd(main+0x3e) [0x55dfb8ba0a5e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f865f1720b3]
ndbd(_start+0x2e) [0x55dfb8b9e38e]
2021-03-05 21:18:01 [ndbd] INFO -- /var/lib/pb2/sb_1-1373275-1607727651.83/release/mysql-cluster-gpl-8.0.23/storage/ndb/src/kernel/blocks/tsman.cpp
2021-03-05 21:18:01 [ndbd] INFO -- TSMAN (Line: 2984) 0x00000006 Check m_lcp_ongoing failed
2021-03-05 21:18:01 [ndbd] INFO -- Error handler shutting down system
2021-03-05 21:18:01 [ndbd] INFO -- Error handler shutdown completed - exiting
2021-03-05 21:18:01 [ndbd] ALERT -- Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming >



Edited 2 time(s). Last edit at 03/05/2021 04:45PM by Pradeep Manickam.

Options: ReplyQuote


Subject
Views
Written By
Posted
Re: Node 6: Forced node shutdown completed. Occurred during startphase 5. Caused by error 2341
227
March 05, 2021 04:01PM


Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.