ndb_mgmd dies without errors

[Andrew Harrison]

We are currently using 5.0.18 for an NDB Cluster configuration across four boxes. Box A & B host the Node Management daemon (ndb_mgmd) and boxes C & D host a mysql node and 4 ndbd nodes on each box.

For the node replication we have been using the Dolphin SCI cards to allow for scaleability up to the level required. Recently we have needed to switch to the backup Gigabit LAN as we were having problems with the SCI cards. Around the same time the Node Management daemons on boxes A & B started failing for apparently no reason. This left us with no backup facility (backups run during the night with no-one on hand to monitor). We have re-introduced the SCI cards following updated drivers but still the ndb_mgmd processes continue to die without any errors.

Has anyone else come across this problem or does anyone have any clues as to why this is happening.

Below is the contents of the Cluster log (Box A) from the time the ndb_mgmd process is started to when it lost contact:

2006-07-24 15:23:55 [MgmSrvr] INFO -- NDB Cluster Management Server. Version 5.0.18
2006-07-24 15:23:55 [MgmSrvr] INFO -- Id: 1, Command port: 1186
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 10 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 8 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 7 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 6 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 9 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 3 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 5 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 1: Node 4 Connected
2006-07-24 15:23:55 [MgmSrvr] INFO -- Node 3: Started arbitrator node 1 [ticket=486a00048acdf4e9]

Below is the contents of the Node Management Console before the Node Management daemons dies:

Connected to Management Server at: 172.23.77.198:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 8 node(s)
id=3 @172.23.77.205 (Version: 5.0.18, Nodegroup: 0, Master)
id=4 @172.23.77.206 (Version: 5.0.18, Nodegroup: 0)
id=5 @172.23.77.205 (Version: 5.0.18, Nodegroup: 1)
id=6 @172.23.77.206 (Version: 5.0.18, Nodegroup: 1)
id=7 @172.23.77.205 (Version: 5.0.18, Nodegroup: 2)
id=8 @172.23.77.206 (Version: 5.0.18, Nodegroup: 2)
id=9 @172.23.77.205 (Version: 5.0.18, Nodegroup: 3)
id=10 @172.23.77.206 (Version: 5.0.18, Nodegroup: 3)

[ndb_mgmd(MGM)] 2 node(s)
id=1 @172.23.77.198 (Version: 5.0.18)
id=2 @172.23.77.199 (Version: 5.0.18)

[mysqld(API)] 12 node(s)
id=11 @172.23.77.205 (Version: 5.0.18)
id=12 @172.23.77.206 (Version: 5.0.18)
id=13 (not connected, accepting connect from 172.23.77.205)
id=14 (not connected, accepting connect from 172.23.77.206)



Any help appreciated