MySQL Forums
Forum List  »  Install & Repo

MySQL NDB cluster: Fatal error at service startup of mysqld nodes
Posted by: Patrick Jaeger
Date: January 30, 2014 11:02AM

Hi,

we are using an MySQL cluster with the NDBCLUSTER engine. Everything works fine, but now we are facing errors during the automated startup of the mysqld data nodes of the cluster. Nobody knows if this did work correctly in the past.


During the system restart or when we try to start the service manually ("service mysql start" using the provided standard script for /etc/init.d/mysql) we receive the following error: see [0]


This is understandable, because the check in /etc/init.d/mysql for this file must fail, the file is not there. Even if we thin out the start script and just try a "/usr/bin/mysqld_safe &", neither the service does start, nor the pid-file is created.

If we start it as root "mysqld_safe &" everything is fine and the datanodes start and are recognized by the management node (pid file is created after a while). So we checked the file/directory permissions for user mysql in /var/lib/mysql, but everything seems to be ok.

See the error logs for successful ("root" mysqld_safe) [1] and failed (service/"mysql" mysqld_safe) [2] start of a data node,



Searching for the unknown error 157 or "Incorrect information in file: './mysql/user.frm'" we found information about recreating tables. BUT after a succesful start as root, this table is still accessible, users and privileges can be created/managed.

I would appreciate any helpful ideas.

Thanks,
Patrick


OS: CentOS release 6.4
Version: Server version: 5.6.11-ndb-7.3.2-cluster-gpl MySQL Cluster Community Server (GPL)


[0]
Starting MySQL...............................The server quit updating PID file (/var/lib/mysql/hostname.pid). [FAILED]


[1]
error log during a successful start attempt:
2014-01-30 16:43:22 19873 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2014-01-30 16:43:23 19873 [Note] NDB: NodeID is 5, management server 'XXX.XXX.XXX.XXX:1186'
2014-01-30 16:43:53 19873 [Note] NDB[0]: NodeID: 5, some storage nodes connected
2014-01-30 16:43:53 19873 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2014-01-30 16:43:53 19873 [Note] Starting Cluster Binlog Thread
2014-01-30 16:43:53 19873 [Note] Server hostname (bind-address): '*'; port: 3306
2014-01-30 16:43:53 19873 [Note] IPv6 is available.
2014-01-30 16:43:53 19873 [Note] - '::' resolves to '::';
2014-01-30 16:43:53 19873 [Note] Server socket created on IP: '::'.
2014-01-30 16:43:53 19873 [Note] table './mysql/user' opened read only
2014-01-30 16:43:53 19873 [Note] table './mysql/db' opened read only
2014-01-30 16:43:53 19873 [Note] table './mysql/proxies_priv' opened read only
2014-01-30 16:43:53 19873 [Note] table './mysql/tables_priv' opened read only
2014-01-30 16:43:53 19873 [Note] table './mysql/columns_priv' opened read only
2014-01-30 16:43:53 19873 [Note] table './mysql/procs_priv' opened read only
2014-01-30 16:43:53 19873 [Note] Event Scheduler: Loaded 0 events
2014-01-30 16:43:53 19873 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.11-ndb-7.3.2-cluster-gpl' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2014-01-30 16:43:53 19873 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
2014-01-30 16:43:53 19873 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
2014-01-30 16:43:53 19873 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
2014-01-30 16:43:53 19873 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)


[2]
error log during a failed start attempt:
2014-01-30 16:46:36 20365 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
2014-01-30 16:46:36 20365 [Note] NDB: NodeID is 5, management server 'XXX.XXX.XXX.XXX:1186'
2014-01-30 16:47:06 20365 [Note] NDB[0]: NodeID: 5, no storage nodes connected (timed out)
2014-01-30 16:47:06 20365 [Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds
2014-01-30 16:47:06 20365 [Note] Starting Cluster Binlog Thread
2014-01-30 16:47:06 20365 [Note] Server hostname (bind-address): '*'; port: 3306
2014-01-30 16:47:06 20365 [Note] IPv6 is available.
2014-01-30 16:47:06 20365 [Note] - '::' resolves to '::';
2014-01-30 16:47:06 20365 [Note] Server socket created on IP: '::'.
2014-01-30 16:47:07 20365 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './mysql/user.frm'
2014-01-30 16:47:07 20365 [ERROR] Fatal error: Can't open and lock privilege tables: Got error 157 'Unknown error code' from NDBCLUSTER
140130 16:47:07 mysqld_safe mysqld from pid file /var/lib/mysql/concar-mysql02.pid ended

Options: ReplyQuote


Subject
Written By
Posted
MySQL NDB cluster: Fatal error at service startup of mysqld nodes
January 30, 2014 11:02AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.