auto discovery of existing databases not working
We have a problem with the database autodiscovery when you add a MySQL API node to the cluster.
When we add an API node, there is a step when the node first connects to the management node where there is a discovery of the database:
2016-05-26 15:38:54 26109 [Note] - '::' resolves to '::';
2016-05-26 15:38:54 26109 [Note] Server socket created on IP: '::'.
2016-05-26 15:38:54 26109 [Note] Event Scheduler: Loaded 0 events
2016-05-26 15:38:54 26109 [Note] mysqld: ready for connections.
Version: '5.6.21-ndb-7.3.7-cluster-gpl-log' socket: '/nn/mysql/mysql_cluster/mysql.sock' port: 3306 MySQL Cluster Community Server (GPL)
2016-05-26 15:38:54 26109 [Note] ndb_index_stat_proc: Ndb object created with reference : 0x80060034, name : Ndb Index Statistics monitoring
2016-05-26 15:38:54 26109 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
2016-05-26 15:38:54 26109 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_WRITE)
2016-05-26 15:38:54 26109 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
2016-05-26 15:38:54 26109 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_WRITE)
2016-05-26 15:38:54 26109 [Note] NDB: Cleaning stray tables from database 'information_schema'
2016-05-26 15:38:54 26109 [Note] NDB: Cleaning stray tables from database 'ndbinfo'
2016-05-26 15:38:54 26109 [Note] NDB: Cleaning stray tables from database 'performance_schema'
2016-05-26 15:38:54 26109 [Note] NDB: Cleaning stray tables from database 'test'
2016-05-26 15:38:54 26109 [Note] NDB: Discovered missing database 'Mailboxes'
2016-05-26 15:38:54 26109 [Note] NDB: missing frm for Mailboxes.schema_history, discovering...
2016-05-26 15:38:54 26109 [Note] NDB: missing frm for Mailboxes.mail, discovering...
2016-05-26 15:38:54 26109 [Note] NDB: missing frm for Mailboxes.mailbox, discovering...
2016-05-26 15:38:54 26109 [Note] NDB: missing frm for Mailboxes.interface_history, discovering...
2016-05-26 15:38:54 [NdbApi] INFO -- Flushing incomplete GCI:s < 2217166/3
2016-05-26 15:38:54 [NdbApi] INFO -- Flushing incomplete GCI:s < 2217166/3
2016-05-26 15:38:54 26109 [Note] NDB Binlog: starting log at epoch 2217166/3
2016-05-26 15:38:54 26109 [Note] NDB Binlog: ndb tables writable
2016-05-26 15:38:54 26109 [Note] NDB Binlog: Node: 10, subscribe from node 50, Subscriber bitmask 400000
2016-05-26 15:38:54 26109 [Note] NDB Binlog: Node: 10, subscribe from node 58, Subscriber bitmask 40400000
2016-05-26 15:38:54 26109 [Note] NDB Binlog: Node: 11, subscribe from node 58, Subscriber bitmask 40000000
2016-05-26 15:38:54 26109 [Note] NDB Binlog: Node: 11, subscribe from node 50, Subscriber bitmask 40400000
But sometimes we add a node and it doesn’t discover the databases: the DISCOVER TABLE event doesn’t see any missing databases. In those cases we have to run the CREATE DATABASE command from that new API node, which weirdly succeeds despite the databases existing on the data nodes, and then the API node can access that existing database. We have tried a lot of experimentation to determine what governs this behavior of the DISCOVER TABLE event but are coming up short. Has anyone ever encountered this?