ndb_restore

[SHAWN HOGAN]

The short version: I'm curious if a backup restored with ndb_restore needs to be restored to the same version that the backup was taken from?

The best I can tell, we ran into a situation where someone's laptop had mysqld 8.x installed on it. It connected to our MySQL Cluster running 7.6.x and as part of that, it "upgraded" the cluster to 8.0. All of a sudden all the normal SQL nodes running 7.6 disconnect because the 7.6 data nodes are syncing tables/databases differently. We start scrambling to get the system back online and as part of that, we decided to upgrade the whole cluster to 8.0 since the inadvertent connection from mysqld running 8.0 already partially upgraded it. SQL nodes and management nodes upgraded as expected without any issues. We start to do rolling restarts to upgrade the data nodes and the whole cluster comes crashing down. Data nodes start crashing with all sorts different errors... Sometimes "pointer too large", sometimes "regTcPtr.p->m_dealloc_data.m_dealloc_ref_count == 1 failed", sometimes "NGPtr.p->nodeCount <= cnoReplicas failed" even though all other data nodes are up and cluster is working fine.

Basically it's become an utter nightmare. Now we are in a situation where we had to upgrade all the SQL nodes to 8.0 since 7.6 SQL nodes could no longer connect to the cluster after that one mysqld "upgraded" ndb tables when it connected. New tables created don't function because the data nodes are still on 7.6 (since we can't upgrade them), a few tables get truncated by an automated process once in awhile and those are borked now since internally a truncate creates a new table.

It's crazy to me that an inadvertent connection of an 8.0 SQL node to a cluster can basically screw up all the other 7.6 nodes this badly to the point that the whole thing is unusable.

I know 8.0 can't be downgraded normally, but just trying to figure out my options now. The current setup is 7.6 data nodes and 8.0 SQL nodes (it was the only combo that I could do to get things online). This is a decent sized cluster (a little more than 1TB of Index/Data spread across 8 data nodes [130GB per node]), interconnected with Infiniband with cluster containing billions of records.

Option 1: Can I take a backup from 8.0, completely reset/empty cluster with 7.6 and ndb_restore a 8.0 backup to a 7.6 cluster?

Option 2: mysqldump everything (crazy if this is the route we have to take with billions of records) and import it into a reset 7.6 cluster.



/rant

Is it by design that any mysqld 8.0 connection to a 7.6 cluster completely wrecks the cluster or did something unknown/crazy happen on our end? Maybe it shouldn't try to be helpful and "upgrade" everything to 8.0 unless the data nodes are running 8.0?

/rant