MySQL Forums
Forum List  »  NDB clusters

Re: ndb_restore
Posted by: Frazer Clement
Date: February 05, 2020 11:05AM

Hi,
Sorry to hear about your bad experiences, thanks for the description.

In general ndb_restore can restore backups taken from different versions of MySQL
Cluster, so could be used to rebuild your cluster.

These are the steps you could use :
1. Run mysqldump with the --no-data option to capture the existing schema
as SQL (Unless you already have it captured/reproducable by other means)

2. Capture an online backup (ndb_mgm -e "START BACKUP")

3. Setup an empty cluster (probably on 7.6 unless you actually want to
upgrade to 8.0)

4. Use the schema dump captured in 1 to recreate the schema.

5. Use ndb_restore --restore-data to restore the data from the Ndb
backup captured in 2.

One thing to note is that in 8.0, the data nodes can now write multiple
backup filesets, one per LDM thread. When restoring such a backup back
to 7.6, ndb_restore --restore-data must be run for each fileset.

It would be great if you could file a bug on bugs.mysql.com describing the
data node crashes you mentioned as I don't think these are expected :
"pointer too large"
"regTcPtr.p->m_dealloc_data.m_dealloc_ref_count == 1 failed"
"NGPtr.p->nodeCount <= cnoReplicas failed"

The ndb_error_reporter tool can be used to capture crash traces across
nodes for upload.

Regarding the source of the initial problem - I think that perhaps the
MySQL Server 8.0 has connected to the 7.6 Cluster with the default
value of ndb_schema_dist_upgrade_allowed=1, causing it to upgrade the
schema distribution table schema, resulting in problems for the
7.6 Servers as you describe.

The documentation for ndb_schema_dist_upgrade_allowed currently
suggests that the upgrade will not happen until all MySQLD servers
are on the same version, but unfortunately this is not correct.

The product should be fixed to either implement this check, or at least
change the default to zero.

Until then it is best to upgrade MySQLDs with --ndb_schema_dist_upgrade
set to 0 so that the schema distribution tables are untouched.

You mention that your application truncates tables and this may have
compounded problems as table truncation is implemented as a table
drop and recreate at the Ndb level.

If a table is recreated by a MySQL Server 8.0 then it will be created
with the new MySQL metadata format (sdi) rather than the old one (frm)
embedded in the Ndb table definition - this can also result in problems for
older MySQL Server 7.6 instances opening the table.

Unfortunately the upgrade to MySQL 8.0 is not as smooth as upgrades have
been in the past, due to the MySQL data dictionary and MySQL Cluster schema
distribution changes in that release, meaning that extra care is needed when
upgrading.

Summary

- You can use mysqldump + ndb_restore together to rebuild your
cluster (and go back to 7.6 or forward to 8.0 as you prefer)

- The problem has probably started due to a MySQL Server 8.0
connecting to a 7.6 cluster with the default value of
ndb_schema_dist_upgrade_allowed=1

- The problem may have been worsened by MySQL Server 8.0
nodes performing TRUNCATE during the upgrade

- The data node crashes are unexpected, please file bug reports
with crash traces and logs.

Sorry again for your bad experience, I hope you can get things fixed up
and thanks for reporting the problems.

We will make some improvements to make this kind of situation less likely.

Frazer

(MySQL Cluster development)

Options: ReplyQuote


Subject
Views
Written By
Posted
295
January 28, 2020 11:32AM
Re: ndb_restore
117
February 05, 2020 11:05AM
85
February 05, 2020 01:12PM
88
February 05, 2020 04:32PM
83
February 09, 2020 03:38PM
75
February 11, 2020 09:51AM


Sorry, only registered users may post in this forum.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.