MySQL Forums
Forum List  »  NDB clusters

NDB Restore freeze
Posted by: Giuseppe Rossi
Date: February 17, 2021 09:05AM

I have configured a ndb database cluster with the following nodes:
-4 x Data Nodes (6CPU, 16GB RAM, Ubuntu 20.04)
-2 x SQL Nodes (4 CPU, 8GB RAM, Ubuntu 20.04)
-2 x Management Nodes (2 CPU, 4GB RAM, Ubuntu 20.04)
-2 x LoadBalancer with a Virtual IP for external access

Cluster is working correctly but I can't manage to perform a full restore using ndb_restore. Database contains milions of data. Dump size is around 8GB.
When I start a restore, process freezes and it gives no error. I tried to wait but nothing happened. I analyzed logs but there's nothing in particular about that. SQL Node drops its ID and connection and then nothing happens.

Every time to restart the cluster I have to shutdown datanodes and restart with --initial reimporting all data.

This is the procedure I follow:
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_meta --backupid=5 --backup_path="BACKUP PATH" --verbose=255
(Only one time to import schema)
ndb_restore --ndb_connectstring=192.168.1.1,192.168.1.2 --nodeid=100 --disable-indexes --restore_data --backupid=5 --backup_path="BACKUP_PATH" --verbose=255
(One for each dataNode 100,101,102,103)

Schema restores succesfully but sometimes data restore freezes. It's strange becuase it freezes at different times and points.

This is the configuration:
SharedGlobalMemory=1G
MaxNoOfExecutionThreads=4
MaxNoOfConcurrentOperations=1000000
MaxNoOfTables=4096
TransactionDeadlockDetectionTimeout=100000000
TransactionInactiveTimeout=100000000
MaxNoOfTriggers=4096 # Default is 768
FragmentLogFileSize=512M
NoOfFragmentLogFiles=100
RealTimeScheduler=1
MinDiskWriteSpeed=10M
MaxDiskWriteSpeed=50M


[ndb_mgmd]
HostName=192.168.1.1
NodeId=1
DataDir=/var/lib/mysql-cluster

[ndb_mgmd]
HostName=192.168.1.2
NodeId=2
DataDir=/var/lib/mysql-cluster

[ndbd]
HostName=192.168.1.200
NodeId=100
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.201
NodeId=101
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.202
NodeId=102
DataDir=/usr/local/mysql/data

[ndbd]
HostName=192.168.1.203
NodeId=103
DataDir=/usr/local/mysql/data

[mysqld]
NodeId=50
HostName=192.168.1.100

[mysqld]
NodeId=51
HostName=192.168.1.101

Process stacks at points like this:
Processing data in table: DBNAME/def/TABLENAME(211) fragment 0

When I start a restore and it freezes this is the log result in the dataNode
DataNode log:
2021-02-17 12:24:27 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:25:32 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:26:20 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:27:23 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:28:50 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:29:25 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:30:08 [ndbd] INFO -- Node 50 disconnected in recv with errnum: 104 in state: 0
2021-02-17 12:30:34 [ndbd] INFO -- part: 1 : time to complete: 1
2021-02-17 12:30:39 [ndbd] INFO -- Node 50 disconnected in state: 0
2021-02-17 12:33:00 [ndbd] INFO -- Node 50 disconnected in state: 0

Any help would be greatly appreciated, thanks a lot!

Giuseppe Rossi

Options: ReplyQuote


Subject
Views
Written By
Posted
NDB Restore freeze
682
February 17, 2021 09:05AM
346
February 26, 2021 09:23AM
376
March 04, 2021 04:37AM
305
March 05, 2021 02:47AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.