MySQL Forums - Cluster

frm File Error (no replies)

Torsten Schumann — Thu, 16 May 2013 08:59:37 +0000

Hello,
we have folowing Cluster Database: mysql-5.1.56 ndb-7.1.14

130430 7:13:41 [ERROR] MySQL: Incorrect information in file: '\ eledb \ t_management.frm.'

how can I fix it? The database is shut down again. I desperately need help. This error is reported by both the MySQL server. It affects multiple files.

[ndbd(NDB)] 3 node(s)
id=3 @10.111.106.29 (mysql-5.1.56 ndb-7.1.14, Nodegroup: 0, Master)
id=4 @10.111.106.30 (mysql-5.1.56 ndb-7.1.14, Nodegroup: 0)
id=5 @10.111.106.31 (mysql-5.1.56 ndb-7.1.14, Nodegroup: 0)

[ndb_mgmd(MGM)] 2 node(s)
id=1 @10.111.106.27 (mysql-5.1.56 ndb-7.1.14)
id=2 @10.111.106.28 (mysql-5.1.56 ndb-7.1.14)

[mysqld(API)] 3 node(s)
id=51 @10.111.106.27 (mysql-5.1.56 ndb-7.1.14)
id=52 @10.111.106.28 (mysql-5.1.56 ndb-7.1.14)
id=53 (not connected, accepting connect from any host)

MySQL Cluster: Connection Thread Scalability (no replies)

Edwin Desouza — Tue, 14 May 2013 15:32:46 +0000

MySQL Cluster: Connection Thread Scalability
http://mikaelronstrom.blogspot.com/2013/05/mysql-cluster-73-improvements.html

truncate and REDO log (1 reply)

Alax James — Wed, 15 May 2013 06:19:24 +0000

Hello,

I am using MYSQL 5.1.61-ndb-7.1.22.

I wonder if truncate-table operaiton is logged in below log,

1) REDO log
- If yes, what would be logged, STATEMENT or row data
- If row data is logged, what is the volume? 1X or 2X table data szie?

2) Binlog (I guess yes, but logged as STATEMENT not row data)
3) Undo log (I guess no, since truncate is DDL)

Thanks!

Alax

rolling restart issue (no replies)

Zhihong He — Tue, 07 May 2013 11:29:24 +0000

Hi,all,

After we increase the DataMemory and IndexMemory and restart the 6 data nodes using cmds:
(We initially start our cluster using --nostart option )
ndb_mgm>5 restart
ndb_mgm>5 start
We got the error in ndb_mgm client as belew:
----------------
Start failed.
* 22: Error
* No contact with the process (dead ?).: Permanent error: Application error
-----------------
But if we repeat the "5 start" several times, we will finally have it successfully started.
I found from the cluster log file that the times I tried to use "5 start", if the error as above generated, then we got a WARNING like this in the cluster log file:
WARNING -- Found timedout nodeid reservation for 5, releasing it.

I am wondering why this happens? Seems the node id 5 is not available when I execute "5 start". How can we tell whether a node id is ready for a "start" command?

Thanks very much :)

The cluster log file as below:
-----------------
2013-05-07 12:41:28 [MgmtSrvr] INFO -- Node 5: Node shutdown initiated
2013-05-07 12:41:37 [MgmtSrvr] INFO -- Node 5: Suma: initiate handover for shutdown with nodes 0000000000000040 GCI: 2262903
2013-05-07 12:41:37 [MgmtSrvr] INFO -- Node 5: Suma: handover to node 6 gci: 2262903 buckets: 00000001 (2)
2013-05-07 12:41:42 [MgmtSrvr] INFO -- Node 5: Suma: handover complete
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 5: Node shutdown completed, restarting, no start.
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 4: Node 5 Disconnected
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 6: Node 5 Disconnected
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 3: Node 5 Disconnected
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 4: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 8: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 7: Communication to Node 5 closed
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 1: Node 5 Disconnected
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 6: Arbitration check won - node group majority
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: President restarts arbitration thread [state=6]
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: GCP Take over started
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: Node 6 taking over as DICT master
2013-05-07 12:41:44 [MgmtSrvr] ALERT -- Node 8: Node 5 Disconnected
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: LCP Take over started
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: ParticipatingDIH = 0000000000000000
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: ParticipatingLQH = 0000000000000000
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:44 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_From_Master_Received = 1
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: GCP Monitor: unlimited lags allowed
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: GCP Take over completed
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: kk: 2262903/10 0 0
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: LCP Take over completed (state = 4)
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: ParticipatingDIH = 0000000000000000
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: ParticipatingLQH = 0000000000000000
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=0 0000000000000000]
2013-05-07 12:41:45 [MgmtSrvr] INFO -- Node 6: m_LCP_COMPLETE_REP_From_Master_Received = 1
2013-05-07 12:41:47 [MgmtSrvr] INFO -- Node 6: Local checkpoint 4266 started. Keep GCI = 2262401 oldest restorable GCI = 2262127
2013-05-07 12:41:48 [MgmtSrvr] ALERT -- Node 7: Node 5 Disconnected
2013-05-07 12:41:55 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 9 seconds (no max lag)
2013-05-07 12:42:05 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 19 seconds (no max lag)
2013-05-07 12:42:15 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 29 seconds (no max lag)
2013-05-07 12:42:20 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:42:25 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 39 seconds (no max lag)
2013-05-07 12:42:35 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 49 seconds (no max lag)
2013-05-07 12:42:43 [MgmtSrvr] WARNING -- Node 3: Failure handling of node 5 has not completed in 1 min - state = 6
2013-05-07 12:42:43 [MgmtSrvr] WARNING -- Node 6: Failure handling of node 5 has not completed in 1 min - state = 6
2013-05-07 12:42:43 [MgmtSrvr] WARNING -- Node 4: Failure handling of node 5 has not completed in 1 min - state = 6
2013-05-07 12:42:43 [MgmtSrvr] WARNING -- Node 8: Failure handling of node 5 has not completed in 1 min - state = 6
2013-05-07 12:42:46 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_SAVE lag 60 seconds (no max lag)
2013-05-07 12:42:46 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 59 seconds (no max lag)
2013-05-07 12:42:53 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:42:56 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 69 seconds (no max lag)
2013-05-07 12:43:06 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 79 seconds (no max lag)
2013-05-07 12:43:14 [MgmtSrvr] INFO -- Node 6: Communication to Node 50 closed
2013-05-07 12:43:14 [MgmtSrvr] INFO -- Node 6: Communication to Node 51 closed
2013-05-07 12:43:14 [MgmtSrvr] ALERT -- Node 7: Node 50 Disconnected
2013-05-07 12:43:14 [MgmtSrvr] ALERT -- Node 7: Node 51 Disconnected
2013-05-07 12:43:14 [MgmtSrvr] INFO -- Node 8: Communication to Node 50 closed
2013-05-07 12:43:14 [MgmtSrvr] INFO -- Node 8: Communication to Node 51 closed
2013-05-07 12:43:14 [MgmtSrvr] ALERT -- Node 8: Node 50 Disconnected
2013-05-07 12:43:14 [MgmtSrvr] ALERT -- Node 8: Node 51 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 6: Node 50 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 6: Node 51 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 4: Communication to Node 50 closed
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 4: Communication to Node 51 closed
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 50 closed
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 51 closed
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 3: Node 50 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 4: Node 50 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 4: Node 51 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] ALERT -- Node 3: Node 51 Disconnected
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 7: Communication to Node 50 closed
2013-05-07 12:43:15 [MgmtSrvr] INFO -- Node 7: Communication to Node 51 closed
2013-05-07 12:43:16 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 89 seconds (no max lag)
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Communication to Node 50 opened
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Communication to Node 51 opened
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Node 50 Connected
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Node 51 Connected
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Node 50: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 8: Node 51: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 50 opened
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 51 opened
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Node 50 Connected
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Node 51 Connected
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Node 50: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:18 [MgmtSrvr] INFO -- Node 3: Node 51: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Communication to Node 50 opened
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Communication to Node 51 opened
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Node 50 Connected
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Node 51 Connected
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Node 50: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:19 [MgmtSrvr] INFO -- Node 4: Node 51: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:22 [MgmtSrvr] INFO -- Node 7: Communication to Node 50 opened
2013-05-07 12:43:22 [MgmtSrvr] INFO -- Node 7: Communication to Node 51 opened
2013-05-07 12:43:23 [MgmtSrvr] WARNING -- Node 7: Failure handling of node 5 has not completed in 1 min - state = 6
2013-05-07 12:43:26 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 99 seconds (no max lag)
2013-05-07 12:43:26 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:43:36 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 109 seconds (no max lag)
2013-05-07 12:43:37 [MgmtSrvr] INFO -- Node 7: Node 50 Connected
2013-05-07 12:43:37 [MgmtSrvr] INFO -- Node 7: Node 51 Connected
2013-05-07 12:43:37 [MgmtSrvr] INFO -- Node 7: Node 50: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:37 [MgmtSrvr] INFO -- Node 7: Node 51: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:43:43 [MgmtSrvr] WARNING -- Node 3: Failure handling of node 5 has not completed in 2 min - state = 6
2013-05-07 12:43:43 [MgmtSrvr] WARNING -- Node 6: Failure handling of node 5 has not completed in 2 min - state = 6
2013-05-07 12:43:44 [MgmtSrvr] WARNING -- Node 4: Failure handling of node 5 has not completed in 2 min - state = 6
2013-05-07 12:43:44 [MgmtSrvr] WARNING -- Node 8: Failure handling of node 5 has not completed in 2 min - state = 6
2013-05-07 12:43:46 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_SAVE lag 120 seconds (no max lag)
2013-05-07 12:43:46 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 119 seconds (no max lag)
2013-05-07 12:43:56 [MgmtSrvr] WARNING -- Node 6: GCP Monitor: GCP_COMMIT lag 129 seconds (no max lag)
2013-05-07 12:43:59 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 50 opened
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 51 opened
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Node 51 Connected
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Node 50 Connected
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Node 51: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:44:05 [MgmtSrvr] INFO -- Node 6: Node 50: API mysql-5.5.29 ndb-7.2.10
2013-05-07 12:44:32 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:44:40 [MgmtSrvr] WARNING -- Node 7: Failure handling of node 5 has not completed in 2 min - state = 6
2013-05-07 12:44:44 [MgmtSrvr] WARNING -- Node 6: Failure handling of node 5 has not completed in 3 min - state = 6
2013-05-07 12:44:44 [MgmtSrvr] WARNING -- Node 3: Failure handling of node 5 has not completed in 3 min - state = 6
2013-05-07 12:44:44 [MgmtSrvr] WARNING -- Node 4: Failure handling of node 5 has not completed in 3 min - state = 6
2013-05-07 12:44:45 [MgmtSrvr] WARNING -- Node 8: Failure handling of node 5 has not completed in 3 min - state = 6
2013-05-07 12:45:05 [MgmtSrvr] WARNING -- Found timedout nodeid reservation for 5, releasing it
2013-05-07 12:45:37 [MgmtSrvr] INFO -- Nodeid 5 allocated for NDB at 10.1.12.243
2013-05-07 12:45:37 [MgmtSrvr] INFO -- Node 8: Communication to Node 5 opened
2013-05-07 12:45:37 [MgmtSrvr] INFO -- Node 6: Communication to Node 5 opened
2013-05-07 12:45:37 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 opened
2013-05-07 12:45:38 [MgmtSrvr] INFO -- Node 4: Communication to Node 5 opened
2013-05-07 12:45:38 [MgmtSrvr] INFO -- Node 7: Communication to Node 5 opened
2013-05-07 12:45:38 [MgmtSrvr] INFO -- Nodeid 5 allocated for NDB at 10.1.12.243
2013-05-07 12:45:39 [MgmtSrvr] INFO -- Node 1: Node 5 Connected
2013-05-07 12:45:59 [MgmtSrvr] WARNING -- Node 6: Releasing node id allocation for node 5
2013-05-07 12:46:05 [MgmtSrvr] INFO -- Node 5: Start initiated (mysql-5.5.29 ndb-7.2.10)

Mysql Cluster Sharding (no replies)

kristopher therrien — Tue, 07 May 2013 01:06:52 +0000

Ok so i have mysql cluster setup but what i need is say table assets to be split among the nodes not replicated as the purpose of me setting up the cluster was due to lack of hard drive space so how do i get the cluster to split the table among the nodes?

I know about dbshards ect ect so please do not mention them i need a solution using just mysql and no i cant modify the software that ill be using the db

Installing cluster as Windows services (1 reply)

Nicola Gnasso — Thu, 02 May 2013 14:03:16 +0000

Hi all, my cluster works fine, but i found a strange problem while installing processes as Windows services, particullary management process. The process starts then fails due it can’t find the config.ini file. I specified the path in the installation command as below

C:\mysql-cluster\bin\ndb_mgmd.exe –install -f config.ini –configdir=C:\mysql-cluster\bin

and in second instance also

C:\mysql-cluster\bin\ndb_mgmd.exe –install -f config.ini –configdir=.\

In the Windows event log i found this message:
Error opening ‘config.ini’, error 2, No such file or directory.
Clearly is a path error, but i can’t understand where the error in my command line is.

Thanks in advance

indexes disordered in database (no replies)

esteban perez — Mon, 29 Apr 2013 21:20:35 +0000

hi !

i have a database with many tables, in each when open a table, before appear sort by date of insert in db (1,2,3,4...) now appear in disorder (3,12,1,99,2013...)

this happened after performing a migration...

before:

now:

how can solve this problem?, some idea?

thanks !

pd.. sorry for my english

MySql Cluster with InnoDB and MyISAM both (2 replies)

Dilshad Hussain — Tue, 30 Apr 2013 12:53:00 +0000

Has anyone created a MySQL Cluster with InnoDB and MyISAM tables.

We have a MySQL database having 30% tables of MyISAM and 70% tables of InnoDB. Now we want to migrate this on MySQL cluster.

If anyone has been already done something like this Please tell me the steps.

Mysqld failing after DDL on cluster (2 replies)

Ben Clewett — Wed, 24 Apr 2013 09:20:14 +0000

Cluster,

In 7.2.10, I have experienced a serious mysqld lockup. All queries where throwing:

Got error 4547 'RecordSpecification has overlapping offsets' from NDBCLUSTER

This occurred after I completed some DDL on another mysqld server connected to the same cluster:

ALTER TABLE fl_state
ADD mc_odo_change_t TIMESTAMP NULL DEFAULT NULL AFTER mc_odo;

This inserted a new field into the middle of a table with 227 columns.

This was resolved by killing and restarting the 'mysqld' node.

This table receives about 50 DML/sec from 12 mysqld nodes. Only one of them crashed.

Table schema and metrics below.

I hope this is useful to somebody.

Regards, Ben Clewett.

-----------------------------------------------------

Table Schema:

CREATE TABLE `fl_state` (
`terminal_id` varchar(32) NOT NULL,
`ip_addr` varchar(15) NOT NULL DEFAULT '',
`server_name` varchar(40) NOT NULL DEFAULT '',
`server_pid` int(11) NOT NULL DEFAULT '0',
`version_daemon` varchar(20) NOT NULL DEFAULT '',
`version_software` varchar(20) NOT NULL DEFAULT '',
`version_protocol` varchar(20) NOT NULL DEFAULT '',
`connected` timestamp NULL DEFAULT NULL,
`tally` int(11) NOT NULL DEFAULT '0',
`tacho_type` varchar(20) DEFAULT NULL,
`comp_code` varchar(8) DEFAULT NULL,
`veh_code` varchar(16) DEFAULT NULL,
`veh_rtid` int(11) DEFAULT NULL,
`mem_code` int(11) DEFAULT NULL,
`terminal_type` varchar(20) DEFAULT NULL,
`terminal_type_sub` varchar(20) DEFAULT NULL,
`terminal_type_sub_from_veh` varchar(20) DEFAULT NULL,
`service_code` varchar(16) DEFAULT NULL,
`service_sub_code` varchar(8) DEFAULT NULL,
`entity_type` enum('driven','trailer','member') NOT NULL DEFAULT 'driven',
`site_no` int(11) DEFAULT NULL,
`reset_required` tinyint(1) NOT NULL DEFAULT '0',
`tag_master_veh_rtid` int(11) DEFAULT NULL,
`tag_master_veh_rtid_prev` int(11) DEFAULT NULL,
`tag_master_terminal_id` varchar(32) DEFAULT NULL,
`tag_slave_recorded` tinyint(1) NOT NULL DEFAULT '0',
`tag_slave_count` int(11) NOT NULL DEFAULT '0',
`tag_state_pending` enum('trip','timeout','next_valid','not') NOT NULL DEFAULT 'not',
`tag_state_pending_t` timestamp NULL DEFAULT NULL,
`tag_state_pending_lat` decimal(9,6) DEFAULT NULL,
`tag_state_pending_lon` decimal(9,6) DEFAULT NULL,
`tag_state_pending_km_h` int(11) DEFAULT NULL,
`tag_state_pending_id` int(11) DEFAULT NULL,
`tag_state_timeout_start` timestamp NULL DEFAULT NULL,
`tag_state_timeout_t` int(11) DEFAULT NULL,
`tag_state_pending_trip` enum('wait_primed','wait_next','found_trip','not') NOT NULL DEFAULT 'not',
`tag_slave_last_drop` timestamp NULL DEFAULT NULL,
`tag_slave_last_hookup` timestamp NULL DEFAULT NULL,
`fl_story_rec` bigint(20) unsigned DEFAULT NULL,
`mo_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`mo_odo` decimal(11,3) DEFAULT NULL,
`mo_fuel` decimal(10,2) DEFAULT NULL,
`md_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`md_t_from` enum('gps','server') NOT NULL DEFAULT 'gps',
`md_valid_fix` tinyint(1) NOT NULL,
`md_ignition` tinyint(1) DEFAULT NULL,
`md_ignition_on_t` timestamp NULL DEFAULT NULL,
`md_ignition_off_t` timestamp NULL DEFAULT NULL,
`md_stationary` tinyint(1) DEFAULT NULL,
`md_lights` tinyint(1) DEFAULT NULL,
`md_battery` tinyint(1) DEFAULT NULL,
`md_panic_active` tinyint(1) DEFAULT NULL,
`md_panic_ack` tinyint(1) DEFAULT NULL,
`md_panic_active_t` timestamp NULL DEFAULT NULL,
`md_panic_cancel_t` timestamp NULL DEFAULT NULL,
`md_panic_ack_t` timestamp NULL DEFAULT NULL,
`md_stored_frame` tinyint(1) DEFAULT NULL,
`md_heading_change` tinyint(1) DEFAULT NULL,
`md_power_save` tinyint(1) DEFAULT NULL,
`md_gps_ariel_fault` enum('good','short','open') DEFAULT NULL,
`md_reset` tinyint(1) DEFAULT NULL,
`md_batt_volts` decimal(4,1) DEFAULT NULL,
`md_batt_percent` decimal(4,1) DEFAULT NULL,
`md_upgrade` tinyint(1) DEFAULT NULL,
`md_engine_by_volts` tinyint(1) DEFAULT NULL,
`md_flag1` tinyint(3) unsigned DEFAULT NULL,
`md_flag2` tinyint(3) unsigned DEFAULT NULL,
`md_tl_rec` bigint(20) unsigned DEFAULT NULL,
`md_server_name` varchar(40) DEFAULT NULL,
`mg_t` timestamp NULL DEFAULT NULL,
`mg_lat` decimal(9,6) DEFAULT NULL,
`mg_lon` decimal(9,6) DEFAULT NULL,
`mg_speed_n_h` decimal(5,1) DEFAULT NULL,
`mg_speed_km_h` decimal(5,1) DEFAULT NULL,
`mg_speed_m_h` decimal(5,1) DEFAULT NULL,
`mg_heading` int(11) DEFAULT NULL,
`mg_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mg_server_name` varchar(40) DEFAULT NULL,
`mz_t` timestamp NULL DEFAULT NULL,
`mz_town` varchar(40) DEFAULT NULL,
`mz_county` varchar(40) DEFAULT NULL,
`mz_country` varchar(3) DEFAULT NULL,
`mz_pcode` varchar(16) DEFAULT NULL,
`mz_gaz_rec` int(11) DEFAULT NULL,
`mc_t` timestamp NULL DEFAULT NULL,
`mc_service_distance` int(11) DEFAULT NULL,
`mc_odo` decimal(11,3) DEFAULT NULL,
`mc_fuel` decimal(10,2) DEFAULT NULL,
`mc_fuel_level` decimal(4,1) DEFAULT NULL,
`mc_temperature` decimal(5,1) DEFAULT NULL,
`mc_cat_level` decimal(4,1) DEFAULT NULL,
`mc_axle_1` decimal(6,1) DEFAULT NULL,
`mc_axle_2` decimal(6,1) DEFAULT NULL,
`mc_axle_3` decimal(6,1) DEFAULT NULL,
`mc_axle_4` decimal(6,1) DEFAULT NULL,
`mc_flag1` tinyint(3) unsigned DEFAULT NULL,
`mc_flag2` tinyint(3) unsigned DEFAULT NULL,
`mc_engine_available` tinyint(1) DEFAULT NULL,
`mc_ending_on` tinyint(1) DEFAULT NULL,
`mc_engine_on` tinyint(1) DEFAULT NULL,
`mc_pto` tinyint(1) DEFAULT NULL,
`mc_from_trip` tinyint(1) DEFAULT NULL,
`mc_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mc_server_name` varchar(40) DEFAULT NULL,
`ms_t` timestamp NULL DEFAULT NULL,
`ms_flag1` tinyint(3) unsigned DEFAULT NULL,
`ms_flag2` tinyint(3) unsigned DEFAULT NULL,
`ms_flag3` tinyint(3) unsigned DEFAULT NULL,
`ms_speed` decimal(5,1) DEFAULT NULL,
`ms_card_1` varchar(16) DEFAULT NULL,
`ms_card_2` varchar(16) DEFAULT NULL,
`ms_card_1_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_card_2_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_mem_code_1` int(11) DEFAULT NULL,
`ms_mem_code_2` int(11) DEFAULT NULL,
`ms_overspeed` tinyint(1) DEFAULT NULL,
`ms_time_state_1` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_time_state_2` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_card_1_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_card_2_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_tl_rec` bigint(20) unsigned DEFAULT NULL,
`ms_server_name` varchar(40) DEFAULT NULL,
`mt_t` timestamp NULL DEFAULT NULL,
`mt_trip_id` int(11) DEFAULT NULL,
`mt_source` enum('sdi1','sdi2','fc01') DEFAULT NULL,
`mt_trip_start` timestamp NULL DEFAULT NULL,
`mt_trip_end` timestamp NULL DEFAULT NULL,
`mt_trip_duration` int(11) DEFAULT NULL,
`mt_ignition_on_duration` int(11) DEFAULT NULL,
`mt_engine_on_duration` int(11) DEFAULT NULL,
`mt_motion_duration` int(11) DEFAULT NULL,
`mt_total_trip_duration` int(11) DEFAULT NULL,
`mt_pto_active_nomotion` int(11) DEFAULT NULL,
`mt_pto_active_motion` int(11) DEFAULT NULL,
`mt_trip_start_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_end_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_total_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_start_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_start_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_fuel_l_100km` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_100km_c` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr_c` decimal(6,1) DEFAULT NULL,
`mt_trip_driving_duration` int(11) DEFAULT NULL,
`mt_trip_working_duration` int(11) DEFAULT NULL,
`mt_trip_rest_duration` int(11) DEFAULT NULL,
`mt_trip_available_duration` int(11) DEFAULT NULL,
`mt_trip_absolute_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_engine_on_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_motion_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_speed_zone1_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone1_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_count` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_duration` int(11) DEFAULT NULL,
`mt_trip_accel_duration` int(11) DEFAULT NULL,
`mt_trip_decel_duration` int(11) DEFAULT NULL,
`mt_trip_accel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_decel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_acc_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone5_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone6_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone7_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone3_duration` int(11) DEFAULT NULL,
`mt_message_type` enum('T','U') DEFAULT NULL,
`mt_time_engine_torque_high` int(11) DEFAULT NULL,
`mt_time_in_high_gear` int(11) DEFAULT NULL,
`mt_time_high_engine_load` int(11) DEFAULT NULL,
`mt_time_low_load_high_throttle` int(11) DEFAULT NULL,
`mt_time_kick_down` int(11) DEFAULT NULL,
`mt_mean_engine_load` decimal(4,1) DEFAULT NULL,
`mt_server_name` varchar(40) DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`ignore_pto` tinyint(1) NOT NULL DEFAULT '0',
`cfg_t` timestamp NULL DEFAULT NULL,
`cfg_apn` varchar(20) DEFAULT NULL,
`cfg_user` varchar(20) DEFAULT NULL,
`cfg_password` varchar(20) DEFAULT NULL,
`cfg_ignition_off_every` varchar(20) DEFAULT NULL,
`cfg_hd_change_rpt` varchar(20) DEFAULT NULL,
`cfg_can_fitted` varchar(20) DEFAULT NULL,
`cfg_extended_can` varchar(20) DEFAULT NULL,
`cfg_trip_every` varchar(20) DEFAULT NULL,
`cfg_tacho_download_enabled` varchar(20) DEFAULT NULL,
`cfg_can_comms_protocol` varchar(20) DEFAULT NULL,
`cfg_network_sel` varchar(20) DEFAULT NULL,
`cfg_ign_rep` varchar(20) DEFAULT NULL,
`cfg_debug` varchar(20) DEFAULT NULL,
`cfg_every` varchar(20) DEFAULT NULL,
`cfg_on_battery_backup_every` varchar(20) DEFAULT NULL,
`cfg_ign_sen` varchar(20) DEFAULT NULL,
`cfg_auto_reset` varchar(20) DEFAULT NULL,
`cfg_red_pwr_mod` varchar(20) DEFAULT NULL,
`cfg_red_pwr_min` varchar(20) DEFAULT NULL,
`cfg_eng_sense_enabled` varchar(20) DEFAULT NULL,
`cfg_eng_on_offset` varchar(20) DEFAULT NULL,
`cfg_panic_enabled` varchar(20) DEFAULT NULL,
`cfg_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_log_period` varchar(20) DEFAULT NULL,
`cfg_reset_fuel` varchar(20) DEFAULT NULL,
`cfg_alarm_time` varchar(20) DEFAULT NULL,
`cnf_auto_request` enum('no','yes_new','yes_done') NOT NULL DEFAULT 'no',
`cnf_auto_request_t` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`terminal_id`),
KEY `comp_code` (`comp_code`,`veh_code`),
KEY `veh_rtid` (`veh_rtid`),
KEY `tag_master_terminal_id` (`tag_master_terminal_id`),
KEY `tag_master_veh_rtid` (`tag_master_veh_rtid`),
KEY `cnf_auto_request` (`cnf_auto_request`,`terminal_type_sub`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1

-----------------------------------------------------

Information Schema:

TABLE_CATALOG: def
TABLE_SCHEMA: ndb
TABLE_NAME: fl_state
TABLE_TYPE: BASE TABLE
ENGINE: ndbcluster
VERSION: 10
ROW_FORMAT: Dynamic
TABLE_ROWS: 6567
AVG_ROW_LENGTH: 844
DATA_LENGTH: 7405568
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 0
DATA_FREE: 0
AUTO_INCREMENT: NULL
CREATE_TIME: NULL
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: latin1_swedish_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:

. (no replies)

Tarran Benson-West — Thu, 18 Apr 2013 10:49:38 +0000

MySQL Cluster 7.3: Auto-Installer, Foreign Keys, Node.js (no replies)

Edwin Desouza — Thu, 18 Apr 2013 01:13:20 +0000

Overview:
MySQL Cluster 7.3 DMR2: Increasing Developer Flexibility and Simplicity

MySQL Cluster: Connection Thread Scalability
http://mikaelronstrom.blogspot.com/2013/05/mysql-cluster-73-improvements.html

MySQL Cluster: Auto-Installer
http://www.clusterdb.com/mysql-cluster/mysql-cluster-7-3-auto-installer/

MySQL Cluster: Foreign Keys
http://www.clusterdb.com/mysql-cluster/foreign-keys-in-mysql-cluster/

MySQL Cluster: Node.js
https://blogs.oracle.com/MySQL/entry/mysql_cluster_tutorial_nosql_api

Forced node shutdown at specific time (no replies)

Jabba Jabba — Wed, 17 Apr 2013 15:46:51 +0000

Recently we found that one of the cluster node disconnect at specific time in midnight. Checking the management node log, following error message was found:
2013-04-14 01:03:16 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
2013-04-16 01:03:09 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
2013-04-17 01:02:30 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown complete
d. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report
a bug). Temporary error, restart node'.
-------------------------------------------------------------------------

The log pattern before error happen:
2013-04-17 00:58:45 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 closed
2013-04-17 00:58:45 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-04-17 00:58:45 [MgmtSrvr] ALERT -- Node 2: Node 5 Disconnected
2013-04-17 00:58:46 [MgmtSrvr] ALERT -- Node 3: Node 5 Disconnected
2013-04-17 00:58:46 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 opened
2013-04-17 00:58:46 [MgmtSrvr] INFO -- Node 3: Node 5 Connected
2013-04-17 00:58:47 [MgmtSrvr] INFO -- Node 3: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 opened
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Node 5 Connected
2013-04-17 00:58:48 [MgmtSrvr] INFO -- Node 2: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 00:58:57 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:58:58 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:01 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:20 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:22 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:22 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 00:59:24 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 00:59:49 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:56 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 00:59:56 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:00:05 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:13 [MgmtSrvr] INFO -- Node 3: Local checkpoint 8535 completed
2013-04-17 01:00:16 [MgmtSrvr] INFO -- Node 3: Local checkpoint 8536 started. Keep GCI = 396148 oldest restorable GCI = 396103
2013-04-17 01:00:28 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:35 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:00:36 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:39 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:41 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:44 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:49 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:00:54 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:00:59 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 4
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 2: Node 5 declared dead due to missed heartbeat
2013-04-17 01:01:01 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 closed
2013-04-17 01:01:01 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 closed
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 3: Node 5 Disconnected
2013-04-17 01:01:01 [MgmtSrvr] ALERT -- Node 2: Node 5 Disconnected
2013-04-17 01:01:01 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 3
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Communication to Node 5 opened
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Node 5 Connected
2013-04-17 01:01:02 [MgmtSrvr] INFO -- Node 2: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 01:01:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 5 opened
2013-04-17 01:01:00 [MgmtSrvr] INFO -- Node 3: Node 5 Connected
2013-04-17 01:01:00 [MgmtSrvr] INFO -- Node 3: Node 5: API mysql-5.1.56 ndb-7.1.15
2013-04-17 01:01:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:04 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:05 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:08 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:08 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:09 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:12 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 60 seconds (max lag: 145s)
2013-04-17 01:01:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:17 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:20 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 1 seconds (max lag: 27)
2013-04-17 01:01:23 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:26 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:30 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:33 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 120 seconds (max lag: 145s)
2013-04-17 01:01:33 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:35 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:39 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:40 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 0 seconds (max lag: 27)
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 2
2013-04-17 01:01:43 [MgmtSrvr] WARNING -- Node 2: Node 5 missed heartbeat 3
2013-04-17 01:01:44 [MgmtSrvr] WARNING -- Node 2: Node 4 missed heartbeat 2
2013-04-17 01:01:53 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected
2013-04-17 01:01:56 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 2
2013-04-17 01:02:00 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 3
2013-04-17 01:02:04 [MgmtSrvr] WARNING -- Node 2: Node 3 missed heartbeat 4
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Node 3 declared dead due to missed heartbeat
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Communication to Node 3 closed
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Network partitioning - arbitration required
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: President restarts arbitration thread [state=7]
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Take over started
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Node 2 taking over as DICT master
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Monitor: Computed max GCP_SAVE lag to 145 seconds
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Monitor: Computed max GCP_COMMIT lag to 27 seconds
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: GCP Take over completed
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: kk: 396202/112 1 0
2013-04-17 01:02:04 [MgmtSrvr] ALERT -- Node 2: Node 3 Disconnected
2013-04-17 01:02:04 [MgmtSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=54c500233951f1d4]
2013-04-17 01:02:30 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-04-17 01:02:31 [MgmtSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 1 and 2.
2013-04-17 01:02:32 [MgmtSrvr] WARNING -- Node 2: Failure handling of node 3 has not completed in 1 min - state = 6
2013-04-17 01:02:34 [MgmtSrvr] INFO -- Node 2: LCP Take over started
------------------------------------------------------------------------

ndb_3_out.log
------------------------------------------------------------------------
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1533 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961668499 sent: 961666952 diff: 1547
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961667218 sent: 961668711 diff: -1493
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1503 ms
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1520 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961669290 sent: 961667770 diff: 1520
2013-04-17 01:02:00 [ndbd] WARNING -- timerHandlingLab now: 961667832 sent: 961669331 diff: -1499
2013-04-17 01:02:00 [ndbd] WARNING -- Time moved backwards with 1509 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved forward with 1526 ms
2013-04-17 01:02:01 [ndbd] WARNING -- timerHandlingLab now: 961670318 sent: 961668783 diff: 1535
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved backwards with 1502 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved forward with 1523 ms
2013-04-17 01:02:01 [ndbd] WARNING -- Time moved backwards with 1506 ms
2013-04-17 01:01:56 [ndbd] WARNING -- timerHandlingLab now: 961668994 sent: 961670477 diff: -1483
2013-04-17 01:01:58 [ndbd] WARNING -- Time moved forward with 1530 ms
2013-04-17 01:01:58 [ndbd] WARNING -- timerHandlingLab now: 961672212 sent: 961670671 diff: 1541
2013-04-17 01:01:58 [ndbd] WARNING -- Time moved backwards with 1512 ms
2013-04-17 01:01:58 [ndbd] WARNING -- timerHandlingLab now: 961670724 sent: 961672212 diff: -1488
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1522 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961672969 sent: 961671447 diff: 1522
2013-04-17 01:01:59 [ndbd] INFO -- findNeighbours from: 4891 old (left: 3 right: 3) new (65535 65535)
2013-04-17 01:01:59 [ndbd] INFO -- Watchdog: User time: 15145 System time: 34380
2013-04-17 01:01:59 [ndbd] WARNING -- Watchdog: Warning overslept 1614 ms, expected 100 ms.
start_resend(1, empty bucket (396202/112 396202/111) -> active
REMOVING lcp: 8536 from table: 2 frag: 0 node: 3
REMOVING lcp: 8536 from table: 2 frag: 1 node: 3
Finished with handling node-failure
execGCP_NOMORETRANS(396202/112) c_ongoing_take_over_cnt -> seize
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1462 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961671549 sent: 961672998 diff: -1449
completing gcp 396202/112 in execTAKE_OVERTCCONF
REMOVING lcp: 8536 from table: 3 frag: 0 node: 3
REMOVING lcp: 8536 from table: 3 frag: 1 node: 3
REMOVING lcp: 8536 from table: 4 frag: 0 node: 3
REMOVING lcp: 8536 from table: 4 frag: 1 node: 3
REMOVING lcp: 8536 from table: 5 frag: 0 node: 3
REMOVING lcp: 8536 from table: 5 frag: 1 node: 3
REMOVING lcp: 8536 from table: 6 frag: 0 node: 3
REMOVING lcp: 8536 from table: 6 frag: 1 node: 3
REMOVING lcp: 8536 from table: 7 frag: 0 node: 3
REMOVING lcp: 8536 from table: 7 frag: 1 node: 3
REMOVING lcp: 8536 from table: 9 frag: 0 node: 3
REMOVING lcp: 8536 from table: 9 frag: 1 node: 3
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved forward with 1537 ms
2013-04-17 01:01:59 [ndbd] WARNING -- timerHandlingLab now: 961673333 sent: 961671787 diff: 1546
2013-04-17 01:01:59 [ndbd] WARNING -- Time moved backwards with 1511 ms
------------------------------------------------------------------------
ndb_3_out.log
------------------------------------------------------------------------
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969629986 sent: 969632245 diff: -2259
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved forward with 2286 ms
2013-04-17 01:01:30 [ndbd] INFO -- Watchdog: User time: 162515 System time: 425964
2013-04-17 01:01:30 [ndbd] WARNING -- Watchdog: Warning overslept 2376 ms, expected 100 ms.
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969632301 sent: 969630006 diff: 2295
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969630050 sent: 969632302 diff: -2252
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969632557 sent: 969630263 diff: 2294
2013-04-17 01:01:30 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:30 [ndbd] WARNING -- timerHandlingLab now: 969630302 sent: 969632558 diff: -2256
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2286 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633055 sent: 969630751 diff: 2304
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969630828 sent: 969633078 diff: -2250
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2293 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633549 sent: 969631234 diff: 2315
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969631287 sent: 969633549 diff: -2262
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved forward with 2282 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969633590 sent: 969631297 diff: 2293
2013-04-17 01:01:31 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:31 [ndbd] WARNING -- timerHandlingLab now: 969631406 sent: 969633658 diff: -2252
2013-04-17 01:01:33 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:33 [ndbd] WARNING -- Time moved backwards with 1283 ms
2013-04-17 01:01:34 [ndbd] WARNING -- timerHandlingLab now: 969636046 sent: 969633744 diff: 2302
2013-04-17 01:01:34 [ndbd] WARNING -- Time moved forward with 2292 ms
2013-04-17 01:01:34 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:34 [ndbd] WARNING -- timerHandlingLab now: 969633790 sent: 969636046 diff: -2256
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969639889 sent: 969637605 diff: 2284
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved backwards with 2270 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969637643 sent: 969639890 diff: -2247
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969640512 sent: 969638217 diff: 2295
2013-04-17 01:01:38 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:38 [ndbd] WARNING -- timerHandlingLab now: 969638251 sent: 969640512 diff: -2261
2013-04-17 01:01:40 [ndbd] WARNING -- timerHandlingLab now: 969642658 sent: 969640363 diff: 2295
2013-04-17 01:01:41 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=99
2013-04-17 01:01:41 [ndbd] INFO -- Watchdog: User time: 162516 System time: 425966
2013-04-17 01:01:41 [ndbd] WARNING -- timerHandlingLab now: 969640826 sent: 969642658 diff: -1832
2013-04-17 01:01:41 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425969
2013-04-17 01:01:41 [ndbd] WARNING -- Watchdog: Warning overslept 427 ms, expected 100 ms.
2013-04-17 01:01:42 [ndbd] WARNING -- Time moved forward with 2284 ms
2013-04-17 01:01:42 [ndbd] WARNING -- timerHandlingLab now: 969644280 sent: 969641992 diff: 2288
2013-04-17 01:01:42 [ndbd] WARNING -- Time moved backwards with 2272 ms
2013-04-17 01:01:42 [ndbd] WARNING -- timerHandlingLab now: 969642029 sent: 969644281 diff: -2252
2013-04-17 01:01:43 [ndbd] WARNING -- Time moved forward with 2283 ms
2013-04-17 01:01:43 [ndbd] WARNING -- timerHandlingLab now: 969645600 sent: 969643311 diff: 2289
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 1450 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 13 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
Detected GCP stop(3)...sending kill to [SignalCounter: m_count=1 0000000000000008]
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 0 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 0 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
file[0] status: 2 type: 1 reqStatus: 0 file1: 2 1 4
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 0 m_gcp_save.m_max_lag: 1450
m_micro_gcp.m_counter: 0 m_micro_gcp.m_max_lag: 271
m_gcp_save.m_state: 3
m_gcp_save.m_master.m_state: 3
m_micro_gcp.m_state: 1
m_micro_gcp.m_master.m_state: 1
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=1 0000000000000008]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_SUB_GCP_COMPLETE_REP_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 3 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 3 nextWord: 0
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=200
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=299
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=401
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=501
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=601
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=701
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=802
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=903
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:44 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1002
2013-04-17 01:01:44 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1103
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1208
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1319
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1420
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1520
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1619
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1719
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1820
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:45 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=1920
2013-04-17 01:01:45 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2021
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2121
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2220
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2320
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2420
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2520
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2621
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2720
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2820
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:46 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=2920
2013-04-17 01:01:46 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3020
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3121
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3220
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3321
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3422
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3521
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3621
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3722
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3822
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:47 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=3922
2013-04-17 01:01:47 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4022
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4122
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4222
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4322
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4422
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4521
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4621
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4722
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4823
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:48 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=4923
2013-04-17 01:01:48 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5023
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5123
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5223
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425971
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5323
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5423
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5524
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5623
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5724
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5823
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:49 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=5924
2013-04-17 01:01:49 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=6023
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=100
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=200
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=299
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=399
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=499
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=600
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=700
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:50 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=800
2013-04-17 01:01:50 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=899
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=999
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Print Job Buffers at crash elapsed=1099
2013-04-17 01:01:51 [ndbd] INFO -- Watchdog: User time: 162518 System time: 425972
2013-04-17 01:01:51 [ndbd] INFO -- Node 3 killed this node because GCP stop was detected
2013-04-17 01:01:51 [ndbd] INFO -- NDBCNTR (Line: 276) 0x00000002
2013-04-17 01:01:51 [ndbd] INFO -- Error handler shutting down system
2013-04-17 01:01:53 [ndbd] INFO -- Error handler shutdown completed - exiting
2013-04-17 01:02:29 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
-------------------------------------------------------------------------

The following is the management node config.ini:
-------------------------------------------------------------------------
[ndbd default]
NoOfReplicas=2 # Number of replicas
DataMemory=2048M
IndexMemory=20M
RedoBuffer=512M
TimeBetweenLocalCheckpoints=0
NoOfFragmentLogFiles=64
HeartbeatIntervalDbDb=5000
# TCP/IP options:
[tcp default]

[ndb_mgmd]
hostname=10.0.104.132 # Hostname or IP address of management node
datadir=/var/lib/mysql-cluster # Directory for management node log files
# Options for data node "A":
[ndbd]
hostname=10.0.104.123 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files
[ndbd]
hostname=10.0.104.122 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's data files

[mysqld]
[mysqld]
-------------------------------------------------------------------------

Checked that there is no schedule job execute at that moment. The network status is good during incident period.

Is there any suggestion to troubleshoot the problem? Thanks

ndbd over 100% (2 replies)

Paolo de marco — Fri, 26 Apr 2013 12:39:38 +0000

Hello, I'm new of cluster.
I have a cluster of two servers.
Server 1: management, ndb and mysql
Server 2: ndb e mysql
The cluster works good after same change to default value.
A customer has a website on the two servers.
Sometimes, after/during big queries, nbd comes up tp 120% of the cpu and everithing is slow.
If i kill Server2 everything works better, but sometime the same thing happen also on Server1.
I cannot understand why. ANy ideas?
Thanks

Indexes used in Cluster (4 replies)

Ben Clewett — Fri, 26 Apr 2013 20:37:53 +0000

Dear Cluster,

I am trying to understand what index types are available on Cluster for the user to make use of. But I have not having much luck finding accurate and recent documentation. I wonder whether any members know?

I believe that Cluster used to use a hash-index for both Primary and Secondary indexes?

But now is it possible to use a T-Tree index on Primary and/or Secondary?

If this is the case, what is the syntax? I have trying "USING {BTREE | HASH}" on a "CREATE TABLE" without conclusive results.

Of if anybody has a good URL on the subject, that would be very useful.

Many thanks!!

Ben.

SQL node can't join cluster, claims no storage nodes connected (no replies)

Chad Austin — Tue, 16 Apr 2013 06:34:24 +0000

I'm trying to configure my first MySQL cluster using the documentation at dev.mysql.com. I'm using the latest RPMs on CentOS 6. I have my management and NDB nodes up, but I'm running into this error when I start mysqld on the API node:

[Note] NDB: NodeID is 4, management server '172.16.186.137:1186'
[Note] NDB[0]: NodeID: 4, no storage nodes connected (timed out)
[Warning] NDB: server id set to zero - changes logged to bin log with server id zero will be logged with another server id by slave mysqlds

It seems that most people experiencing this error were having their connections blocked by a firewall, but my hosts are non-firewalled and are able to talk to each other on 1186 and 2202 (where applicable) without problem.

Someone on irc suggested I try running "ndb_show_tables" from the API node, which I was able to execute successfully. They were unable to explain why this would work, but mysqld wouldn't be able to start normally. Here's log output and confs:

## ndb_mgm output:

[ndbd(NDB)] 2 node(s)
id=2 @172.16.186.134 (mysql-5.5.30 ndb-7.2.12, Nodegroup: 0, Master)
id=3 @172.16.186.135 (mysql-5.5.30 ndb-7.2.12, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @172.16.186.137 (mysql-5.5.30 ndb-7.2.12)

[mysqld(API)] 1 node(s)
id=4 (not connected, accepting connect from 172.16.186.136)

## MGM log:

[MgmtSrvr] INFO -- Nodeid 4 allocated for API at 172.16.186.136
[MgmtSrvr] INFO -- Node 4: mysqld --server-id=0
[MgmtSrvr] WARNING -- Node 2: Releasing node id allocation for node 4

## The API node's my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
ndbcluster

[mysql_cluster]
ndb-connectstring=172.16.186.137

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

mysqld still crashing (3 replies)

Ben Clewett — Fri, 12 Apr 2013 16:19:01 +0000

Hi MySQL/Oracle,

A while ago I posted that my mysqld was crashing about once a week. This is still the case. I wonder whether any member may have an idea why?

Using 7.2.10 on Linux 3.1.10-1.9.

Only a single NDB table at abut 50 DML/sec from various clients.
Table with 227 columns, 5 keys, 6567 rows, and average row length of 884 bytes.

Table schema, information schema report and MySQL crash stack trace to follow.

Because Cluster keep crashing, it's giving cluster a bad name in this company. Which is a shame, as I want this to work. If anybody has any ideas, please let me know.

Regards, Ben.

-----------------------------------------------------

Crash log:

05:54:07 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=5242880
max_used_connections=37
max_threads=200
thread_count=34
connection_count=34
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2058485 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x6ca3750
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fd9081b9e98 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x81cb65]
/usr/sbin/mysqld(handle_fatal_signal+0x403)[0x6e8f33]
/lib64/libpthread.so.0(+0xfd00)[0x7fd99e25fd00]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x158)[0x6ecc08]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x1d)[0x69ad8d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4)[0x5ee4b4]
/usr/sbin/mysqld(_ZN21Execute_sql_statement19execute_server_codeEP3THD+0xf3)[0x5fea43]
/usr/sbin/mysqld(_ZN18Prepared_statement23execute_server_runnableEP15Server_runnable+0x159)[0x5fe2b9]
/usr/sbin/mysqld(_ZN13Ed_connection14execute_directEP15Server_runnable+0xc9)[0x5ffdf9]
/usr/sbin/mysqld(_ZN13Ed_connection14execute_directE19st_mysql_lex_string+0x2d)[0x5ffecd]
/usr/sbin/mysqld(_ZN20Ndb_local_connection13execute_queryE19st_mysql_lex_stringPKjPK10Suppressor+0x6b)[0x9db87b]
/usr/sbin/mysqld(_ZN20Ndb_local_connection17execute_query_isoE19st_mysql_lex_stringPKjPK10Suppressor+0xc8)[0x9dba48]
/usr/sbin/mysqld(_ZN20Ndb_local_connection11delete_rowsEPKcmS1_mbz+0x182)[0x9dc002]
/usr/sbin/mysqld[0x9cb70f]
/usr/sbin/mysqld[0x6ea2bf]
/usr/sbin/mysqld(_Z26ha_binlog_index_purge_fileP3THDPKc+0x24)[0x6ea304]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG17purge_index_entryEP3THDPyb+0x3c9)[0x77dc39]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG10purge_logsEPKcbbbPy+0x1a9)[0x77f189]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG22purge_logs_before_dateEl+0x320)[0x77fda0]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG5writeEP3THDP11st_io_cacheP9Log_eventb+0x418)[0x783b58]
/usr/sbin/mysqld[0x78443c]
/usr/sbin/mysqld[0x7845e3]
/usr/sbin/mysqld(_Z19ha_commit_one_phaseP3THDb+0x87)[0x6ec9e7]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x1d1)[0x6ecc81]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x1d)[0x69ad8d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c4)[0x5ee4b4]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x18a)[0x5f285a]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1331)[0x5f4411]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x167)[0x68d657]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x68d6c4]
/lib64/libpthread.so.0(+0x7f05)[0x7fd99e257f05]
/lib64/libc.so.6(clone+0x6d)[0x7fd99d4fb10d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fd884071bf0): is an invalid pointer
Connection ID (thread ID): 24
Status: NOT_KILLED

-----------------------------------------------------

Table Schema:

CREATE TABLE `fl_state` (
`terminal_id` varchar(32) NOT NULL,
`ip_addr` varchar(15) NOT NULL DEFAULT '',
`server_name` varchar(40) NOT NULL DEFAULT '',
`server_pid` int(11) NOT NULL DEFAULT '0',
`version_daemon` varchar(20) NOT NULL DEFAULT '',
`version_software` varchar(20) NOT NULL DEFAULT '',
`version_protocol` varchar(20) NOT NULL DEFAULT '',
`connected` timestamp NULL DEFAULT NULL,
`tally` int(11) NOT NULL DEFAULT '0',
`tacho_type` varchar(20) DEFAULT NULL,
`comp_code` varchar(8) DEFAULT NULL,
`veh_code` varchar(16) DEFAULT NULL,
`veh_rtid` int(11) DEFAULT NULL,
`mem_code` int(11) DEFAULT NULL,
`terminal_type` varchar(20) DEFAULT NULL,
`terminal_type_sub` varchar(20) DEFAULT NULL,
`terminal_type_sub_from_veh` varchar(20) DEFAULT NULL,
`service_code` varchar(16) DEFAULT NULL,
`service_sub_code` varchar(8) DEFAULT NULL,
`entity_type` enum('driven','trailer','member') NOT NULL DEFAULT 'driven',
`site_no` int(11) DEFAULT NULL,
`reset_required` tinyint(1) NOT NULL DEFAULT '0',
`tag_master_veh_rtid` int(11) DEFAULT NULL,
`tag_master_veh_rtid_prev` int(11) DEFAULT NULL,
`tag_master_terminal_id` varchar(32) DEFAULT NULL,
`tag_slave_recorded` tinyint(1) NOT NULL DEFAULT '0',
`tag_slave_count` int(11) NOT NULL DEFAULT '0',
`tag_state_pending` enum('trip','timeout','next_valid','not') NOT NULL DEFAULT 'not',
`tag_state_pending_t` timestamp NULL DEFAULT NULL,
`tag_state_pending_lat` decimal(9,6) DEFAULT NULL,
`tag_state_pending_lon` decimal(9,6) DEFAULT NULL,
`tag_state_pending_km_h` int(11) DEFAULT NULL,
`tag_state_pending_id` int(11) DEFAULT NULL,
`tag_state_timeout_start` timestamp NULL DEFAULT NULL,
`tag_state_timeout_t` int(11) DEFAULT NULL,
`tag_state_pending_trip` enum('wait_primed','wait_next','found_trip','not') NOT NULL DEFAULT 'not',
`tag_slave_last_drop` timestamp NULL DEFAULT NULL,
`tag_slave_last_hookup` timestamp NULL DEFAULT NULL,
`fl_story_rec` bigint(20) unsigned DEFAULT NULL,
`mo_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`mo_odo` decimal(11,3) DEFAULT NULL,
`mo_fuel` decimal(10,2) DEFAULT NULL,
`md_t` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`md_t_from` enum('gps','server') NOT NULL DEFAULT 'gps',
`md_valid_fix` tinyint(1) NOT NULL,
`md_ignition` tinyint(1) DEFAULT NULL,
`md_ignition_on_t` timestamp NULL DEFAULT NULL,
`md_ignition_off_t` timestamp NULL DEFAULT NULL,
`md_stationary` tinyint(1) DEFAULT NULL,
`md_lights` tinyint(1) DEFAULT NULL,
`md_battery` tinyint(1) DEFAULT NULL,
`md_panic_active` tinyint(1) DEFAULT NULL,
`md_panic_ack` tinyint(1) DEFAULT NULL,
`md_panic_active_t` timestamp NULL DEFAULT NULL,
`md_panic_cancel_t` timestamp NULL DEFAULT NULL,
`md_panic_ack_t` timestamp NULL DEFAULT NULL,
`md_stored_frame` tinyint(1) DEFAULT NULL,
`md_heading_change` tinyint(1) DEFAULT NULL,
`md_power_save` tinyint(1) DEFAULT NULL,
`md_gps_ariel_fault` enum('good','short','open') DEFAULT NULL,
`md_reset` tinyint(1) DEFAULT NULL,
`md_batt_volts` decimal(4,1) DEFAULT NULL,
`md_batt_percent` decimal(4,1) DEFAULT NULL,
`md_upgrade` tinyint(1) DEFAULT NULL,
`md_engine_by_volts` tinyint(1) DEFAULT NULL,
`md_flag1` tinyint(3) unsigned DEFAULT NULL,
`md_flag2` tinyint(3) unsigned DEFAULT NULL,
`md_tl_rec` bigint(20) unsigned DEFAULT NULL,
`md_server_name` varchar(40) DEFAULT NULL,
`mg_t` timestamp NULL DEFAULT NULL,
`mg_lat` decimal(9,6) DEFAULT NULL,
`mg_lon` decimal(9,6) DEFAULT NULL,
`mg_speed_n_h` decimal(5,1) DEFAULT NULL,
`mg_speed_km_h` decimal(5,1) DEFAULT NULL,
`mg_speed_m_h` decimal(5,1) DEFAULT NULL,
`mg_heading` int(11) DEFAULT NULL,
`mg_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mg_server_name` varchar(40) DEFAULT NULL,
`mz_t` timestamp NULL DEFAULT NULL,
`mz_town` varchar(40) DEFAULT NULL,
`mz_county` varchar(40) DEFAULT NULL,
`mz_country` varchar(3) DEFAULT NULL,
`mz_pcode` varchar(16) DEFAULT NULL,
`mz_gaz_rec` int(11) DEFAULT NULL,
`mc_t` timestamp NULL DEFAULT NULL,
`mc_service_distance` int(11) DEFAULT NULL,
`mc_odo` decimal(11,3) DEFAULT NULL,
`mc_fuel` decimal(10,2) DEFAULT NULL,
`mc_fuel_level` decimal(4,1) DEFAULT NULL,
`mc_temperature` decimal(5,1) DEFAULT NULL,
`mc_cat_level` decimal(4,1) DEFAULT NULL,
`mc_axle_1` decimal(6,1) DEFAULT NULL,
`mc_axle_2` decimal(6,1) DEFAULT NULL,
`mc_axle_3` decimal(6,1) DEFAULT NULL,
`mc_axle_4` decimal(6,1) DEFAULT NULL,
`mc_flag1` tinyint(3) unsigned DEFAULT NULL,
`mc_flag2` tinyint(3) unsigned DEFAULT NULL,
`mc_engine_available` tinyint(1) DEFAULT NULL,
`mc_ending_on` tinyint(1) DEFAULT NULL,
`mc_engine_on` tinyint(1) DEFAULT NULL,
`mc_pto` tinyint(1) DEFAULT NULL,
`mc_from_trip` tinyint(1) DEFAULT NULL,
`mc_tl_rec` bigint(20) unsigned DEFAULT NULL,
`mc_server_name` varchar(40) DEFAULT NULL,
`ms_t` timestamp NULL DEFAULT NULL,
`ms_flag1` tinyint(3) unsigned DEFAULT NULL,
`ms_flag2` tinyint(3) unsigned DEFAULT NULL,
`ms_flag3` tinyint(3) unsigned DEFAULT NULL,
`ms_speed` decimal(5,1) DEFAULT NULL,
`ms_card_1` varchar(16) DEFAULT NULL,
`ms_card_2` varchar(16) DEFAULT NULL,
`ms_card_1_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_card_2_null` enum('not_null','unsupported_device','no_k_line','card_not_inserted','driver_unknown') DEFAULT NULL,
`ms_mem_code_1` int(11) DEFAULT NULL,
`ms_mem_code_2` int(11) DEFAULT NULL,
`ms_overspeed` tinyint(1) DEFAULT NULL,
`ms_time_state_1` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_time_state_2` enum('normal','15m_before_430h','430h','15m_before_900h','900h','15m_before_xxxh','xxxh') DEFAULT NULL,
`ms_card_1_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_card_2_mode` enum('w','r','d','a') DEFAULT NULL,
`ms_tl_rec` bigint(20) unsigned DEFAULT NULL,
`ms_server_name` varchar(40) DEFAULT NULL,
`mt_t` timestamp NULL DEFAULT NULL,
`mt_trip_id` int(11) DEFAULT NULL,
`mt_source` enum('sdi1','sdi2','fc01') DEFAULT NULL,
`mt_trip_start` timestamp NULL DEFAULT NULL,
`mt_trip_end` timestamp NULL DEFAULT NULL,
`mt_trip_duration` int(11) DEFAULT NULL,
`mt_ignition_on_duration` int(11) DEFAULT NULL,
`mt_engine_on_duration` int(11) DEFAULT NULL,
`mt_motion_duration` int(11) DEFAULT NULL,
`mt_total_trip_duration` int(11) DEFAULT NULL,
`mt_pto_active_nomotion` int(11) DEFAULT NULL,
`mt_pto_active_motion` int(11) DEFAULT NULL,
`mt_trip_start_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_end_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_total_odometer` decimal(11,3) DEFAULT NULL,
`mt_trip_start_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_start_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_end_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_total_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel` decimal(10,1) DEFAULT NULL,
`mt_trip_idle_fuel_c` decimal(10,1) DEFAULT NULL,
`mt_trip_fuel_l_100km` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_100km_c` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr` decimal(6,1) DEFAULT NULL,
`mt_trip_fuel_l_hr_c` decimal(6,1) DEFAULT NULL,
`mt_trip_driving_duration` int(11) DEFAULT NULL,
`mt_trip_working_duration` int(11) DEFAULT NULL,
`mt_trip_rest_duration` int(11) DEFAULT NULL,
`mt_trip_available_duration` int(11) DEFAULT NULL,
`mt_trip_absolute_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_engine_on_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_motion_km_h` decimal(5,2) DEFAULT NULL,
`mt_trip_speed_zone1_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_off_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone1_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone2_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone3_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_zone4_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_speed_total_cc_on_duration` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_count` int(11) DEFAULT NULL,
`mt_trip_brake_pedal_duration` int(11) DEFAULT NULL,
`mt_trip_accel_duration` int(11) DEFAULT NULL,
`mt_trip_decel_duration` int(11) DEFAULT NULL,
`mt_trip_accel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_decel_max_ms2` decimal(5,2) DEFAULT NULL,
`mt_trip_acc_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone5_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone6_duration` int(11) DEFAULT NULL,
`mt_trip_acc_zone7_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone3_duration` int(11) DEFAULT NULL,
`mt_trip_throttle_zone4_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone1_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone2_duration` int(11) DEFAULT NULL,
`mt_trip_engine_zone3_duration` int(11) DEFAULT NULL,
`mt_message_type` enum('T','U') DEFAULT NULL,
`mt_time_engine_torque_high` int(11) DEFAULT NULL,
`mt_time_in_high_gear` int(11) DEFAULT NULL,
`mt_time_high_engine_load` int(11) DEFAULT NULL,
`mt_time_low_load_high_throttle` int(11) DEFAULT NULL,
`mt_time_kick_down` int(11) DEFAULT NULL,
`mt_mean_engine_load` decimal(4,1) DEFAULT NULL,
`mt_server_name` varchar(40) DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT '2000-01-01 00:00:00',
`updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`ignore_pto` tinyint(1) NOT NULL DEFAULT '0',
`cfg_t` timestamp NULL DEFAULT NULL,
`cfg_apn` varchar(20) DEFAULT NULL,
`cfg_user` varchar(20) DEFAULT NULL,
`cfg_password` varchar(20) DEFAULT NULL,
`cfg_ignition_off_every` varchar(20) DEFAULT NULL,
`cfg_hd_change_rpt` varchar(20) DEFAULT NULL,
`cfg_can_fitted` varchar(20) DEFAULT NULL,
`cfg_extended_can` varchar(20) DEFAULT NULL,
`cfg_trip_every` varchar(20) DEFAULT NULL,
`cfg_tacho_download_enabled` varchar(20) DEFAULT NULL,
`cfg_can_comms_protocol` varchar(20) DEFAULT NULL,
`cfg_network_sel` varchar(20) DEFAULT NULL,
`cfg_ign_rep` varchar(20) DEFAULT NULL,
`cfg_debug` varchar(20) DEFAULT NULL,
`cfg_every` varchar(20) DEFAULT NULL,
`cfg_on_battery_backup_every` varchar(20) DEFAULT NULL,
`cfg_ign_sen` varchar(20) DEFAULT NULL,
`cfg_auto_reset` varchar(20) DEFAULT NULL,
`cfg_red_pwr_mod` varchar(20) DEFAULT NULL,
`cfg_red_pwr_min` varchar(20) DEFAULT NULL,
`cfg_eng_sense_enabled` varchar(20) DEFAULT NULL,
`cfg_eng_on_offset` varchar(20) DEFAULT NULL,
`cfg_panic_enabled` varchar(20) DEFAULT NULL,
`cfg_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_logging` varchar(20) DEFAULT NULL,
`cfg_rmt_log_period` varchar(20) DEFAULT NULL,
`cfg_reset_fuel` varchar(20) DEFAULT NULL,
`cfg_alarm_time` varchar(20) DEFAULT NULL,
`cnf_auto_request` enum('no','yes_new','yes_done') NOT NULL DEFAULT 'no',
`cnf_auto_request_t` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`terminal_id`),
KEY `comp_code` (`comp_code`,`veh_code`),
KEY `veh_rtid` (`veh_rtid`),
KEY `tag_master_terminal_id` (`tag_master_terminal_id`),
KEY `tag_master_veh_rtid` (`tag_master_veh_rtid`),
KEY `cnf_auto_request` (`cnf_auto_request`,`terminal_type_sub`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1

-----------------------------------------------------

Information Schema:

TABLE_CATALOG: def
TABLE_SCHEMA: ndb
TABLE_NAME: fl_state
TABLE_TYPE: BASE TABLE
ENGINE: ndbcluster
VERSION: 10
ROW_FORMAT: Dynamic
TABLE_ROWS: 6567
AVG_ROW_LENGTH: 844
DATA_LENGTH: 7405568
MAX_DATA_LENGTH: 0
INDEX_LENGTH: 0
DATA_FREE: 0
AUTO_INCREMENT: NULL
CREATE_TIME: NULL
UPDATE_TIME: NULL
CHECK_TIME: NULL
TABLE_COLLATION: latin1_swedish_ci
CHECKSUM: NULL
CREATE_OPTIONS:
TABLE_COMMENT:

NDB node memory usage different (1 reply)

parkdonghyuk park — Wed, 24 Apr 2013 18:39:30 +0000

hi~

i wonder

2 NDB node config.ini index memory + data memory set 45G

so NDB3 NDB4 memory usage different

why different ?

NDB 3
[mysql@mysql ]$ free -m
total used free shared buffers cached
Mem: 64454 62768 1686 0 168 25767
-/+ buffers/cache: 36832 27622
Swap: 32765 25566 7198

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23551 mysql 15 0 59.3g 34g 3832 S 26.2 54.4 27:27.12 ndbmtd

NDB 4

[mysql@mysql]$ free -m
total used free shared buffers cached
Mem: 64454 64285 169 0 94 17796
-/+ buffers/cache: 46394 18060
Swap: 32765 13583 19181
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15113 mysql 15 0 59.2g 44g 3304 S 25.2 70.6 18:46.54 ndbmtd

General error: 1297 Got temporary error 899 'Rowid already allocated' from NDBCLUSTER (no replies)

Mateusz Kamola — Wed, 10 Apr 2013 13:01:49 +0000

Hello.

I've got problem with cluster getting following error:
General error: 1297 Got temporary error 899 'Rowid already allocated' from NDBCLUSTER

I'm running cluster (ndb 7.2.10) with two machines - one data node and one mysql node on each machine, with diskless turned on.
The problem starts occuring under heavy load - few hundreds inserts and updates per second on each machine. What is strange, after it starts occuring it doesn't stop even when load gets very small.
Only inserts cause this problem.
Queries sometimes cause this error and sometimes don't (ie. every second 50% of queries fail with rowid error).
Error stops appearing for inserts into table that was just truncated.
Some tables have primary key with auto_increment, others don't - insert into all can cause error.
Insert into biggest table (about 100M inserts a day) and few other tables don't cause error (didn't find any correlation yet).
Error stops occuring after restarting/stopping one of data nodes.

I've got no idea how to approach this problem - any suggestions? Or maybe it's a known problem which can be easily fixed? :)

mysql cluster .NET connector (4 replies)

Gregorio Barberio — Mon, 15 Apr 2013 17:51:06 +0000

Hi,I'm trying mysql cluster to use in a project for our customer.

The configuration I setted up is the following:
Hardware S.O. equipment
Server1 - S.O. Windows 2008 R2 64bit
Server2 - S.O. Windows 2008 R2 64bit
Client1 - S.O. Windows 7 64bit
Client2 - S.O. Windows 7 64bit

UserClient2 - S.O. Windows 7 64bit
UserClient3 - S.O. Windows 7 64bit
UserClient4 - S.O. Windows 7 64bit
UserClient5 - S.O. Windows 7 64bit
UserClientXX - S.O. Windows 7 64bit (where XX is less than 15)

My Sql Cluster Configuration
Server1 - act as cluster data node 1
Server2 - act as cluster data node 2
Server1 - act as mysql node 1 (from now MYSQL-D1)
Server2 - act as mysql node 2 (from now MYSQL-D2)

Client1 - act as mysql cluster management node 1
Client2 - act as mysql cluster management node 2

Client application configuration
The following clients (from now CLIENTAPP):
Client1, Client2, UserClient2, .., UserClientXX
have a Microsoft .Net application that interact with mysql cluster database.

After a while the My sql cluster configuration appears to work correctly and any of CLIENTAPP it's able to connect to MYSQL-D1 or MYSQL-D2 (this is possibile by changing the connection string of .NET client application and pointing to IP of MYSQL-D1 or to the IP of MYSQL-D2).

In a perfect world CLIENTAPP automatically it's able to connect to the right server (MYSQL-D1 or MYSQL-D2), pratically:
- if MYSQL-D1 it's down CLIENTAPP connect to MYSQL-D2
- if MYSQL-D2 it's down CLIENTAPP connect to MYSQL-D1

How .NET client application can interact with mysql cluster?

The best solutions I have found for my scenario are the followings:
1. use mysql proxy but this is in a alpha stage and I can't use it in production enviroment
2. install cluster data node on each CLIENTAPP
3. use a connection string that support multiple servers (I don't know if it's supported by mysql .NET connector)
4. rewrite my application to interact with both MYSQL-DX node (what I want to avoid)

Is there a right solutions? Which one is the best one?

Thank you

Need info on cluster setup.. (no replies)

swayam kumar — Tue, 09 Apr 2013 16:32:16 +0000

Hi,
I am new to the mysql. I am trying to setup mysql cluster. Could you please tell me whether I need to download ONLY "mysql-cluster-gpl-7.2.10-win32" or I should download the MySQL installer "mysql-installer-community-5.6.10.1.msi" too? I don't find any docs which will tell step by step guide to install mysql cluster in windows 7 or even any flavor of windows. All that I got is install mysql (*.msi) on windows.
I followed the steps that are mentioned in the doc "MySQL Cluster Quick Start Guide – Windows" but not able to connect as I don't know what is the root password (mysql -h 127.0.0.1 -P 3306 -u root -p) I should provide as no where it asked me to setup any root password. So I am bit doubt whether I should install mysql installer prior to cluster setup.
Could you please tell me if any docs exist so that I can follow the cluster setup?
Regards,
Saroj

After optimize table statement my data usage increased Mysql Cluster (no replies)

Anshul Jain — Tue, 09 Apr 2013 14:34:02 +0000

I deleted 500000 rows in my mysql cluster table having 12468000 rows and then i tried to OPTIMIZE TABLE so that all the freed up data memory can be used by another table also, but after optimizing table the data usage increased instead of decreasing it. Please explain how OPTIMIZE TABLE works and why data usage increased.

DATA USAGE BEFORE OPTIMIZING TABLE:

Node 2: Data usage is 71%(7348 32K pages of total 10240)
Node 2: Index usage is 4%(4382 8K pages of total 102432)
Node 3: Data usage is 69%(7167 32K pages of total 10240)
Node 3: Index usage is 4%(4291 8K pages of total 102432)
Node 4: Data usage is 70%(7229 32K pages of total 10240)
Node 4: Index usage is 4%(4312 8K pages of total 102432)
Node 5: Data usage is 67%(6899 32K pages of total 10240)
Node 5: Index usage is 4%(4201 8K pages of total 102432)

DATA USAGE AFTER OPTIMIZING TABLE:

Node 2: Data usage is 74%(7589 32K pages of total 10240)
Node 2: Index usage is 4%(4382 8K pages of total 102432)
Node 3: Data usage is 72%(7392 32K pages of total 10240)
Node 3: Index usage is 4%(4291 8K pages of total 102432)
Node 4: Data usage is 72%(7461 32K pages of total 10240)
Node 4: Index usage is 4%(4312 8K pages of total 102432)
Node 5: Data usage is 69%(7117 32K pages of total 10240)
Node 5: Index usage is 4%(4201 8K pages of total 102432)

Re-use disk data extents after delete Mysql Cluster (no replies)

Anshul Jain — Tue, 09 Apr 2013 14:32:15 +0000

In my mysql cluster i have 1 million rows and my disk data tables are using all the disk usage available to them ;i.e; free extents are 0 . Now i deleted 50K rows from ap_perf_tbl but still it show free extents as 0 but i am able to insert into that table . But when i tried to insert into another table associated with that tablespace it says TABLE FULL error. I am not able to use those freed up extents for another table . Is there any way to use those freed extents from one table to be used by another table ?

Why data usage increases when deleting rows Mysql Cluster? (no replies)

Anshul Jain — Tue, 09 Apr 2013 14:31:27 +0000

My mysql cluster data nodes data usage increased to 95%. Then i tried to delete 500000 rows and my data usage of nodes started increasing initially and reached to 99% and above and TABLE FULL error is encountered. Please can anyone tell me why data usage of existing nodes increased when deleting rows? What is the internal working of mysql cluster when deleting rows?

My MaxNoOfConcurrentOperations parameter in config.ini 600000 and it is not due to this parameter.

Two data nodes on the same machine - what is the point? (2 replies)

Michał Herman — Mon, 08 Apr 2013 09:00:02 +0000

I am new to MySql Cluster.
I have evaluated MySQL Cluster Quick Start Guide.
In this guide in "Run" section are commands to run two data nodes on the same host. What is the point of such operation?
If I want run cluster for test purposes (do not need failure safeness) with one host only do I need two data nodes or only one data node can work?

SQL Node not synchronizing (3 replies)

LAL CHAND Prasad — Thu, 18 Apr 2013 15:12:13 +0000

I have successfully installed the Cluster with 5 nodes (2 Data, 1 Mgmt and 2 SQL Nodes). While testing the cluster I have hit on one scenario where I am stuck and cannot make it to work. Here is the scenario:

While all nodes in the cluster are working as a part of my test I shut down one of the SQL A node and while this node is offline, I drop a database which is part of the cluster databases on SQL Node B. When I bring up the offline SQL Node A and rejoin to the cluster the dropped database still shows up. It should sync with the old cluster databases meaning the dropped database when the SQL Node A was offline should be removed from the cluster and should not show up on SQL Node A. This is a real scenario that can happen.

Also, I am searching for MySQL Cluster Test document which describes these scenarios and cannot seem to find it.

Any Help will be greatly appreciated.

Thanks
Lal

MySql Cluster for educational use (1 reply)

Michał Herman — Mon, 01 Apr 2013 15:30:21 +0000

I am student and I am writing Master of Science paper about in-memory databases. I want to use MySql Cluster for benchmarks in my work.
Can I use MySql Cluster free of charge for educational purposes or do I need to pay for it?

REDO log? (1 reply)

Michael Oxford — Wed, 27 Mar 2013 23:54:07 +0000

So my first experience with MySQL cluster/NDB has been far less than stellar.

I have a corrupt database, which has never happened with InnoDB.

Set up and ran a cluster, low volume, NoCopies=2

1 x NDB + mysql on metal where all clients connected
1 x NDB + mysql on Oracle VirtualBox
1 x mnmt on secondary Oracle VirtualBox

During the course of operation, I got some table read errors, Err 233. (No text error message associated.) Looked it up, very suspicious that I was hitting the 32k MaxNoOfConcurrentOperations limit, but moved it up anyways to 65535.

Eventually ran out of disk on the NDB+mysql VM. Shut everything down, restarted with another VM disk attached for data and bumped memory alloc while it was down.

Still getting MaxNoOfConcurrentOperations-type errors, but fewer, even at 65535. This is dumb because we're sub 20 (TWENTY!) TPS with almost all of them being reads. No way it should be hitting 32k concurrent, much less 64k. These showed up on the bare-metal box, not the VM box.

Looking at the database, it's showing corruption ... it looks like some of transactions were completely replayed... the UPDATE calls which used "col = col + x" are all skewed. (Note: I did not tell it to use binlog or =row.)

1) Why would I possibly be getting Err: 233 off such a stupid low number of twenty TPS on the bare-metal box? I could see the VM potentially having issues, but it was literally just a replication target in the cluster.

2) Why would I end up with getting replayed statement transactions? How can I even start to trace this? How does this even happen? And importantly, how can I ensure that this never (ever) happens again?

Thanks ... hopefully someone can shed some light on this!

Connecting to Cluster (2 replies)

JJ Comb — Wed, 24 Apr 2013 09:16:33 +0000

Hello,

Running MySQL Cluster 7.2.10 on ubuntu 12.04 in Amazon.
Trying to connect with Java oracle jdk 1.7 on Windows 7.

I built a cluster and would like to connect to it. Successfully connected using JDBC. But encountered one specific exception when trying both ClusterJPA and CluterJ.
When connecting, it gives me this error (explaining the ClusterJ scenario and replacing the IP address of a management server (also a MySQL node in my cluster setup) with XX-XXX... in this email).

INFO: Created cluster connection 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:1186' with node id 0.
Exception in thread "main" com.mysql.clusterj.ClusterJFatalUserException: Error getting connection to cluster with properties {com.mysql.clusterj.connect.verbose=1, com.mysql.clusterj.connect.retries=4, com.mysql.clusterj.connect.delay=5, com.mysql.clusterj.connectstring=ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:1186, com.mysql.clusterj.max.transactions=1024, com.mysql.clusterj.connect.timeout.before=30, com.mysql.clusterj.connect.timeout.after=20, com.mysql.clusterj.database=clusterdb}:
Caused by com.mysql.clusterj.ClusterJDatastoreException:Datastore exception on connectString 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:1186' nodeId 0; Return code: -1 error code: 0 message: .
at com.mysql.clusterj.core.SessionFactoryImpl.createClusterConnection(SessionFactoryImpl.java:252)
at com.mysql.clusterj.core.SessionFactoryImpl.createClusterConnectionPool(SessionFactoryImpl.java:226)
at com.mysql.clusterj.core.SessionFactoryImpl.(SessionFactoryImpl.java:174)
at com.mysql.clusterj.core.SessionFactoryImpl.getSessionFactory(SessionFactoryImpl.java:129)
at com.mysql.clusterj.core.SessionFactoryServiceImpl.getSessionFactory(SessionFactoryServiceImpl.java:36)
at com.mysql.clusterj.core.SessionFactoryServiceImpl.getSessionFactory(SessionFactoryServiceImpl.java:27)
at com.mysql.clusterj.ClusterJHelper.getSessionFactory(ClusterJHelper.java:69)
at com.mysql.clusterj.ClusterJHelper.getSessionFactory(ClusterJHelper.java:54)

The error happens when running this statement early in my Main function:
SessionFactory factory = ClusterJHelper.getSessionFactory(props);
After the first line ("INFO...") is shown, it takes 15-20 seconds before the exceptions are shown.

The contents of clusterj.properties from which the props object is being loaded:
com.mysql.clusterj.connectstring=ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:1186
com.mysql.clusterj.database=clusterdb
com.mysql.clusterj.connect.retries=4
com.mysql.clusterj.connect.delay=5
com.mysql.clusterj.connect.verbose=1
com.mysql.clusterj.connect.timeout.before=30
com.mysql.clusterj.connect.timeout.after=20
com.mysql.clusterj.max.transactions=1024

In the exception message, For the node id, it says 0. This is while my node Ids start from 1.
In case you need other information (e.g. config.ini), please kindly note the command I need to execute in order to get that output. I am a novice.

Thank you,
JJ

ClusterJ and multiple management nodes (1 reply)

Alex Turetsky — Fri, 22 Mar 2013 10:48:35 +0000

In the MySQL cluster documentation there is a recommendation to have multiple management nodes (http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-basics.html).
The question is how to configure the ClusterJ to make use of multiple management nodes?
Is it anyhow related to the connection pooling in ClusterJ (com.mysql.clusterj.connection.pool.size and com.mysql.clusterj.connection.pool.nodeids properties)? What the connection pooling gives you and how to use it?

Thanks in advance,
Alex

Perform an Initial Restart (2 replies)

IGG t — Wed, 03 Apr 2013 19:05:57 +0000

I'm sure there is a simple solution (so sorry in advance).

I have installed a Cluster for testing purposes on my PC (Windows 7 - 64bit). It has:
2 x Management Nodes (ID's 1, 2)
2 x Data Nodes (ID's 11, 12)
4 x SQL Nodes (ID's 50, 51, 52, 53)

They are all installed as MySQL Services.

This all works very well so far. However I am now looking at making configuration changes, so need to perform an --initial restart.

I decided the obvious option is to change the nodeid of one of the data nodes (from 12 to 14)

I make the change in the 2 x config.ini files. I then start a mangement CLient and run:
1 RESTART -I
2 RESTART -I
11 RESTART -I
14 RESTART -I

But when I do a SHOW command, everything still has the same ID's as they started with. I'm assuming I have missed something obvious, but have been trawling the internet, and can't for the life of me find the answer.