Unable to connect to MySQL Cluster with ClusterJ

[Alexander Potapov]

**Database server machine**

On database server I have created user: mysqlcluster. This user does not have admin privileges.
I have downloaded generic Linux archive from Oracle website: mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64.tar.gz. This archive was unpacked in mysqlcluster user home directory. Data directories for cluster nodes are also in mysqlcluster user home directory.
I have installed MySql Cluster using auto installer wizard from external Windows machine (the same cluster archive version but for windows). During this installation I used mysqlcluster user to access machine. I specified for Mysql basedir location of the unpacked archive files. And data folder in user home directory. Nothing was changed in node settings except one thing. One API node was removed.

Executed hostnamectl command on database server

Icon name: computer-vm
Chassis: vm
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-327.36.3.el7.x86_64
Architecture: x86-64

Management node configuration (config.ini):

#
# Configuration file for MyCluster
#

[NDB_MGMD DEFAULT]
Portnumber=1186

[NDB_MGMD]
NodeId=49
HostName=192.168.175.11
DataDir=/home/mysqlcluster/data/49/
Portnumber=1186

[TCP DEFAULT]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[NDBD DEFAULT]
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M
BackupReportFrequency=10
MemReportFrequency=30
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15
DataMemory=2987M
IndexMemory=474M
MaxNoOfTables=4096
MaxNoOfTriggers=3500
NoOfReplicas=2
StringMemory=25
DiskPageBufferMemory=64M
SharedGlobalMemory=20M
LongMessageBuffer=32M
MaxNoOfConcurrentTransactions=16384
BatchSizePerLocalScan=512
FragmentLogFileSize=64M
NoOfFragmentLogFiles=16
RedoBuffer=32M
MaxNoOfExecutionThreads=2
StopOnError=false
LockPagesInMainMemory=1
TimeBetweenEpochsTimeout=32000
TimeBetweenWatchdogCheckInitial=60000
TransactionInactiveTimeout=60000
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000

[NDBD]
NodeId=1
HostName=192.168.175.11
DataDir=/home/mysqlcluster/data/1/

[NDBD]
NodeId=2
HostName=192.168.175.11
DataDir=/home/mysqlcluster/data/2/

[MYSQLD DEFAULT]

[MYSQLD]
NodeId=53

[MYSQLD]
NodeId=54

[API]
NodeId=50

[API]
NodeId=51

SQL node configuration (my.cnf):

#
# Configuration file for MyCluster
# Generated by mcc
#
[mysqld]
log-error=mysqld.53.err
datadir="/home/mysqlcluster/data/53/data"
tmpdir="/home/mysqlcluster/data/53/tmp"
basedir="/home/mysqlcluster/mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/"
port=3306
ndbcluster=on
ndb-nodeid=53
ndb-connectstring=192.168.175.11:1186,
socket="/home/mysqlcluster/data/53/mysql.socket"

Same goes for second SQL node (54).

Management node indicates following state:

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 @192.168.175.11 (mysql-5.7.16 ndb-7.5.4, Nodegroup: 0, *)
id=2 @192.168.175.11 (mysql-5.7.16 ndb-7.5.4, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=49 @192.168.175.11 (mysql-5.7.16 ndb-7.5.4)

[mysqld(API)] 4 node(s)
id=50 (not connected, accepting connect from any host)
id=51 (not connected, accepting connect from any host)
id=53 @192.168.175.11 (mysql-5.7.16 ndb-7.5.4)
id=54 @192.168.175.11 (mysql-5.7.16 ndb-7.5.4)

Running processes output:

mysqlcl+ 8873 1 0 13:27 ? 00:00:39 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/ndb_mgmd --initial --ndb-nodeid=49 --config-dir=/home/mysqlcluster/data/49/ --config-file=/home/mysqlcluster/data/49/config.ini
mysqlcl+ 8885 1 0 13:27 ? 00:00:01 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/ndbmtd --ndb-nodeid=1 --ndb-connectstring=192.168.175.11:1186,
mysqlcl+ 8886 8885 2 13:27 ? 00:03:08 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/ndbmtd --ndb-nodeid=1 --ndb-connectstring=192.168.175.11:1186,
mysqlcl+ 8934 1 0 13:28 ? 00:00:01 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/ndbmtd --ndb-nodeid=2 --ndb-connectstring=192.168.175.11:1186,
mysqlcl+ 8935 8934 2 13:28 ? 00:03:00 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/ndbmtd --ndb-nodeid=2 --ndb-connectstring=192.168.175.11:1186,
mysqlcl+ 8982 7466 0 13:28 pts/1 00:00:45 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/home/mysqlcluster/data/53/my.cnf
mysqlcl+ 9019 7466 0 13:28 pts/1 00:00:44 mysql-cluster-gpl-7.5.4-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/home/mysqlcluster/data/54/my.cnf

Database application user privileges:

GRANT USAGE ON *.* TO 'someuser'@'192.168.175.1'
GRANT ALL PRIVILEGES ON `somedb`.* TO 'someuser'@'192.168.175.1'


Executed netstat -nat | grep :1186 command on database server:

tcp 0 0 0.0.0.0:1186 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:1186 127.0.0.1:33818 ESTABLISHED
tcp 0 0 192.168.175.11:45890 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:45882 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:45884 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45876 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45878 ESTABLISHED
tcp 0 0 192.168.175.11:45878 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45884 ESTABLISHED
tcp 0 0 192.168.175.11:45880 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45882 ESTABLISHED
tcp 0 0 127.0.0.1:33818 127.0.0.1:1186 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45880 ESTABLISHED
tcp 0 0 192.168.175.11:45876 192.168.175.11:1186 ESTABLISHED
tcp 0 0 192.168.175.11:1186 192.168.175.11:45890 ESTABLISHED

In iptables we have following declarations:

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
ACCEPT tcp -- 192.168.175.9 0.0.0.0/0 tcp
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:10001
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1186
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:3306
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8070
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8090
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8170
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8171
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination


**Application machine**

I use ClusterJ from my application to work with database data. Jars are taken from MySql Cluster archive (v 7.5.4).
We use com.mysql.clusterj.connectstring and com.mysql.clusterj.database parameters for database connection. This is how it is done on Java level:

Properties props = new Properties();

props.put("com.mysql.clusterj.connectstring", databaseConf.getHost() + ":" + databaseConf.getPort());
props.put("com.mysql.clusterj.database", databaseConf.getDatabase());

sessionFactory = ClusterJHelper.getSessionFactory(props);

In configuration we have:

"database": {
"database": "somedb",
"host": "192.168.175.11",
"port": "1186"
}

We have Java application running on external VM inside the same network. Ip of this machine is 192.168.175.1.

When we execute application it hangs on com.mysql.ndbjtie.ndbapi.Ndb_cluster_connection.wait_until_ready(Native Method) method.
After 30 seconds, com.mysql.ndbjtie.ndbapi.Ndb_cluster_connection.delete(Native Method) is called.
After that, exception is thrown:

INFO: An exception was caught and reported. Message: com.mysql.clusterj.ClusterJDatastoreException: Datastore exception on connectString '192.168.175.11:1186' nodeId 0; Return code: -1 error code: 0 message: .
com.mysql.clusterj.ClusterJFatalUserException: Error getting connection to cluster with properties {com.mysql.clusterj.connectstring=192.168.175.11:1186, com.mysql.clusterj.database=somedb}:
Caused by com.mysql.clusterj.ClusterJDatastoreException:Datastore exception on connectString '192.168.175.11:1186' nodeId 0; Return code: -1 error code: 0 message: .
at com.mysql.clusterj.core.SessionFactoryImpl.createClusterConnection(SessionFactoryImpl.java:289)
at com.mysql.clusterj.core.SessionFactoryImpl.createClusterConnectionPool(SessionFactoryImpl.java:261)
at com.mysql.clusterj.core.SessionFactoryImpl.<init>(SessionFactoryImpl.java:193)
at com.mysql.clusterj.core.SessionFactoryImpl.getSessionFactory(SessionFactoryImpl.java:139)
at com.mysql.clusterj.core.SessionFactoryServiceImpl.getSessionFactory(SessionFactoryServiceImpl.java:36)
at com.mysql.clusterj.core.SessionFactoryServiceImpl.getSessionFactory(SessionFactoryServiceImpl.java:27)
at com.mysql.clusterj.ClusterJHelper.getSessionFactory(ClusterJHelper.java:72)
at com.mysql.clusterj.ClusterJHelper.getSessionFactory(ClusterJHelper.java:57)


**Can somebody tell me the reason why I cannot connect to database?**

**NOTE: If I run application from the same machine where MySql Cluster is running then everything is fine.**

Also tried this command on database server: tcpdump -i eno16777984 -vv port 1186
After executing application on second machine I got following output:

....
15:50:33.847381 IP (tos 0x0, ttl 64, id 56694, offset 0, flags [DF], proto TCP (6), length 109)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [P.], cksum 0xdfbd (incorrect -> 0x44af), seq 172:229, ack 225, win 227, options [nop,nop,TS val 3733789260 ecr 3733975901], length 57
15:50:33.887097 IP (tos 0x0, ttl 64, id 40735, offset 0, flags [DF], proto TCP (6), length 52)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [.], cksum 0x78a8 (correct), seq 225, ack 229, win 229, options [nop,nop,TS val 3733975941 ecr 3733789260], length 0
15:50:33.948050 IP (tos 0x0, ttl 64, id 40736, offset 0, flags [DF], proto TCP (6), length 77)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [P.], cksum 0xc974 (correct), seq 225:250, ack 229, win 229, options [nop,nop,TS val 3733976002 ecr 3733789260], length 25
15:50:33.987186 IP (tos 0x0, ttl 64, id 56695, offset 0, flags [DF], proto TCP (6), length 52)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [.], cksum 0xdf84 (incorrect -> 0x77c8), seq 229, ack 250, win 227, options [nop,nop,TS val 3733789400 ecr 3733976002], length 0
15:50:33.987369 IP (tos 0x0, ttl 64, id 40737, offset 0, flags [DF], proto TCP (6), length 83)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [P.], cksum 0x8938 (correct), seq 250:281, ack 229, win 229, options [nop,nop,TS val 3733976041 ecr 3733789400], length 31
15:50:33.987426 IP (tos 0x0, ttl 64, id 56696, offset 0, flags [DF], proto TCP (6), length 52)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [.], cksum 0xdf84 (incorrect -> 0x7782), seq 229, ack 281, win 227, options [nop,nop,TS val 3733789400 ecr 3733976041], length 0
15:50:33.987484 IP (tos 0x0, ttl 64, id 56697, offset 0, flags [DF], proto TCP (6), length 109)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [P.], cksum 0xdfbd (incorrect -> 0x3c21), seq 229:286, ack 281, win 227, options [nop,nop,TS val 3733789400 ecr 3733976041], length 57
15:50:33.987547 IP (tos 0x0, ttl 64, id 40738, offset 0, flags [DF], proto TCP (6), length 52)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [.], cksum 0x7747 (correct), seq 281, ack 286, win 229, options [nop,nop,TS val 3733976041 ecr 3733789400], length 0
15:50:33.987795 IP (tos 0x0, ttl 64, id 40739, offset 0, flags [DF], proto TCP (6), length 77)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [P.], cksum 0xc850 (correct), seq 281:306, ack 286, win 229, options [nop,nop,TS val 3733976041 ecr 3733789400], length 25
15:50:34.027178 IP (tos 0x0, ttl 64, id 56698, offset 0, flags [DF], proto TCP (6), length 52)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [.], cksum 0xdf84 (incorrect -> 0x7708), seq 286, ack 306, win 227, options [nop,nop,TS val 3733789440 ecr 3733976041], length 0
15:50:34.027437 IP (tos 0x0, ttl 64, id 40740, offset 0, flags [DF], proto TCP (6), length 83)
192.168.175.1.42216 > mysql-cluster-hostname: Flags [P.], cksum 0x8876 (correct), seq 306:337, ack 286, win 229, options [nop,nop,TS val 3733976081 ecr 3733789440], length 31
15:50:34.027479 IP (tos 0x0, ttl 64, id 56699, offset 0, flags [DF], proto TCP (6), length 52)
mysql-cluster-hostname > 192.168.175.1.42216: Flags [.], cksum 0xdf84 (incorrect -> 0x76c1), seq 286, ack 337, win 227, options [nop,nop,TS val 3733789440 ecr 3733976081], length 0
....

Appreciate some help.