Re: How to scale MySQL Cluster to support PB storage

[Mahmood Bazaar]

Not sure if my earlier post went thru.

===
Rick,

Thank you for quick response.

Please find the details of the system below:
1. Target scale:
a. The system should sustain several 10s of thousands connections and
b. several thousand requests per second.
c. Several PB is the size of the actual data that the system will store; the size of metadata (that will be stored in DB) is going to be several 10s of TB.
d. These number are not final; the architecture should be flexible to allow further growing without de-designing.
2. The planned architecture is about that you mentioned:
a. Client layer (Apache) – several hundreds of machines
b. DB layer (MySQL) – several tens of machines
c. Storage layer – many hundreds of machines (stores actual data)
3. Availability
a. It is important to notice that there will be several data centers and each layer is to be split between them.
i. The requirement is that the system must be available in case of failure of a data center.
b. Availability of each level will be achieved (semi-) independently
i. For Client layer it is pretty straightforward
ii. There is a way to do this for Storage layer.
iii. For DB layer the replication the dual-master approach should work
1. Will it work efficiently in case of remote MySQL replication?
2. Does it make sense to use MySQL cluster for shards in this case?
4. Other details
a. The system will be used for standard “data warehousing”.
b. The metadata information stored in DB is loosely coupled can be easily sharded between several DB servers.
i. Typical operations will affect only one DB server.
ii. The capacity of the system will increase (very fast); re-sharding (without downtime) must be supported.
c. The only issue is that the primary key is an object name is a long text string.

Some questions:
1) I know that there are some framework that can facilitate MySQL sharding available. These are: HSCALE, Spock Proxy, Hibernate Shards and maybe some others.
What would you suggest to use?
2) The main issues with sharding is “data migration” in case of re-sharding.
I wonder if there are any solutions that can facilitate re-sharding available? What do you think about HiveDB (http://hivedb.org)?

Thanks in advance.

==