MySQL Forums
Forum List  »  Performance

Re: How to scale MySQL Cluster to support PB storage
Posted by: Rick James
Date: January 20, 2009 08:29PM

Automatic failover -- virtually impossible. There are too many things that can fool the automatic system. You don't want to ever have both masters thinking they are alone -- something that can happen if there is a partial network outage.

Each shard should have 2 masters, one of which is being written to. The masters should be in different geo-locations and be replicating from each other. For maximal ease of recovery from certain disasters, also have at least one slave hanging off each master.

Read scaling: add slaves.

Write scaling: add shards.

I'm ignorant of Hbase and HyperTable. Be cautious -- an abstraction layer may or may not abstract the right things. An abstraction that does single row lookups is good if that is what you are doing. A layer that hashes the items is clumsy when you ask for a range. JOINs are either not handled or have performance that would make you want to punt. Etc.

Given that you need to fetch a "range" of items by key, BIGINT and Hash are impractical (as you say).

My "in between -- hash to a 16-bit value, then do dictionary lookup"...
Well, replace the hashing with a roughly even partitioning of the lexically ordered keys into several thousand buckets. Depending on the keys, this might work. (For something like timestamps, there are always new buckets coming into existence. And possibly old ones being purged. Doable, but not straight forward.)

Good Luck.

Options: ReplyQuote

Written By
Re: How to scale MySQL Cluster to support PB storage
January 20, 2009 08:29PM

Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.