Re: Need help to review a serializability implementation for MySQL Cluster

[Alex Ou]

There has been a consistency-related catastrophe lately and this time it is Amazon: https://arstechnica.com/gadgets/2025/10/a-single-point-of-failure-triggered-the-amazon-outage-affecting-millions/.

From the description of this article, we can see clearly that there is a pair of WW conflicts between the two Enactors on the eventually deleted plan(let's assume that is a tuple). If they were executed under a serializable isolation level, this conflict loop wouldn't have survived and left the database in an inconsistent state.

DynamoDB's default isolation level is called 'serializable'. But from the specification in DynamoDB's doc here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html#transaction-isolation,

it only provides serializable isolation at the operation(read/write) level, NOT at the transaction level.

Failure to anticipate two Enactors could pop up in the system also contributes to this disaster. Since there is apparently a Swiss cheese effect in play here and human mistakes are not avoidable, placing a critical database application under a genuine serializable isolation level or something equivalent to prevent this kind of catastrophe is extremely important.

It is said that this incident has incurred a multi-billion-dolar loss to Amazon: https://slashdot.org/story/25/10/21/1942240/amazons-dns-problem-knocked-out-half-the-web-likely-costing-billions.



Edited 1 time(s). Last edit at 11/10/2025 08:00PM by Alex Ou.