DDL stuck on "Waiting for handler commit"

[Shravan Reddy Kotla]

Our application team tried adding a column to a busy table in production, and the DDL (ALTER TABLE ADD COLUMN) was stuck on "Waiting for handler commit" and started blocking all other transactions on the said table with "waiting for metadata lock" wait event. This caused the application outage for the duration of the transaction and resulted in a major incident.

We understand that the DDL requires metadata lock on the table for a brief period and the DDL session starts blocking new transactions until it acquires the metadata lock, however we don't understand what makes it wait on "Handler commit" and start blocking new transactions on "Metadata lock"?

The DDL was run using our tooling which runs the DDL in a standalone session with no prior transactions in it.

Does anyone know what can cause a DDL session to wait on handler commit while blocking other sessions on metadata lock?