How to debug very slow transactions stuck at "waiting for handler commit" state?

[Erfan Safari]

- My MySQL Setup:
I have a production MySQL 8.0.33 database.

All tables use the InnoDB storage engine.

The database has been running smoothly for the last 9 months.

Size: ~60GB of data and ~200 million rows.

Primary keys are CHAR(24) BSON ObjectIDs. These are chronologically sortable, similar to UUIDv1. We are aware that the most efficient way to store BSON ObjectIDs is as BINARY(12), but migrating to that would require extensive breaking changes, which are difficult to implement.

innodb_flush_log_at_trx_commit is set to 1, and sync_binlog is also set to 1.

- The Incident:
We have observed sudden MySQL slowdowns over the past few weeks. These incidents were temporary, lasting for several hours, and resolved on their own.

During the incidents, the output of SHOW FULL PROCESSLIST showed that many write transactions were stuck in the "waiting for handler commit" state for several minutes. These were simple INSERT or UPDATE statements—no heavy or bulk writes were involved.

We are currently unable to reproduce the issue.

- My Question:
In the event of a future incident, how can I troubleshoot the issue more deeply?
I'm not sure whether this is caused by disk saturation, an application-level issue (e.g., deadlocks), or possibly a MySQL bug.