MySQL: 8.0.31 - Large gaps (thousands) in ID Field after few months of usage

[Alex Anto Navis Lawrence]

We see large gaps in primary key column with auto_increment in mysql 8.0.31 release version in multiple tables.

We have 'order' table (id bigint) and 'order_line_items' table (id bigint).

TABLE_NAME | CURRENT AUTO_INCREMENT | TABLE ROW COUNT
order | 27504 | 1367
order_line_items | 430930 | 34970

We have a total of 1367 order rows with current auto increment value of '27504', and 34970 (order line item rows) with current auto increment value of '430930'. We save, order and order_line_items all within the same transaction in a API call.

Gaps:

Order table gaps continued to happen for 1-2 day once we noticed and went back to incrementing by '1' again as of today. Jumps are happening together in clusters continuously (check below) and went back to incrementing by '1'.

Order table gap details:

id | difference from previous 'id'
-------------------------------------
27489 1
27488 1
27487 1
27486 232
27254 232
27022 693
26329 240
26089 401
25688 175
25513 348
25165 864
24301 7656
16645 1709
14936 184
14752 13
14739 541
14198 82
14116 215
13901 68
13833 117
13716 140
....
1468 1
1467 10
1457 1
1456 1
1455 1
1454 9
1445 1
1444 1
1443 1

Observations:

1) We use (OpenJPA) use the configuration (@Id, @GeneratedValue(strategy = GenerationType.IDENTITY) for the ID. Id's are not set in the code.

2) We see the same pattern in other tables as well. We tested the same API with parallel creation of the same with 20 threads but we didn't similar large gaps in the create id's for both tables.

3) Till id: 1445 (we have count of 1225 rows) in order table, we don't see any issue with bigger jumps. After id: 1445, we started seeing multiple jumps in order table till 27487 for 1-2 days continuously.

Questions:

1) Does anyone has any idea what can be the cause for this ?. Why id's are jumping in multiple thousands (e.g., 7656, 1709 in above example) when the total records are below 1367 in order tables ?.

2) I see this happens in multiple cluster of islands of the ids jumping and starts back to create one by one (like the above example). Has this relate to any other internal handling of mysql data relate processes. ?. Is this a possible bug in mysql ?

3) How can I simulate this issue locally ?.

Any ideas would be really helpful.