server rebooted, now mysql will not start

[Troy Rockwood]

The server rebooted and mysqld will not start. I have tried adding the following to my.cnf:

innodb_force_recovery=1
log_error_verbosity=3


Here are the messages that I get:

../bin/mysqld --user=<username>
2023-09-22T01:40:09.496787Z 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
2023-09-22T01:40:09.317264Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2023-09-22T01:40:09.473095Z 0 [Note] [MY-010098] [Server] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2023-09-22T01:40:09.473146Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=9ad00ba9c420608e3d596a40a133fd20d019a80e
2023-09-22T01:40:09.473149Z 0 [Note] [MY-010949] [Server] Basedir set to /home/<usernamte>/mysql-8.1.0-linux-glibc2.28-x86_64/.
2023-09-22T01:40:09.473155Z 0 [System] [MY-010116] [Server] /home/<username>/mysql-8.1.0-linux-glibc2.28-x86_64/bin/mysqld (mysqld 8.1.0) starting as process 159439
2023-09-22T01:40:09.499152Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2023-09-22T01:40:09.499330Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2023-09-22T01:40:09.499393Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2023-09-22T01:40:09.499408Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2023-09-22T01:40:09.499415Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2023-09-22T01:40:09.501184Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-09-22T01:40:09.501271Z 1 [Note] [MY-013546] [InnoDB] Atomic write enabled
2023-09-22T01:40:09.501621Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2023-09-22T01:40:09.501668Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2023-09-22T01:40:09.501682Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2023-09-22T01:40:09.501695Z 1 [Note] [MY-012946] [InnoDB] !!! innodb_force_recovery is set to 1 !!!
2023-09-22T01:40:09.501722Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.13
2023-09-22T01:40:09.503277Z 1 [Note] [MY-012951] [InnoDB] Using hardware accelerated crc32 and polynomial multiplication.
2023-09-22T01:40:09.503584Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2023-09-22T01:40:09.503656Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2023-09-22T01:40:09.562913Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 7 files.
2023-09-22T01:40:09.563368Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M
2023-09-22T01:40:09.567362Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2023-09-22T01:40:09.568883Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-09-22T01:40:09.573091Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2023-09-22T01:40:09.574022Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2023-09-22T01:40:09.582062Z 1 [Note] [MY-013566] [InnoDB] Double write buffer files: 2
2023-09-22T01:40:09.582081Z 1 [Note] [MY-013565] [InnoDB] Double write buffer pages per instance: 4
2023-09-22T01:40:09.582113Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2023-09-22T01:40:09.582147Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2023-09-22T01:40:09.775884Z 1 [ERROR] [MY-013882] [InnoDB] Missing redo log file ./#innodb_redo/#ib_redo1038 (with start_lsn = 3399200768).
2023-09-22T01:40:09.775941Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2023-09-22T01:40:10.069122Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2023-09-22T01:40:10.069340Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-09-22T01:40:10.069390Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-09-22T01:40:10.069462Z 0 [Note] [MY-010120] [Server] Binlog end
2023-09-22T01:40:10.069613Z 0 [Note] [MY-015019] [Server] MySQL Server: Plugins Shutdown - start.
2023-09-22T01:40:10.069637Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2023-09-22T01:40:10.069705Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2023-09-22T01:40:10.069763Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2023-09-22T01:40:10.069778Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin'
2023-09-22T01:40:10.069909Z 0 [Note] [MY-015020] [Server] MySQL Server: Plugins Shutdown - end.
2023-09-22T01:40:10.070244Z 0 [System] [MY-010910] [Server] /home/<username>/mysql-8.1.0-linux-glibc2.28-x86_64/bin/mysqld: Shutdown complete (mysqld 8.1.0) MySQL Community Server - GPL.
2023-09-22T01:40:10.070352Z 0 [Note] [MY-015021] [Server] MySQL Server: Components Shutdown - start.
2023-09-22T01:40:10.070734Z 0 [Note] [MY-015022] [Server] MySQL Server: Components Shutdown - end (with return value = 0).
2023-09-22T01:40:10.070816Z 0 [System] [MY-015016] [Server] MySQL Server - end.

The redo directory looks like:
ls -alt \#innodb_redo/
total 57601
drwxr-x--- 8 <username> <groupname> 4096 Sep 21 21:40 ..
drwxr-x--- 2 <username> <groupname> 4096 Sep 21 18:36 .
-rw-r----- 1 <username> <groupname> 3276800 Sep 21 05:22 '#ib_redo1085'
-rw-r----- 1 <username> <groupname> 3276800 Sep 21 05:09 '#ib_redo1084'
-rw-r----- 1 <username> <groupname> 3276800 Sep 21 04:43 '#ib_redo1083'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 10:59 '#ib_redo1037'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 10:50 '#ib_redo1036'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 10:33 '#ib_redo1035'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 10:14 '#ib_redo1034'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 09:57 '#ib_redo1033'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 09:39 '#ib_redo1032'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 09:21 '#ib_redo1031'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 09:04 '#ib_redo1030'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 08:47 '#ib_redo1029'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 08:29 '#ib_redo1028'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 08:11 '#ib_redo1027'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 07:53 '#ib_redo1026'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 07:35 '#ib_redo1025'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 07:17 '#ib_redo1024'
-rw-r----- 1 <username> <groupname> 3276800 Sep 20 07:00 '#ib_redo1023'

ls -alth
total 2.9G
drwxr-x--- 8 <username> <groupname> 4.0K Sep 21 22:58 .
-rw-r----- 1 <username> <groupname> 6.1K Sep 21 18:45 <hostname>.err
drwxr-x--- 2 <username> <groupname> 4.0K Sep 21 18:36 '#innodb_redo'
-rw-r----- 1 <username> <groupname> 192K Sep 21 05:22 '#ib_16384_0.dblwr'
-rw-r----- 1 <username> <groupname> 12M Sep 21 05:22 ibdata1
-rw-r----- 1 <username> <groupname> 16M Sep 21 05:22 undo_001
-rw-r----- 1 <username> <groupname> 16M Sep 21 05:22 undo_002
-rw-r----- 1 <username> <groupname> 24M Sep 21 05:22 mysql.ibd
-rw-r----- 1 <username> <groupname> 805M Sep 21 05:22 binlog.000003
-rw-r----- 1 <username> <groupname> 8.2M Sep 21 05:21 '#ib_16384_1.dblwr'
-rw-r----- 1 <username> <groupname> 48 Sep 17 15:31 binlog.index
-rw-r----- 1 <username> <groupname> 1.1G Sep 17 15:31 binlog.000002
-rw-r----- 1 <username> <groupname> 1.1G Sep 14 21:03 binlog.000001
drwxr-x--- 2 <username> <groupname> 4.0K Sep 13 21:06 SIMULATIONS
-rw-r----- 1 <username> <groupname> 12M Sep 13 20:47 ibtmp1
drwxr-x--- 2 <username> <groupname> 4.0K Sep 13 20:47 '#innodb_temp'
-rw-r----- 1 <username> <groupname> 5.5K Sep 13 20:43 ib_buffer_pool
drwxr-x--- 2 <username> <groupname> 4.0K Sep 13 20:43 sys
drwxr-x--- 2 <username> <groupname> 4.0K Sep 13 20:42 mysql
-rw------- 1 <username> <groupname> 1.7K Sep 13 20:42 private_key.pem
-rw-r--r-- 1 <username> <groupname> 452 Sep 13 20:42 public_key.pem
-rw-r--r-- 1 <username> <groupname> 1.1K Sep 13 20:42 client-cert.pem
-rw------- 1 <username> <groupname> 1.7K Sep 13 20:42 client-key.pem
-rw-r--r-- 1 <username> <groupname> 1.1K Sep 13 20:42 server-cert.pem
-rw------- 1 <username> <groupname> 1.7K Sep 13 20:42 server-key.pem
-rw-r--r-- 1 <username> <groupname> 1.1K Sep 13 20:42 ca.pem
-rw------- 1 <username> <groupname> 1.7K Sep 13 20:42 ca-key.pem
drwxr-x--- 2 <username> <groupname> 8.0K Sep 13 20:42 performance_schema
-rw-r----- 1 <username> <groupname> 56 Sep 13 20:42 auto.cnf
drwxrwx--- 10 <username> <groupname> 4.0K Sep 13 20:42 ..

This is a user-space installation which means that I start with:
mysqld --user=<username>

I have been putting research simulation results in it for the past week or so basically nonstop. I had probably 13 different threads adding to the database. I have another copy of the data (the original results from the simulations) but I would prefer not to spend another week waiting for them to be placed in the database. I'm also pretty discouraged that the first hiccup killed the database. Is there anything that I should be doing to make this less likely? Thank you for your help.