You might want to compare what you're doing with what I've tried below:
mysqld.exe <your other options> --plugin-dir=<directory of keyring_file.dll> --early-plugin-load=keyring_file.dll --keyring_file_data=<absolute path to an writable file location>
I was then able to do:
mysql> create table t1(a int) encryption='y';
Query OK, 0 rows affected (0.06 sec)
mysql> show tables;
+----------------+
| Tables_in_haha |
+----------------+
| t1 |
+----------------+
1 row in set (0.06 sec)
mysql> show create table t1;
+-------+-------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------+
| t1 | CREATE TABLE `t1` (
`a` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ENCRYPTION='y' |
+-------+-------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
Please share all of your config details and the exact steps you've taken.