I have finally been able to get the data you asked!
Here is the data usage by engine:
engine data_length index_length total_length
InnoDB 4682006528 3599335424 8281341952
MEMORY 0 0 0
The total is about 8 GB, and of this it's about 4.5 GB data and 3.5 GB indexes.
I think the "MEMORY" engine shows 0 usage because of the recent restart, anyway it is really not used much.
And here is the usage for each database, only for InnoDB:
schema data_length index_length total_length
db_master 110313472 143933440 254246912
db_00000517 239206400 212860928 452067328
db_00000617 173932544 163627008 337559552
db_00000717 313556992 190988288 504545280
db_00000917 113082368 147668992 260751360
db_00001113 339443712 248692736 588136448
db_00001917 170541056 177848320 348389376
db_00002017 133267456 170622976 303890432
db_00002117 257327104 220954624 478281728
db_00002217 139657216 156516352 296173568
db_00002417 1193951232 305692672 1499643904
db_00002517 105005056 146817024 251822080
db_00002617 216088576 199163904 415252480
db_00003017 157581312 178061312 335642624
db_00003217 273268736 223592448 496861184
db_00003416 103350272 146767872 250118144
db_00003616 123600896 152977408 276578304
db_00003617 266485760 197640192 464125952
db_00003816 250789888 210550784 461340672
logs_db_master 81920 229376 311296
logs_db_00000517 81920 229376 311296
logs_db_00000617 81920 229376 311296
logs_db_00000717 81920 229376 311296
logs_db_00000917 81920 229376 311296
logs_db_00001113 81920 229376 311296
logs_db_00001917 81920 229376 311296
logs_db_00002017 81920 229376 311296
logs_db_00002117 81920 229376 311296
logs_db_00002217 81920 229376 311296
logs_db_00002417 81920 229376 311296
logs_db_00002517 81920 229376 311296
logs_db_00002617 81920 229376 311296
logs_db_00003017 81920 229376 311296
logs_db_00003217 81920 229376 311296
logs_db_00003416 81920 229376 311296
logs_db_00003616 81920 229376 311296
logs_db_00003617 81920 229376 311296
logs_db_00003816 81920 229376 311296
TOTAL 4682006528 3599335424 8281341952
Most databases have about 200-300 MB of data, except for one which has grown up to about 1 GB.
The various logs_* databases are basically empty because the functionality which uses them is not enabled on this server.
> 1 Given the mem used, it's a surprisingly unbusy system, just 15 queries/sec
It could have been a moment of light load, though I do agree that it doesn't get much traffic, it usually is about 150-200 QPS with peaks of 1-2k QPS.
> 2 But about a quarter of those queries require full table scans---these queries are using lots of mem, need covering index optimisation
I guess I'd need to use the slow query log for those, right?
> 3 Nearly a thousand hrs between InnoDB log rotations is way too lomg
Isn't that done automatically when needed? I'll lookup how to do it.
> 6 innodb_buffer_pool_size use isn't really out of line, but with so many signs of mem distress, it needs to grow to 70-80% or so of RAM availale to MySQL
I'll try that too.
> What is the machine's swap setting?
It has the default swappiness value of 30, with about 14 GB of available swap space.
I didn't try to lower the swappiness, 'cause I thought that, given the memory problems, it'd have been better to allow it to swap more if needed.