Zuban,
Been travelling, sorry I missed your reply.
Running your settings through a little analyser gives
http://www.artfulsoftware.com/wrk/suri-mysql.html. Nothing very dramatic. You need to turn on the slow query cache to find the queries that need full table scans. InnoDB log size could shrink. You could try shrinking innodb_buffer_pool_size. The query cache appears to be useless, try turning it off. But nothing here explains your high MySQL mem usage.
Is there a lot of MyISAM data? What's the result of ...
select
engine,
round(sum(data_length) /1024/1024, 1) as "data mb",
round(sum(index_length)/1024/1024, 1) as "index mb",
round(sum(data_length + index_length)/1024/1024, 1) as "total mb",
count(*) "num tables"
from information_schema.tables
where table_schema not in ("information_schema", "performance_schema", "sys_schema", "ndbinfo")
group by engine;
As mentioned in the thread just above yours, since 5.7 MySQL can summarise current memory usage after you make this performance_schema change (https://dev.mysql.com/doc/refman/5.7/en/memory-summary-tables.html):
update performance_schema.setup_instruments
set enabled = 'yes'
where name like 'memory/%';
Then ...
select * from sys.memory_global_by_current_bytes;