Max_used_connections -- that is the high-water mark since restarting mysql.
innodb_additional_mem_pool_size -- no longer used
tmp_table_size -- complex queries could need multiple of these. However, max_tmp_tables is a limit on such.
I have not found a formula that is worth using. Still, I would not expect you to see 9G when the formula says much less (5.3G). Usually the formulas are seriously pessimistic.
Memory leaks have been very rare in MySQL. You are running 5.6.15; I see this in the next release (but it does not seem relevant?):
Using the InnoDB (
http://dev.mysql.com/doc/refman/5.6/en/innodb-storage-engine.html ) memcached plugin (see InnoDB Integration with memcached (
http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached.html )) with innodb_api_enable_binlog (
http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_api_enable_binlog ) set to 1 caused the server to leak memory. (Bug #70757, Bug #17675622)
Swapping will slow MySQL down terribly. It would be better to decrease various settings rather than let it swap.
Is it swapping?
Sorry, I can't seem to spot the cause of your problem. Please let us know if you discover it.