Thanks for help Peter.
I don't see anything really wrong here , at this point I suspect bug . The memory usage is too high .
I ran mysqltunner and I got following result:
>> MySQLTuner 1.7.2 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at
http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from debian maintenance account.
[OK] Currently running supported MySQL version 5.7.18-0ubuntu0.16.04.1
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysql/error.log(1K)
[OK] Log file /var/log/mysql/error.log exists
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn't contain any warning.
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 0 start(s) detected in /var/log/mysql/error.log
[--] 0 shutdown(s) detected in /var/log/mysql/error.log
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in InnoDB tables: 1G (Tables: 2090)
[OK] Total fragmented tables: 0
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 14d 1h 20m 24s (22M q [18.558 qps], 179K conn, TX: 213G, RX: 11G)
[--] Reads / Writes: 85% / 15%
[--] Binary logging is disabled
[--] Physical Memory : 2.0G
[--] Max MySQL memory : 276.0M
[--] Other process memory: 82.7M
[--] Total buffers: 176.0M global + 1.0M per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 215.0M (10.75% of installed RAM)
[OK] Maximum possible memory usage: 276.0M (13.80% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/22M)
[OK] Highest usage of available connections: 39% (39/100)
[OK] Aborted connections: 0.00% (3/179622)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 74.1% (15M cached / 20M selects)
[!!] Query cache prunes per day: 96699
[OK] Sorts requiring temporary tables: 0% (104 temp sorts / 649K sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 11% (152K on disk / 1M total)
[OK] Thread cache hit rate: 99% (72 created / 179K connections)
[!!] Table cache hit rate: 0% (500 open / 16M opened)
[OK] Open file limit used: 0% (0/65K)
[OK] Table locks acquired immediately: 100% (70K immediate / 70K locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (1M used / 8M cache)
[OK] Key buffer size / total MyISAM indexes: 8.0M/45.0K
[!!] Read Key buffer hit rate: 87.4% (14K cached / 1K reads)
[OK] Write Key buffer hit rate: 100.0% (3 cached / 3 writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M/2.0G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.72% (951694936 hits/ 954379059 total)
[OK] InnoDB Write log efficiency: 98.14% (20410259 hits/ 20797025 total)
[OK] InnoDB log waits: 0.00% (0 waits / 386766 writes)
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.
-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.
-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64:
http://bit.ly/1mi7c4C
Beware that open_files_limit (65536) variable
should be greater than table_open_cache (500)
Performance should be activated for better diagnostics
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_size (> 8M)
table_open_cache (> 500)
performance_schema = ON enable PFS
innodb_buffer_pool_size (>= 1G) if possible.
innodb_log_file_size * innodb_log_files_in_group should be equal to 1/4 of buffer pool size (=64M) if possible.
Mysql should use 276M of memory but in reality it is using 1.1 GB and swapping a lot.
I don't see any long running queries or uncommitted transactions. The overall cpu usage by mysql is 0.7% avg .
I already reduced setting like meax_connections , key_buffer_size , read_buffer_size and query_cache_size , innodb_buffer_pool_size .
Nothing helps