MySQL Server 5.6.43 - crash

[Matthieu Larère]

Hi,

Yesterday one Mysql community server that I have crashed and I am not so sure why. In /var/log/messages I saw messages related with memory as it seems to me, but I am not so sure if this is because of a faulty memory or a lack of memory. It seems to me it's somehow related with swap as it claims "Free swao = 0kB" which might be an issue if swap is needed.

To be honest, I am not fluent in terms of /var/log/messages, it says a lot of debug stuff that I am not sure how to read all. Could some of you have a look at it at tell me what might have cause the issue or what I should look for to get further info ? Thanks a lot !

May 1 11:22:55 db-master kernel: mysqld: page allocation stalls for 11648ms, order:0, mode:0x34200ca(GFP_HIGHUSER_MOVABLE|__GFP_WRITE)
May 1 11:22:55 db-master kernel: CPU: 1 PID: 20596 Comm: mysqld Not tainted 4.9.87-xxxx-std-ipv6-64 #1
May 1 11:22:55 db-master kernel: Hardware name: Intel Corporation S2600CWR/S2600CWR, BIOS SE5C610.86B.01.01.3021.051020171131 05/10/2017
May 1 11:22:55 db-master kernel: ffffacd82481f980 ffffffffa866c036 ffffffffa931f398 0000000000000001
May 1 11:22:55 db-master kernel: ffffacd82481fa08 ffffffffa81be485 034200cab84e9980 ffffffffa931f398
May 1 11:22:55 db-master kernel: ffffacd82481f9a8 0000000000000010 ffffacd82481fa18 ffffacd82481f9c8
May 1 11:22:55 db-master kernel: Call Trace:
May 1 11:22:55 db-master kernel: [<ffffffffa866c036>] dump_stack+0x4d/0x67
May 1 11:22:55 db-master kernel: [<ffffffffa81be485>] warn_alloc+0x135/0x150
May 1 11:22:55 db-master kernel: [<ffffffffa81bee16>] __alloc_pages_slowpath+0x906/0xc00
May 1 11:22:55 db-master kernel: [<ffffffffa81bf2c9>] __alloc_pages_nodemask+0x149/0x1e0
May 1 11:22:55 db-master kernel: [<ffffffffa82071c4>] alloc_pages_current+0x84/0x160
May 1 11:22:55 db-master kernel: [<ffffffffa81b5da3>] __page_cache_alloc+0x93/0xf0
May 1 11:22:55 db-master kernel: [<ffffffffa81b6ca6>] pagecache_get_page+0x96/0x210
May 1 11:22:55 db-master kernel: [<ffffffffa81b8bfe>] grab_cache_page_write_begin+0x1e/0x40
May 1 11:22:55 db-master kernel: [<ffffffffa8302e64>] ext4_da_write_begin+0x94/0x300
May 1 11:22:55 db-master kernel: [<ffffffffa8303d8f>] ? ext4_da_write_end+0x11f/0x270
May 1 11:22:55 db-master kernel: [<ffffffffa81b8cdd>] generic_perform_write+0xbd/0x1c0
May 1 11:22:55 db-master kernel: [<ffffffffa81b8ee4>] __generic_file_write_iter+0x104/0x1c0
May 1 11:22:55 db-master kernel: [<ffffffffa82f7f8c>] ext4_file_write_iter+0x1ec/0x340
May 1 11:22:55 db-master kernel: [<ffffffffa822d08c>] __vfs_write+0xec/0x150
May 1 11:22:55 db-master kernel: [<ffffffffa822d28d>] vfs_write+0xad/0x1d0
May 1 11:22:55 db-master kernel: [<ffffffffa822d4c0>] SyS_write+0x50/0xc0
May 1 11:22:55 db-master kernel: [<ffffffffa800150c>] do_syscall_64+0x5c/0xc0
May 1 11:22:55 db-master kernel: [<ffffffffa8e93f3e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
May 1 11:22:55 db-master kernel: Mem-Info:
May 1 11:22:55 db-master kernel: active_anon:7408677 inactive_anon:436077 isolated_anon:0#012 active_file:7519305 inactive_file:729049 isolated_file:128#012 unevictable:0 dirty:766666 writeback:9695 unstable:0#012 slab_reclaimable:197141 slab_unreclaimable:25448#012 mapped:7918 shmem:66142 pagetables:16110 bounce:0#012 free:83283 free_pcp:29 free_cma:0
May 1 11:22:55 db-master kernel: Node 0 active_anon:29634708kB inactive_anon:1744308kB active_file:30077220kB inactive_file:2916196kB unevictable:0kB isolated(anon):0kB isolated(file):512kB mapped:31672kB dirty:3066664kB writeback:38780kB shmem:264568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 30674944kB writeback_tmp:0kB unstable:0kB pages_scanned:29374569 all_unreclaimable? no
May 1 11:22:55 db-master kernel: Node 0 DMA free:15880kB min:16kB low:28kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15880kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 1 11:22:55 db-master kernel: lowmem_reserve[]: 0 1872 64306 64306
May 1 11:22:55 db-master kernel: Node 0 DMA32 free:251680kB min:1964kB low:3880kB high:5796kB active_anon:1265528kB inactive_anon:395364kB active_file:8588kB inactive_file:4224kB unevictable:0kB writepending:4300kB present:1982628kB managed:1982628kB mlocked:0kB slab_reclaimable:56856kB slab_unreclaimable:240kB kernel_stack:16kB pagetables:96kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 1 11:22:55 db-master kernel: lowmem_reserve[]: 0 0 62434 62434
May 1 11:22:55 db-master kernel: Node 0 Normal free:65572kB min:65600kB low:129532kB high:193464kB active_anon:28369180kB inactive_anon:1348944kB active_file:30068632kB inactive_file:2911972kB unevictable:0kB writepending:3101144kB present:65011712kB managed:63937840kB mlocked:0kB slab_reclaimable:731708kB slab_unreclaimable:101552kB kernel_stack:7648kB pagetables:64344kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB
May 1 11:22:55 db-master kernel: lowmem_reserve[]: 0 0 0 0
May 1 11:22:55 db-master kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 3*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15880kB
May 1 11:22:55 db-master kernel: Node 0 DMA32: 14*4kB (UME) 27*8kB (UE) 1543*16kB (UMEH) 1059*32kB (UMEH) 501*64kB (UMEH) 308*128kB (UMEH) 156*256kB (UME) 81*512kB (UME) 39*1024kB (UM) 0*2048kB 0*4096kB = 251680kB
May 1 11:22:55 db-master kernel: Node 0 Normal: 2937*4kB (UE) 1724*8kB (UME) 394*16kB (UMEH) 32*32kB (UMEH) 511*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65572kB
May 1 11:22:55 db-master kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
May 1 11:22:55 db-master kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 1 11:22:55 db-master kernel: 8316706 total pagecache pages
May 1 11:22:55 db-master kernel: 2082 pages in swap cache
May 1 11:22:55 db-master kernel: Swap cache stats: add 8416989, delete 8414286, find 11541080/12729388
May 1 11:22:55 db-master kernel: Free swap = 0kB
May 1 11:22:55 db-master kernel: Total swap = 2093040kB
May 1 11:22:55 db-master kernel: 16752582 pages RAM
May 1 11:22:55 db-master kernel: 0 pages HighMem/MovableOnly
May 1 11:22:55 db-master kernel: 268495 pages reserved
May 1 11:22:55 db-master kernel: 0 pages hwpoisoned
May 1 11:22:59 db-master kernel: mysqld: page allocation stalls for 12708ms, order:0, mode:0x24000c0(GFP_KERNEL)
May 1 11:22:59 db-master kernel: CPU: 5 PID: 24366 Comm: mysqld Not tainted 4.9.87-xxxx-std-ipv6-64 #1
May 1 11:22:59 db-master kernel: Hardware name: Intel Corporation S2600CWR/S2600CWR, BIOS SE5C610.86B.01.01.3021.051020171131 05/10/2017
May 1 11:22:59 db-master kernel: ffffacd80a43faa0 ffffffffa866c036 ffffffffa931f398 0000000000000001
May 1 11:22:59 db-master kernel: ffffacd80a43fb28 ffffffffa81be485 024000c0c54c72c0 ffffffffa931f398
May 1 11:22:59 db-master kernel: ffffacd80a43fac8 0000000000000010 ffffacd80a43fb38 ffffacd80a43fae8
May 1 11:22:59 db-master kernel: Call Trace:
May 1 11:22:59 db-master kernel: [<ffffffffa866c036>] dump_stack+0x4d/0x67
May 1 11:22:59 db-master kernel: [<ffffffffa81be485>] warn_alloc+0x135/0x150
May 1 11:22:59 db-master kernel: [<ffffffffa81bee16>] __alloc_pages_slowpath+0x906/0xc00
May 1 11:22:59 db-master kernel: [<ffffffffa81bf2c9>] __alloc_pages_nodemask+0x149/0x1e0
May 1 11:22:59 db-master kernel: [<ffffffffa82071c4>] alloc_pages_current+0x84/0x160
May 1 11:22:59 db-master kernel: [<ffffffffa8bfddfc>] skb_page_frag_refill+0x9c/0xc0
May 1 11:22:59 db-master kernel: [<ffffffffa8bfde38>] sk_page_frag_refill+0x18/0x80
May 1 11:22:59 db-master kernel: [<ffffffffa8d0b100>] tcp_sendmsg+0x210/0xbc0
May 1 11:22:59 db-master kernel: [<ffffffffa8d3743d>] inet_sendmsg+0x5d/0xa0
May 1 11:22:59 db-master kernel: [<ffffffffa8bf8452>] ? sockfd_lookup_light+0x12/0x70
May 1 11:22:59 db-master kernel: [<ffffffffa8bf9180>] SYSC_sendto+0x150/0x1b0
May 1 11:22:59 db-master kernel: [<ffffffffa824461a>] ? dput.part.28+0x2a/0x210
May 1 11:22:59 db-master kernel: [<ffffffffa81937b3>] ? __audit_syscall_entry+0xb3/0x100
May 1 11:22:59 db-master kernel: [<ffffffffa8001264>] ? syscall_trace_enter+0x1d4/0x260
May 1 11:22:59 db-master kernel: [<ffffffffa8193a31>] ? __audit_syscall_exit+0x231/0x2b0
May 1 11:22:59 db-master kernel: [<ffffffffa8bfa559>] SyS_sendto+0x9/0x10
May 1 11:22:59 db-master kernel: [<ffffffffa800150c>] do_syscall_64+0x5c/0xc0
May 1 11:22:59 db-master kernel: [<ffffffffa8e93f3e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
May 1 11:22:59 db-master kernel: Mem-Info:
May 1 11:22:59 db-master kernel: active_anon:7408677 inactive_anon:436077 isolated_anon:0#012 active_file:7519305 inactive_file:729145 isolated_file:32#012 unevictable:0 dirty:766666 writeback:9695 unstable:0#012 slab_reclaimable:197141 slab_unreclaimable:25448#012 mapped:7923 shmem:66142 pagetables:16110 bounce:0#012 free:83283 free_pcp:29 free_cma:0
May 1 11:22:59 db-master kernel: Node 0 active_anon:29634708kB inactive_anon:1744308kB active_file:30077220kB inactive_file:2916580kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:31692kB dirty:3066664kB writeback:38780kB shmem:264568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 30674944kB writeback_tmp:0kB unstable:0kB pages_scanned:55239173 all_unreclaimable? no
May 1 11:22:59 db-master kernel: Node 0 DMA free:15880kB min:16kB low:28kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15880kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 1 11:22:59 db-master kernel: lowmem_reserve[]: 0 1872 64306 64306
May 1 11:22:59 db-master kernel: Node 0 DMA32 free:251680kB min:1964kB low:3880kB high:5796kB active_anon:1265528kB inactive_anon:395364kB active_file:8588kB inactive_file:4224kB unevictable:0kB writepending:4300kB present:1982628kB managed:1982628kB mlocked:0kB slab_reclaimable:56856kB slab_unreclaimable:240kB kernel_stack:16kB pagetables:96kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 1 11:22:59 db-master kernel: lowmem_reserve[]: 0 0 62434 62434
May 1 11:22:59 db-master kernel: Node 0 Normal free:65572kB min:65600kB low:129532kB high:193464kB active_anon:28369180kB inactive_anon:1348944kB active_file:30068632kB inactive_file:2912356kB unevictable:0kB writepending:3101144kB present:65011712kB managed:63937840kB mlocked:0kB slab_reclaimable:731708kB slab_unreclaimable:101552kB kernel_stack:7648kB pagetables:64344kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB
May 1 11:22:59 db-master kernel: lowmem_reserve[]: 0 0 0 0
May 1 11:22:59 db-master kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 3*16kB (U) 2*32kB (U) 2*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15880kB
May 1 11:22:59 db-master kernel: Node 0 DMA32: 14*4kB (UME) 27*8kB (UE) 1543*16kB (UMEH) 1059*32kB (UMEH) 501*64kB (UMEH) 308*128kB (UMEH) 156*256kB (UME) 81*512kB (UME) 39*1024kB (UM) 0*2048kB 0*4096kB = 251680kB
May 1 11:22:59 db-master kernel: Node 0 Normal: 2937*4kB (UE) 1724*8kB (UME) 394*16kB (UMEH) 32*32kB (UMEH) 511*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65572kB
May 1 11:22:59 db-master kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
May 1 11:22:59 db-master kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 1 11:22:59 db-master kernel: 8316706 total pagecache pages
May 1 11:22:59 db-master kernel: 2082 pages in swap cache
May 1 11:22:59 db-master kernel: Swap cache stats: add 8416989, delete 8414286, find 11541080/12729388
May 1 11:22:59 db-master kernel: Free swap = 0kB
May 1 11:22:59 db-master kernel: Total swap = 2093040kB
May 1 11:22:59 db-master kernel: 16752582 pages RAM
May 1 11:22:59 db-master kernel: 0 pages HighMem/MovableOnly
May 1 11:22:59 db-master kernel: 268495 pages reserved
May 1 11:22:59 db-master kernel: 0 pages hwpoisoned