Re: Problems with system locking
Ingo,
We have also captured system stats when thjis happened.
What is the suggested analysis you want me to do?
See below:
Thanks
ananth
Here is VMSTATS output:
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 117456 18716 1946596 0 0 0 0 0 1 0 1 1 0
1 0 0 117408 18716 1946624 0 0 0 0 968 720 25 1 75 0
Here is TOP output with lot of mysql threads and one of thread peg it CPU
15:09:51 up 68 days, 23:38, 4 users, load average: 1.08, 0.93, 1.08
276 processes: 274 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 25.1% 0.0% 0.9% 0.0% 0.2% 0.0% 73.6%
cpu00 0.9% 0.0% 2.8% 0.0% 0.9% 0.0% 95.1%
cpu01 0.0% 0.0% 0.9% 0.0% 0.0% 0.0% 99.0%
cpu02 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
cpu03 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
Mem: 3857188k av, 3739828k used, 117360k free, 0k shrd, 18716k buff
2996420k actv, 329572k in_d, 58928k in_c
Swap: 0k av, 0k used, 0k free 1946656k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
24467 mysql 25 0 1601M 1.6G 1780 R 24.8 42.5 1:27 2 mysqld
26206 mysql 22 0 1292 1292 876 R 0.7 0.0 0:00 0 top
26149 mysql 15 0 1601M 1.6G 1780 S 0.2 42.5 325:52 0 mysqld
1 root 15 0 500 500 440 S 0.0 0.0 1:28 1 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2
5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3
6 root 15 0 0 0 0 SW 0.0 0.0 0:23 3 keventd
7 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
8 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd/2
10 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd/3
11 root 15 0 0 0 0 SW 0.0 0.0 176:13 0 kswapd
12 root 15 0 0 0 0 SW 0.0 0.0 453:27 1 kscand
13 root 15 0 0 0 0 SW 0.0 0.0 2:57 1 bdflush
14 root 15 0 0 0 0 SW 0.0 0.0 20:26 1 kupdated
15 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 mdrecoveryd
22 root 25 0 0 0 0 SW 0.0 0.0 0:00 2 scsi_eh_0
25 root 15 0 0 0 0 SW 0.0 0.0 17:25 0 kjournald
71 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
576 root 15 0 0 0 0 SW 0.0 0.0 3:15 3 kjournald
577 root 15 0 0 0 0 SW 0.0 0.0 0:00 2 kjournald
578 root 15 0 0 0 0 SW 0.0 0.0 3:59 3 kjournald
579 root 15 0 0 0 0 SW 0.0 0.0 53:05 3 kjournald
580 root 15 0 0 0 0 SW 0.0 0.0 222:44 3 kjournald
581 root 15 0 0 0 0 SW 0.0 0.0 25:39 0 kjournald
582 root 15 0 0 0 0 SW 0.0 0.0 4:48 0 kjournald
583 root 15 0 0 0 0 SW 0.0 0.0 24:43 1 kjournald
911 root 15 0 580 580 496 S 0.0 0.0 1:06 1 syslogd
915 root 15 0 444 444 380 S 0.0 0.0 0:00 2 klogd
925 root 15 0 436 436 376 S 0.0 0.0 6:03 3 irqbalance
996 root 16 0 684 676 524 S 0.0 0.0 0:00 1 pppd
1112 root 15 0 2480 2480 996 S 0.0 0.0 58:51 2 snmpd
1119 root 15 0 736 736 536 S 0.0 0.0 80:46 1 percagent
1120 root 15 0 736 736 536 S 0.0 0.0 0:37 0 percagent
1121 root 15 0 736 736 536 S 0.0 0.0 5:03 0 percagent
1130 root 15 0 848 848 604 S 0.0 0.0 0:38 1 sshd
1147 ntp 15 0 2560 2560 2192 S 0.0 0.0 4:41 1 ntpd
1198 root 15 0 796 796 484 S 0.0 0.0 0:43 1 master
1205 postfix 15 0 864 864 532 S 0.0 0.0 0:43 0 nqmgr
1209 root 15 0 3348 3348 480 S 0.0 0.0 1:05 2 python
1218 root 18 0 580 580 488 S 0.0 0.0 1:48 2 crond
1227 daemon 15 0 496 496 420 S 0.0 0.0 0:05 2 atd
1333 root 15 0 0 0 0 SW 0.0 0.0 1:03 1 dcd_tmr_threa
1354 root 25 0 1388 1388 408 S 0.0 0.0 0:00 0 omsad32
1355 root 25 0 1388 1388 408 S 0.0 0.0 0:00 1 omsad32
1357 root 25 0 1388 1388 408 S 0.0 0.0 0:00 1 omsad32
1358 root 15 0 1388 1388 408 S 0.0 0.0 0:00 0 omsad32
1359 root 25 0 1388 1388 408 S 0.0 0.0 0:00 1 omsad32
1360 root 25 0 1388 1388 408 S 0.0 0.0 0:00 1 omsad32
1431 root 17 0 5124 5124 980 S 0.0 0.1 0:00 2 dcstor32d
1659 root 21 0 5124 5124 980 S 0.0 0.1 0:00 1 dcstor32d
1787 root 15 0 5124 5124 980 S 0.0 0.1 2:13 2 dcstor32d
1788 root 15 0 5124 5124 980 S 0.0 0.1 16:04 0 dcstor32d
1789 root 15 0 5124 5124 980 S 0.0 0.1 14:39 3 dcstor32d
1790 root 17 0 5124 5124 980 S 0.0 0.1 0:00 1 dcstor32d
1801 root 25 0 1100 1100 676 S 0.0 0.0 0:00 2 dcevt32d
1802 root 15 0 1100 1100 676 S 0.0 0.0 0:54 0 dcevt32d
1812 root 25 0 2088 2088 1048 S 0.0 0.0 0:00 2 dcsnmp32d
1813 root 15 0 2088 2088 1048 S 0.0 0.0 1:04 0 dcsnmp32d
1814 root 15 0 2088 2088 1048 S 0.0 0.0 0:49 3 dcsnmp32d
1815 root 15 0 2088 2088 1048 S 0.0 0.0 111:39 3 dcsnmp32d
1816 root 15 0 2088 2088 1048 S 0.0 0.0 14:33 0 dcsnmp32d
1841 root 25 0 484 484 384 S 0.0 0.0 0:00 0 omaws32
1842 root 15 0 31552 30M 2200 S 0.0 0.8 0:02 3 omaws32
1843 root 25 0 31552 30M 2200 S 0.0 0.8 0:00 2 omaws32
1844 root 15 0 31552 30M 2200 S 0.0 0.8 108:42 0 omaws32
1845 root 15 0 31552 30M 2200 S 0.0 0.8 0:03 1 omaws32
1846 root 15 0 31552 30M 2200 S 0.0 0.8 0:03 0 omaws32
1847 root 25 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1848 root 15 0 31552 30M 2200 S 0.0 0.8 0:00 3 omaws32
1849 root 25 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1850 root 15 0 31552 30M 2200 S 0.0 0.8 90:47 0 omaws32
1860 root 15 0 476 476 424 S 0.0 0.0 0:03 3 rhnsd
1868 root 18 0 396 396 344 S 0.0 0.0 0:00 2 mingetty
1869 root 18 0 392 392 344 S 0.0 0.0 0:00 0 mingetty
1870 root 19 0 392 392 344 S 0.0 0.0 0:00 0 mingetty
1871 root 19 0 392 392 344 S 0.0 0.0 0:00 3 mingetty
1872 root 19 0 392 392 344 S 0.0 0.0 0:00 3 mingetty
1873 root 19 0 392 392 344 S 0.0 0.0 0:00 1 mingetty
1875 root 21 0 31552 30M 2200 S 0.0 0.8 0:00 2 omaws32
1876 root 15 0 31552 30M 2200 S 0.0 0.8 0:06 2 omaws32
1877 root 15 0 31552 30M 2200 S 0.0 0.8 2:48 3 omaws32
1878 root 15 0 31552 30M 2200 S 0.0 0.8 0:32 2 omaws32
1887 root 15 0 31552 30M 2200 S 0.0 0.8 0:02 3 omaws32
1888 root 15 0 31552 30M 2200 S 0.0 0.8 0:02 0 omaws32
1889 root 23 0 31552 30M 2200 S 0.0 0.8 0:00 2 omaws32
1891 root 23 0 31552 30M 2200 S 0.0 0.8 0:00 3 omaws32
1892 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1893 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1894 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1895 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1896 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1897 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1898 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1899 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1900 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1901 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1902 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1903 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1904 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1905 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1906 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1907 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1908 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1909 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1910 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 1 omaws32
1911 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1912 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1913 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1914 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1915 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1916 root 20 0 31552 30M 2200 S 0.0 0.8 0:00 0 omaws32
1917 root 23 0 31552 30M 2200 S 0.0 0.8 0:00 3 omaws32
1918 root 23 0 31552 30M 2200 S 0.0 0.8 0:00 3 omaws32
1919 root 15 0 31552 30M 2200 S 0.0 0.8 0:23 1 omaws32
1921 root 15 0 31552 30M 2200 S 0.0 0.8 5:00 3 omaws32
1922 root 15 0 31552 30M 2200 S 0.0 0.8 5:56 1 omaws32
1923 root 15 0 31552 30M 2200 S 0.0 0.8 0:01 1 omaws32
1996 root 15 0 5124 5124 980 S 0.0 0.1 0:00 2 dcstor32d
2020 root 15 0 1388 1388 408 S 0.0 0.0 0:00 1 omsad32
2024 root 15 0 5124 5124 980 S 0.0 0.1 2:29 1 dcstor32d
2025 root 15 0 5124 5124 980 S 0.0 0.1 8:30 1 dcstor32d
2203 root 15 0 31552 30M 2200 S 0.0 0.8 0:50 2 omaws32
2204 root 15 0 31552 30M 2200 S 0.0 0.8 0:06 2 omaws32
2205 root 15 0 31552 30M 2200 S 0.0 0.8 0:03 1 omaws32
2206 root 15 0 31552 30M 2200 S 0.0 0.8 0:20 1 omaws32
27989 root 15 0 428 428 372 S 0.0 0.0 0:00 1 agetty
4902 root 15 0 1332 1332 756 S 0.0 0.0 28:53 0 watchmeminfo
1597 root 16 0 996 996 624 S 0.0 0.0 0:00 0 sshd
1599 mysql 15 0 1304 1304 808 S 0.0 0.0 2:53 0 sshd
1601 mysql 24 0 432 432 296 S 0.0 0.0 0:00 1 ksh
1627 mysql 15 0 1104 1104 884 S 0.0 0.0 0:00 0 bash
17440 root 20 0 996 996 632 S 0.0 0.0 0:00 0 sshd
17442 mysql 15 0 1116 1116 692 S 0.0 0.0 0:02 2 sshd
17446 mysql 16 0 428 428 296 S 0.0 0.0 0:00 3 ksh
19563 root 21 0 996 996 628 S 0.0 0.0 0:00 2 sshd
19565 mysql 15 0 1108 1108 680 S 0.0 0.0 0:02 2 sshd
19570 mysql 15 0 428 428 296 S 0.0 0.0 0:00 1 ksh
20577 mysql 15 0 956 956 624 S 0.0 0.0 0:02 3 ssh
25998 mysql 25 0 1000 1000 844 S 0.0 0.0 0:00 0 mysqld_safe
26040 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:29 1 mysqld
26041 mysql 16 0 1601M 1.6G 1780 S 0.0 42.5 0:12 1 mysqld
26042 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 69:40 1 mysqld
26277 mysql 15 0 1176 1176 724 S 0.0 0.0 90:38 1 top
14262 mysql 15 0 1000 1000 872 S 0.0 0.0 0:12 1 sh
21897 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 120:03 3 mysqld
4787 mysql 16 0 1601M 1.6G 1780 S 0.0 42.5 490:30 2 mysqld
22203 root 15 0 1292 1292 832 S 0.0 0.0 0:12 0 cupsd
21531 root 15 0 1396 1396 992 S 0.0 0.0 0:30 3 sshd
21537 root 25 0 1220 1220 1024 S 0.0 0.0 0:00 0 bash
21584 root 25 0 928 928 752 S 0.0 0.0 0:00 0 su
21585 mysql 25 0 584 584 444 S 0.0 0.0 0:00 0 ksh
21610 mysql 15 0 1244 1244 1020 S 0.0 0.0 0:00 3 bash
21935 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 31:05 2 mysqld
21995 mysql 15 0 1244 1244 796 S 0.0 0.0 14:57 1 top
22817 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 7:22 0 mysqld
26492 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:59 1 mysqld
4005 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:03 3 mysqld
7702 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:35 3 mysqld
7709 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:28 0 mysqld
7710 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:12 2 mysqld
7711 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:54 2 mysqld
7816 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:53 2 mysqld
9318 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:59 2 mysqld
11881 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:09 2 mysqld
13790 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:49 3 mysqld
13819 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 6:12 2 mysqld
13861 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:43 3 mysqld
13868 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:23 0 mysqld
15051 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:36 2 mysqld
15053 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:45 2 mysqld
15550 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:06 3 mysqld
15555 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:14 3 mysqld
16499 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 3:30 1 mysqld
16796 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:48 1 mysqld
16801 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 4:59 2 mysqld
17769 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:23 2 mysqld
17777 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:10 3 mysqld
17783 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:42 1 mysqld
17791 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:13 1 mysqld
17797 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:13 1 mysqld
17798 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:57 3 mysqld
18119 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:26 2 mysqld
18125 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:38 0 mysqld
18199 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:04 2 mysqld
19735 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:22 0 mysqld
19817 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 2:44 0 mysqld
19818 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:38 3 mysqld
20264 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:21 0 mysqld
20574 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:38 2 mysqld
21141 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:35 1 mysqld
21233 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:19 2 mysqld
21235 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:50 0 mysqld
21398 postfix 15 0 1060 1060 768 S 0.0 0.0 0:00 2 pickup
21864 mysql 17 0 5700 5700 2100 S 0.0 0.1 0:03 3 dbmon
22125 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:46 2 mysqld
22250 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:22 3 mysqld
22253 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:08 1 mysqld
24394 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:24 1 mysqld
24422 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:14 1 mysqld
24423 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:02 1 mysqld
24436 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:09 1 mysqld
24437 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 1:52 2 mysqld
24466 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:01 1 mysqld
24472 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:01 0 mysqld
24549 mysql 15 0 1601M 1.6G 1780 S 0.0 42.5 0:19 1 mysqld
26103 mysql 15 0 268 268 232 S 0.0 0.0 0:00 2 time
26104 mysql 23 0 568 568 360 S 0.0 0.0 0:00 2 mysql
26105 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26106 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26107 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26108 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26109 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26112 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26113 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26115 mysql 21 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26116 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26119 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26120 mysql 21 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26121 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26123 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26124 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26127 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26128 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26129 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26131 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26134 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26135 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26136 mysql 15 0 568 568 492 S 0.0 0.0 0:00 1 sleep
26137 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26139 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26140 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26141 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26142 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26145 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26146 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26147 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26150 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26151 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26152 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26155 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26156 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26157 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26158 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26160 mysql 21 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26161 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26164 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26166 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26167 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26170 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26172 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26173 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26174 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26177 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26178 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26179 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26181 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26182 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26183 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26184 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26187 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26188 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26189 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26190 mysql 21 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26192 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26193 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26194 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26197 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26199 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26200 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26201 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 0 mysqld
26202 mysql 20 0 996 996 888 S 0.0 0.0 0:00 0 sh
26204 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
26205 mysql 20 0 1601M 1.6G 1780 S 0.0 42.5 0:00 1 mysqld
Here is show processlist with lot of SELECT 1 waiting and other select statements usually take sub-second are waiting with lot of time:
Id User Host db Command Time State Info
18 system user Connect 1187101 Waiting for master to send event
113793 repl d37somedomain.com:55819 Binlog Dump 691083 Has sent all binlog to slave; waiting for binlog to be updated
149752 system user silo_s10 Connect 29 end update TotalAggregateVal set reference=63921, version=2747, aggType=1116, bucketType=1, bucketVal=0, val=103035, sequence=3702397646, minDate='2006-06-14 14:50:47', maxDate='2006-06-14 15:05:13', createDate='2006-06-08 08:47:45' where id=4205914 and version=2746
256768 repl d37somedomain.com:59159 Binlog Dump 168515 Has sent all binlog to slave; waiting for binlog to be updated
331779 mysql localhost silo_s10 Query 0 show full processlist
333473 user-mmemdb d37somedomain.com:52050 silo_s10 Sleep 33
333498 user a14somedomain.com:34630 silo_s10 Query 25 select 1
333514 user-mmemdb d37somedomain.com:52133 silo_s10 Query 26 select 1
333517 user-mmemdb d37somedomain.com:52139 silo_s10 Sleep 29
333533 user-mmemdb d37somedomain.com:52236 silo_s10 Query 9 select 1
333557 user-mmemdb d37somedomain.com:52278 silo_s10 Query 24 select 1
333646 user a14somedomain.com:34861 silo_s10 Query 27 select 1
333651 user a14somedomain.com:34868 silo_s10 Query 26 select 1
333662 user a14somedomain.com:34896 silo_s10 Query 26 select 1
333668 user a14somedomain.com:34904 silo_s10 Query 25 select 1
333673 user-mdbro a15somedomain.com:38120 silo_s10 Query 9 select 1
333678 user a14somedomain.com:34909 silo_s10 Query 27 select 1
333699 user-mmemdb d37somedomain.com:52708 silo_s10 Query 24 select 1
333704 user a14somedomain.com:34938 silo_s10 Query 26 select 1
333720 user-mmemdb d37somedomain.com:52742 silo_s10 Sleep 33
333735 user-mdbro a15somedomain.com:38301 silo_s10 Query 9 select 1
333740 user-mmemdb d37somedomain.com:52762 silo_s10 Query 27 select 1
333741 user-mmemdb d37somedomain.com:52764 silo_s10 Query 21 select 1
333745 user-mmemdb d37somedomain.com:52795 silo_s10 Query 4 select 1
333746 user-mmemdb d37somedomain.com:52796 silo_s10 Query 1 select 1
333747 user-mmemdb d37somedomain.com:52797 silo_s10 Sleep 33
333749 user-mmemdb d37somedomain.com:52799 silo_s10 Sleep 32
333757 user a14somedomain.com:35008 silo_s10 Query 25 select 1
333758 user a14somedomain.com:35009 silo_s10 Query 26 select 1
333760 user a14somedomain.com:35010 silo_s10 Query 28 select 1
333761 user a14somedomain.com:35011 silo_s10 Query 26 select 1
333762 user a14somedomain.com:35012 silo_s10 Query 29 update insert into MemberEmailHistory (email, endDate, member_id, emailVersion) values ('gemm@comcast.net', '2006-06-14 15:09:20', 61689589, 1)
333764 user a14somedomain.com:35013 silo_s10 Query 28 select 1
333765 user a14somedomain.com:35015 silo_s10 Query 28 select 1
333785 user-mmemdb d37somedomain.com:52917 silo_s10 Query 1 select 1
333791 user a14somedomain.com:35025 silo_s10 Query 25 select 1
333807 user-mmemdb d37somedomain.com:52954 silo_s10 Sleep 32
333808 user a14somedomain.com:35033 silo_s10 Query 25 select 1
333810 user-mmemdb d37somedomain.com:52957 silo_s10 Sleep 33
333817 user a14somedomain.com:35050 silo_s10 Query 27 select 1
333818 user a14somedomain.com:35051 silo_s10 Query 26 select 1
333819 user a14somedomain.com:35052 silo_s10 Query 25 select 1
333832 user-mmemdb d37somedomain.com:52984 silo_s10 Sleep 33
333847 user-mmemdb d37somedomain.com:53004 silo_s10 Sleep 29
333848 user-mmemdb d37somedomain.com:53005 silo_s10 Query 9 select 1
333857 user a14somedomain.com:35084 silo_s10 Query 29 Waiting on cond drop table if exists test.LocalMemberPageOfMatchingIds
333859 user a14somedomain.com:35085 silo_s10 Query 25 select count(*) from BusinessEntity_CellAttributes
333860 user a14somedomain.com:35086 silo_s10 Query 25 select count(*) from Cell
333861 user a14somedomain.com:35087 silo_s10 Query 25 select count(*) from Ad
333862 user a14somedomain.com:35089 silo_s10 Query 25 select count(*) from CellAttribute
333863 mysql localhost silo_s10 Query 25 select 1
333864 user a14somedomain.com:35090 silo_s10 Query 25