MySQL Forums
Forum List  »  Performance

Slow performing query - why?
Posted by: Derek Burket
Date: October 27, 2005 03:13PM

Hello,

I maintain a web-based app that allows users to select what criteria they want to search by, then it creates the SQL and executes it. Currently, it performs very poorly...using correlated subqueries. So I'm trying to optimize the SQL it generates. Query1 (below) is an example. While trying to optimize this, I came up with query2. I thought that using joins instead of correlated subqueries would give better performance, but that does not appear to be the case. The EXPLAIN plan for query2 looks good, so I can't understand why query1 runs in 18 seconds while query2 takes 10+ minutes to run. I'm doing some benchmarking, so I've turned off query caching and loaded the keys into their own named key cache.

Does anyone know why query2 would take so long to run? Is there a better way to optimize query1? I'm open to any suggestions. See details below.


/* query1 with explain */
SELECT COUNT(u.user_id)
FROM USER_OPTOUT_CMP u where u.list_id IN (445600)
AND
(
(NOT EXISTS (SELECT 1
FROM
USER_OPTOUT_CMP ula1
WHERE (u.user_id = ula1.user_id AND
ula1.list_id IN (430663) )) )
AND
(NOT EXISTS (SELECT 1
FROM
USER_OPTOUT_CMP ula2
WHERE (u.user_id = ula2.user_id AND
ula2.list_id IN (897626) )) )
AND
(EXISTS (SELECT 1
FROM USER_MAG_DEMO_CMP umd
WHERE u.user_id = umd.user_id
AND umd.magazine_id = 1
and (umd.question_id, umd.answer_id) IN ((17,18),(17,19),(17,20),(17,16),(17,17))))
AND
(EXISTS (SELECT 1
FROM USER_MAG_DEMO_CMP umd2
WHERE u.user_id = umd2.user_id
and umd2.magazine_id = 1
and (umd2.question_id, umd2.answer_id) IN ((30,5),(30,7),(30,2),(30,4),(30,1),(30,3))))
AND
(EXISTS (SELECT 1
FROM USER_MAG_DEMO_CMP umd3
WHERE u.user_id = umd3.user_id
and umd3.magazine_id = 1
and (umd3.question_id, umd3.answer_id) IN ((42,65),(42,73),(42,74),(42,68),(42,66),(42,64))))
AND
(NOT EXISTS (SELECT 1
FROM
USER_NAMED_LIST_CMP unl
WHERE (u.user_id = unl.user_id AND
unl.named_list_id IN (629097, 629323, 629412, 627422, 627605))))
)

+----+--------------------+-------+--------+-------------------------------+---------+---------+----------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+--------+-------------------------------+---------+---------+----------------------------+-------+--------------------------+
| 1 | PRIMARY | u | ref | list_id | list_id | 3 | const | 86802 | Using where |
| 7 | DEPENDENT SUBQUERY | unl | ref | PRIMARY,user_id,named_list_id | PRIMARY | 3 | listpuller.u.user_id | 13 | Using where; Using index |
| 6 | DEPENDENT SUBQUERY | umd3 | ref | PRIMARY,user_id,magazine_id | PRIMARY | 4 | listpuller.u.user_id,const | 23 | Using where; Using index |
| 5 | DEPENDENT SUBQUERY | umd2 | ref | PRIMARY,user_id,magazine_id | PRIMARY | 4 | listpuller.u.user_id,const | 23 | Using where; Using index |
| 4 | DEPENDENT SUBQUERY | umd | ref | PRIMARY,user_id,magazine_id | PRIMARY | 4 | listpuller.u.user_id,const | 23 | Using where; Using index |
| 3 | DEPENDENT SUBQUERY | ula2 | eq_ref | PRIMARY,list_id,user_id | PRIMARY | 6 | listpuller.u.user_id,const | 1 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | ula1 | eq_ref | PRIMARY,list_id,user_id | PRIMARY | 6 | listpuller.u.user_id,const | 1 | Using where; Using index |
+----+--------------------+-------+--------+-------------------------------+---------+---------+----------------------------+-------+--------------------------+



/* query2 with explain */
SELECT COUNT(DISTINCT uo.user_id)
FROM USER_OPTOUT_CMP uo
JOIN USER_OPTOUT_CMP uo2 USING (user_id)
JOIN USER_OPTOUT_CMP uo3 USING (user_id)
JOIN USER_MAG_DEMO_CMP umd USING (user_id)
JOIN USER_MAG_DEMO_CMP umd1 USING (user_id)
JOIN USER_MAG_DEMO_CMP umd2 USING (user_id)
JOIN USER_NAMED_LIST_CMP unla USING (user_id)
WHERE uo.list_id = 445600
AND uo2.list_id != 430663
AND uo3.list_id != 897626
AND (umd.magazine_id, umd.question_id, umd.answer_id) IN ((1,17,18), (1,17,19), (1,17,20), (1,17,16), (1,17,17))
AND (umd1.magazine_id, umd1.question_id, umd1.answer_id) IN ((1,30,7), (1,30,2), (1,30,4), (1,30,1), (1,30,3))
AND (umd2.magazine_id, umd2.question_id, umd2.answer_id) IN ((1,42,65), (1,42,73), (1,42,74), (1,42,68), (1,42,66), (1,42,64))
AND unla.named_list_id NOT IN (629097, 629323, 629412, 627422, 627605)

+----+-------------+-------+------+-------------------------+---------+---------+-------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+-------------------------+---------+---------+-------------------------+-------+--------------------------+
| 1 | SIMPLE | uo | ref | PRIMARY,list_id,user_id | list_id | 3 | const | 86802 | Using where |
| 1 | SIMPLE | uo2 | ref | PRIMARY,list_id,user_id | PRIMARY | 3 | listpuller.uo.user_id | 11 | Using where; Using index |
| 1 | SIMPLE | uo3 | ref | PRIMARY,list_id,user_id | PRIMARY | 3 | listpuller.uo2.user_id | 11 | Using where; Using index |
| 1 | SIMPLE | umd | ref | PRIMARY,user_id | PRIMARY | 3 | listpuller.uo3.user_id | 25 | Using where; Using index |
| 1 | SIMPLE | umd1 | ref | PRIMARY,user_id | PRIMARY | 3 | listpuller.umd.user_id | 25 | Using where; Using index |
| 1 | SIMPLE | umd2 | ref | PRIMARY,user_id | PRIMARY | 3 | listpuller.umd1.user_id | 25 | Using where; Using index |
| 1 | SIMPLE | unla | ref | PRIMARY,user_id | PRIMARY | 3 | listpuller.umd2.user_id | 13 | Using where; Using index |
+----+-------------+-------+------+-------------------------+---------+---------+-------------------------+-------+--------------------------+




Here are the table defs:

CREATE TABLE `USER_OPTOUT_CMP` (
`user_id` mediumint(8) unsigned NOT NULL default '0',
`list_id` mediumint(8) unsigned NOT NULL default '0',
`list_type_cd` tinyint(3) unsigned NOT NULL default '0',
PRIMARY KEY (`user_id`,`list_id`),
KEY `list_id` (`list_id`),
KEY `user_id` (`user_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 PACK_KEYS=1

CREATE TABLE `USER_MAG_DEMO_CMP` (
`user_id` mediumint(8) unsigned NOT NULL default '0',
`magazine_id` tinyint(3) unsigned NOT NULL default '0',
`question_id` smallint(5) unsigned NOT NULL default '0',
`answer_id` smallint(5) unsigned NOT NULL default '0',
`audited_ind` char(1) NOT NULL default '',
PRIMARY KEY (`user_id`,`magazine_id`,`question_id`,`answer_id`),
KEY `user_id` (`user_id`),
KEY `magazine_id` (`magazine_id`,`question_id`,`answer_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 PACK_KEYS=1

CREATE TABLE `USER_NAMED_LIST_CMP` (
`user_id` mediumint(8) unsigned NOT NULL default '0',
`named_list_id` mediumint(8) unsigned NOT NULL default '0',
PRIMARY KEY (`user_id`,`named_list_id`),
KEY `user_id` (`user_id`),
KEY `named_list_id` (`named_list_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 PACK_KEYS=1

Other info:

mysql> show processlist;
+------+------------+--------------------------------------+------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------------+--------------------------------------+------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| 1652 | listpuller | localhost | listpuller | Query | 1915 | Sending data | SELECT COUNT(DISTINCT uo.user_id) |
| 1957 | es | localhost | es | Sleep | 55 | | NULL |
| 1979 | es | localhost | es | Sleep | 4348 | | NULL |
| 2052 | listpuller | localhost | listpuller | Query | 0 | NULL | show processlist |
| 2055 | es_app | mailhost8.lists.techtarget.com:48667 | es | Sleep | 41 | | NULL |
+------+------------+--------------------------------------+------------+---------+------+--------------+------------------------------------------------------------------------------------------------------+

Server status:
Variable_name Value
Aborted_clients 379
Aborted_connects 57
Binlog_cache_disk_use 6947
Binlog_cache_use 6954
Bytes_received 7803128994
Bytes_sent 13552604323
Com_admin_commands 0
Com_alter_db 0
Com_alter_table 69
Com_analyze 23
Com_backup_table 0
Com_begin 0
Com_change_db 45
Com_change_master 0
Com_check 0
Com_checksum 0
Com_commit 11650
Com_create_db 0
Com_create_function 0
Com_create_index 6304
Com_create_table 162
Com_dealloc_sql 0
Com_delete 17
Com_delete_multi 0
Com_do 0
Com_drop_db 0
Com_drop_function 0
Com_drop_index 0
Com_drop_table 156
Com_drop_user 0
Com_execute_sql 0
Com_flush 12
Com_grant 27
Com_ha_close 0
Com_ha_open 0
Com_ha_read 0
Com_help 0
Com_insert 29388912
Com_insert_select 2
Com_kill 6
Com_load 0
Com_load_master_data 0
Com_load_master_table 0
Com_lock_tables 35
Com_optimize 2
Com_preload_keys 23
Com_prepare_sql 0
Com_purge 0
Com_purge_before_date 0
Com_rename_table 0
Com_repair 0
Com_replace 0
Com_replace_select 0
Com_reset 0
Com_restore_table 0
Com_revoke 0
Com_revoke_all 0
Com_rollback 43
Com_savepoint 0
Com_select 1501
Com_set_option 1654
Com_show_binlog_events 0
Com_show_binlogs 0
Com_show_charsets 0
Com_show_collations 262
Com_show_column_types 0
Com_show_create_db 12
Com_show_create_table 147
Com_show_databases 12
Com_show_errors 0
Com_show_fields 376
Com_show_grants 0
Com_show_innodb_status 0
Com_show_keys 11
Com_show_logs 0
Com_show_master_status 1036
Com_show_new_master 0
Com_show_open_tables 1
Com_show_privileges 0
Com_show_processlist 1072
Com_show_slave_hosts 0
Com_show_slave_status 1036
Com_show_status 16
Com_show_storage_engines 0
Com_show_tables 217
Com_show_variables 274
Com_show_warnings 54
Com_slave_start 0
Com_slave_stop 0
Com_truncate 0
Com_unlock_tables 43
Com_update 5444900
Com_update_multi 0
Connections 2064
Created_tmp_disk_tables 0
Created_tmp_files 275
Created_tmp_tables 188
Delayed_errors 0
Delayed_insert_threads 0
Delayed_writes 0
Flush_commands 13
Handler_commit 7009
Handler_delete 2
Handler_discover 0
Handler_read_first 765
Handler_read_key 216141180
Handler_read_next 4187778765
Handler_read_prev 0
Handler_read_rnd 15904
Handler_read_rnd_next 6307182157
Handler_rollback 2351
Handler_update 12716507
Handler_write 1423402560
Key_blocks_not_flushed 0
Key_blocks_unused 254263
Key_blocks_used 69096
Key_read_requests 16252947
Key_reads 16252947
Key_write_requests 1431900
Key_writes 1431900
Max_used_connections 28
Not_flushed_delayed_rows 0
Open_files 151
Open_streams 0
Open_tables 151
Opened_tables 19707
Qcache_free_blocks 1
Qcache_free_memory 0
Qcache_hits 18
Qcache_inserts 16
Qcache_lowmem_prunes 0
Qcache_not_cached 0
Qcache_queries_in_cache 0
Qcache_total_blocks 0
Questions 34863005
Rpl_status NULL
Select_full_join 0
Select_full_range_join 0
Select_range 87
Select_range_check 0
Select_scan 614
Slave_open_temp_tables 0
Slave_running OFF
Slow_launch_threads 0
Slow_queries 291
Sort_merge_passes 285
Sort_range 38
Sort_rows 71215613
Sort_scan 68
Table_locks_immediate 34672837
Table_locks_immediate 34672837
Table_locks_waited 159200
Threads_cached 3
Threads_connected 6
Threads_created 138
Threads_running 2
Uptime 193637

Server Variables:
Variable_name Value
back_log 50
basedir /
binlog_cache_size 32768
bulk_insert_buffer_size 8388608
character_set_client latin1
character_set_connection latin1
character_set_database utf8
character_set_results latin1
character_set_server utf8
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
collation_connection latin1_swedish_ci
collation_database utf8_general_ci
collation_server utf8_general_ci
concurrent_insert ON
connect_timeout 5
datadir /apps/mysql/db/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
expire_logs_days 0
flush OFF
flush_time 0
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
group_concat_max_len 1024
have_archive YES
have_bdb NO
have_compress YES
have_crypt YES
have_csv NO
have_example_engine NO
have_geometry YES
have_innodb YES
have_isam NO
have_ndbcluster NO
have_openssl NO
have_query_cache YES
have_raid NO
have_rtree_keys YES
have_symlink YES
init_connect
init_file
init_slave
innodb_additional_mem_pool_size 20971520
innodb_autoextend_increment 8
innodb_buffer_pool_awe_mem_mb 0
innodb_buffer_pool_size 402653184
innodb_data_file_path ibdata1:2000M;ibdata2:10M:autoextend
innodb_data_home_dir /apps/mysql/db/
innodb_fast_shutdown ON
innodb_file_io_threads 4
innodb_file_per_table OFF
innodb_flush_log_at_trx_commit 1
innodb_flush_method
innodb_force_recovery 0
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_arch_dir /apps/mysql/db/
innodb_log_archive OFF
innodb_log_buffer_size 8388608
innodb_log_file_size 104857600
innodb_log_files_in_group 2
innodb_log_group_home_dir /apps/mysql/db/
innodb_max_dirty_pages_pct 90
innodb_max_purge_lag 0
innodb_mirrored_log_groups 1
innodb_open_files 300
innodb_table_locks ON
innodb_thread_concurrency 8
interactive_timeout 28800
join_buffer_size 131072
key_buffer_size 8
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
language /usr/share/mysql/english/
large_files_support ON
license GPL
local_infile ON
locked_in_memory OFF
log OFF
log_bin ON
log_error
log_slave_updates OFF
log_slow_queries OFF
log_update OFF
log_warnings 1
long_query_time 10
low_priority_updates OFF
lower_case_file_system OFF
lower_case_table_names 0
max_allowed_packet 1047552
max_binlog_cache_size 18446744073709551615
max_binlog_size 1073741824
max_connect_errors 10
max_connections 100
max_delayed_threads 20
max_error_count 64
max_heap_table_size 4194302976
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_relay_log_size 0
max_seeks_for_key 18446744073709551615
max_sort_length 1024
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 18446744073709551615
myisam_data_pointer_size 4
myisam_max_extra_sort_file_size 2147483648
myisam_max_sort_file_size 9223372036854775807
myisam_recover_options OFF
myisam_repair_threads 1
myisam_sort_buffer_size 67108864
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
new OFF
old_passwords OFF
open_files_limit 1134
pid_file /apps/mysql/db/largo.techtarget.com.pid
port 3306
preload_buffer_size 32768
protocol_version 10
query_alloc_block_size 8192
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 0
query_cache_type ON
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 2048
read_buffer_size 2093056
read_only OFF
read_rnd_buffer_size 8384512
relay_log_purge ON
rpl_recovery_rank 0
secure_auth OFF
server_id 1
skip_external_locking ON
skip_networking OFF
skip_show_database OFF
slave_net_timeout 3600
slow_launch_time 2
socket /apps/mysql/mysql.sock
sort_buffer_size 2097144
sql_mode
storage_engine MyISAM
sync_binlog 0
sync_replication 0
sync_replication_slave_id 0
sync_replication_timeout 0
sync_frm ON
system_time_zone EDT
table_cache 512
table_type MyISAM
thread_cache_size 8
thread_stack 196608
time_format %H:%i:%s
time_zone SYSTEM
tmp_table_size 33554432
tmpdir /apps/mysql/tmp
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation REPEATABLE-READ
version 4.1.10a-pro-gpl-log
version_comment MySQL Pro Certified Server (GPL)
version_compile_machine x86_64
version_compile_os unknown-linux-gnu
wait_timeout 28800

Options: ReplyQuote


Subject
Views
Written By
Posted
Slow performing query - why?
1930
October 27, 2005 03:13PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.