Re: Partitions and performance

[mahesh karipe]

Thanks Rick,

> >> KEY `idx_big`
>It's a tradeoff...

> >> Key_reads/Key_read_requests = 98%
>That's terrible. It says that 98% of the time it has to go to disk.

> Are ther 2.6M rows in tmp_qfact_p2? How many of these tables do you have?

> A summary table will _probably_ speed up the queries 10-fold, and avoid blowing about the key_buffer.

This application contains seven dimensions, out of this seven five dimensions are mapped to the main fact table(patients_main_fact), these five are very crutial. If we can tune this five dimension fact table then, we can propose to make other two dimensions available at some filter level.

I have used the name 'tmp'as I am trying with sub set of the data as there are many measures associated with the actual table( but stats are from actual table).

This main fact table is of size around 2.5 GB, not sure if we can cache this entire table( to reduce dist I/Os)and run the join queries along with dimension table! (not to forget we have 16 GB RAM).

If we can do this we can even get rid of summary tables too.(we have two summary tables on top of this patient_month_summary,patient_quater_summary)

I have good faith in MYSQL, I strongly belive it can perform well on just 2.5 GB table using 16 GB RAM!!, may be my knowledge is strugling to make it run ( I have experice with Oralce DB, using MYSQL for the first time!!).


here are the stats...


Variable_name : Value
archive_aio : OFF
auto_increment_increment : 1
auto_increment_offset : 1
autocommit : ON
automatic_sp_privileges : ON
back_log : 50
backup_history_log : ON
backup_history_log_file : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\backup_history.log
backup_progress_log : ON
backup_progress_log_file : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\backup_progress.log
backup_wait_timeout : 50
backupdir : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\
basedir : C:\\Program Files\\MySQL\\MySQL Server 6.0\\
big_tables : OFF
binlog_cache_size : 32768
binlog_format : MIXED
bulk_insert_buffer_size : 268435456
character_set_client : utf8
character_set_connection : utf8
character_set_database : utf8
character_set_filesystem : binary
character_set_results : utf8
character_set_server : latin1
character_set_system : utf8
character_sets_dir : C:\\Program Files\\MySQL\\MySQL Server 6.0\\share\\charsets\\
collation_connection : utf8_general_ci
collation_database : utf8_bin
collation_server : latin1_swedish_ci
completion_type : 0
concurrent_insert : 1
connect_timeout : 10
datadir : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\
date_format : %Y-%m-%d
datetime_format : %Y-%m-%d %H:%i:%s
deadlock_search_depth_long : 15
deadlock_search_depth_short : 4
deadlock_timeout_long : 50000000
deadlock_timeout_short : 10000
default_week_format : 0
delay_key_write : ON
delayed_insert_limit : 100
delayed_insert_timeout : 300
delayed_queue_size : 1000
div_precision_increment : 4
engine_condition_pushdown : ON
error_count : 0
event_scheduler : OFF
expire_logs_days : 0
falcon_checkpoint_schedule : 7 * * * * *
falcon_checksums : ON
falcon_consistent_read : ON
falcon_debug_mask : 0
falcon_debug_server : OFF
falcon_debug_trace : 0
falcon_direct_io : 1
falcon_gopher_threads : 5
falcon_index_chill_threshold : 4194304
falcon_io_threads : 2
falcon_large_blob_threshold : 160000
falcon_lock_wait_timeout : 50
falcon_page_cache_size : 4194304
falcon_page_size : 4096
falcon_record_chill_threshold : 5242880
falcon_record_memory_max : 262144000
falcon_record_scavenge_floor : 50
falcon_record_scavenge_threshold : 67
falcon_scavenge_schedule : 15,45 * * * * *
falcon_serial_log_block_size : 0
falcon_serial_log_buffers : 20
falcon_serial_log_dir : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\
falcon_serial_log_file_size : 10485760
falcon_serial_log_priority : 1
falcon_support_xa : OFF
falcon_use_deferred_index_hash : OFF
falcon_use_sectorcache : OFF
falcon_use_supernodes : ON
flush : OFF
flush_time : 1800
foreign_key_checks : ON
ft_boolean_syntax : + -><()~*:""&|
ft_max_word_len : 84
ft_min_word_len : 4
ft_query_expansion_limit : 20
ft_stopword_file : (built-in)
general_log : OFF
general_log_file : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\mp441198.log
group_concat_max_len : 1024
have_community_features : NO
have_compress : YES
have_crypt : NO
have_csv : YES
have_dynamic_loading : YES
have_geometry : YES
have_innodb : YES
have_ndbcluster : NO
have_openssl : DISABLED
have_partitioning : YES
have_query_cache : YES
have_rtree_keys : YES
have_ssl : DISABLED
have_symlink : YES
hostname : mp441198
identity : 0
init_connect :
init_file :
init_slave :
innodb_adaptive_hash_index : ON
innodb_additional_mem_pool_size : 268435456
innodb_autoextend_increment : 8
innodb_autoinc_lock_mode : 1
innodb_buffer_pool_size : 1073741824
innodb_checksums : ON
innodb_commit_concurrency : 0
innodb_concurrency_tickets : 500
innodb_data_file_path : ibdata1:10M:autoextend
innodb_data_home_dir :
innodb_doublewrite : ON
innodb_fast_shutdown : 1
innodb_file_io_threads : 4
innodb_file_per_table : OFF
innodb_flush_log_at_trx_commit : 1
innodb_flush_method :
innodb_force_recovery : 0
innodb_lock_wait_timeout : 50
innodb_locks_unsafe_for_binlog : OFF
innodb_log_buffer_size : 1048576
innodb_log_file_size : 10485760
innodb_log_files_in_group : 2
innodb_log_group_home_dir : .\\
innodb_max_dirty_pages_pct : 90
innodb_max_purge_lag : 0
innodb_mirrored_log_groups : 1
innodb_open_files : 300
innodb_rollback_on_timeout : OFF
innodb_support_xa : ON
innodb_sync_spin_loops : 20
innodb_table_locks : ON
innodb_thread_concurrency : 18
innodb_thread_sleep_delay : 10000
insert_id : 0
interactive_timeout : 28800
join_buffer_size : 536870912
keep_files_on_create : OFF
key_buffer_size : 2147483648
key_cache_age_threshold : 300
key_cache_block_size : 1024
key_cache_division_limit : 100
language : C:\\Program Files\\MySQL\\MySQL Server 6.0\\share\\english\\
large_files_support : ON
large_page_size : 0
large_pages : OFF
last_insert_id : 0
lc_time_names : en_US
license : GPL
local_infile : ON
log : OFF
log_backup_output : TABLE
log_bin : OFF
log_bin_trust_function_creators : OFF
log_error : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\mp441198.err
log_output : FILE
log_queries_not_using_indexes : OFF
log_slave_updates : OFF
log_slow_queries : OFF
log_warnings : 1
long_query_time : 10.000000
low_priority_updates : OFF
lower_case_file_system : ON
lower_case_table_names : 1
maria_block_size : 8192
maria_checkpoint_interval : 30
maria_force_start_after_recovery_failures : 0
maria_log_file_size : 1073741824
maria_log_purge_type : immediate
maria_max_sort_file_size : 9223372036854775807
maria_page_checksum : ON
maria_pagecache_age_threshold : 300
maria_pagecache_buffer_size : 8388600
maria_pagecache_division_limit : 100
maria_recover : OFF
maria_repair_threads : 1
maria_sort_buffer_size : 8388608
maria_stats_method : nulls_unequal
maria_sync_log_dir : NEWFILE
max_allowed_packet : 1073741824
max_binlog_cache_size : 4294967295
max_binlog_size : 1073741824
max_connect_errors : 10
max_connections : 100
max_delayed_threads : 20
max_error_count : 64
max_heap_table_size : 16777216
max_insert_delayed_threads : 20
max_join_size : 18446744073709551615
max_length_for_sort_data : 1024
max_prepared_stmt_count : 16382
max_relay_log_size : 0
max_seeks_for_key : 4294967295
max_sort_length : 1024
max_sp_recursion_depth : 0
max_tmp_tables : 32
max_user_connections : 0
max_write_lock_count : 4294967295
min_examined_row_limit : 0
myisam_data_pointer_size : 6
myisam_max_sort_file_size : 2147483647
myisam_recover_options : OFF
myisam_repair_threads : 1
myisam_sort_buffer_size : 773849088
myisam_stats_method : nulls_unequal
myisam_use_mmap : OFF
named_pipe : OFF
net_buffer_length : 16384
net_read_timeout : 30
net_retry_count : 10
net_write_timeout : 60
new : OFF
old : OFF
old_alter_table : OFF
old_passwords : OFF
open_files_limit : 2670
optimizer_prune_level : 1
optimizer_search_depth : 62
optimizer_switch :
optimizer_use_mrr : force
pid_file : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\mp441198.pid
plugin_dir : C:\\Program Files\\MySQL\\MySQL Server 6.0\\lib/plugin
port : 3306
preload_buffer_size : 32768
profiling : OFF
profiling_history_size : 15
protocol_version : 10
pseudo_thread_id : 179
query_alloc_block_size : 8192
query_cache_limit : 1048576
query_cache_min_res_unit : 4096
query_cache_size : 4284481536
query_cache_type : ON
query_cache_wlock_invalidate : OFF
query_prealloc_size : 8192
rand_seed1 :
rand_seed2 :
range_alloc_block_size : 4096
read_buffer_size : 536870912
read_only : OFF
read_rnd_buffer_size : 536870912
relay_log :
relay_log_index :
relay_log_info_file : relay-log.info
relay_log_purge : ON
relay_log_space_limit : 0
report_host :
report_password :
report_port : 3306
report_user :
rpl_recovery_rank : 0
secure_auth : OFF
secure_file_priv :
server_id : 0
shared_memory : OFF
shared_memory_base_name : MYSQL
skip_external_locking : ON
skip_networking : OFF
skip_show_database : OFF
slave_allow_batching : OFF
slave_compressed_protocol : OFF
slave_exec_mode : STRICT
slave_load_tmpdir : C:\\Windows\\TEMP
slave_net_timeout : 3600
slave_skip_errors : OFF
slave_transaction_retries : 10
slow_launch_time : 2
slow_query_log : OFF
slow_query_log_file : C:\\ProgramData\\MySQL\\MySQL Server 6.0\\Data\\mp441198-slow.log
sort_buffer_size : 536870912
sql_auto_is_null : ON
sql_big_selects : ON
sql_big_tables : OFF
sql_buffer_result : OFF
sql_log_bin : ON
sql_log_off : OFF
sql_log_update : ON
sql_low_priority_updates : OFF
sql_max_join_size : 18446744073709551615
sql_mode :
sql_notes : ON
sql_quote_show_create : ON
sql_safe_updates : OFF
sql_select_limit : 18446744073709551615
sql_slave_skip_counter :
sql_warnings : OFF
ssl_ca :
ssl_capath :
ssl_cert :
ssl_cipher :
ssl_key :
storage_engine : MyISAM
sync_binlog : 0
sync_frm : ON
system_time_zone : Eastern Standard Time
table_definition_cache : 256
table_lock_wait_timeout : 50
table_open_cache : 256
thread_cache_size : 20
thread_handling : one-thread-per-connection
thread_stack : 262144
time_format : %H:%i:%s
time_zone : SYSTEM
timed_mutexes : OFF
timestamp : 1392636477
tmp_table_size : 1572864000
tmpdir : C:\\Windows\\TEMP
transaction_alloc_block_size : 8192
transaction_prealloc_size : 4096
tx_isolation : REPEATABLE-READ
unique_checks : ON
updatable_views_with_limit : YES
version : 6.0.8-alpha-community
version_comment : MySQL Community Server (GPL)
version_compile_machine : unknown
version_compile_os : Win64
wait_timeout : 28800
warning_count : 0


Please advice further

Thanks
Mahesh