MySQL Forums
Forum List  »  Performance

MySQL performance help
Posted by: Arun Shankar
Date: January 07, 2012 06:55AM

Dear All:
I am quite new to MySQL and have a task of making a huge database run fast in all ways :)

My sincere apologies if the data below is not in the expected format. Also, please let me know if you might need additional information.

What I am looking for is any changes that I can do in the my.cnf file to improve performance of the query without hampering the insertion speed. Also if there is are any design best practices for such scenario.

The system:
MySQL 5.1 / 64 bit
The database goes through ~50k row insertion per second. The average row size is 380. All the columns are big int unsigned not null.
The available memory is 4 GB.
Disk size 1.2 TB RAID 1.
The table is partitioned for every 1 hour, example: YYYYMMDDHH. There are no indexes on the table except for the large composite key with 5 columns starting with time stamp column as the first column in the key.
Engine: InnoDB only

The challenge:
The data that is getting inserted every seconds needs to be aggregated at the end of 1 hour for hourly summarization.
The hourly summarized data needs to be aggregated again at the end of the day for daily summarization.
There are set queries(procedures) that can access data for real time analytics. The duration can be for last 10 minutes, last 1 hour and sometimes for a custom time interval which is the past (Custom reports).

Current status:
Insertion speed seems to be good with around 2 seconds for each file.
Query performance for 5 minutes seems to be just fine. Anything beyond 5 minutes is completely not in the acceptable range.

Below is the My.cnf file snapshot:

# Optimized mysql configuration file with MySQL support inputs

# The following options will be passed to all MySQL clients
[client]
port = 3306
socket = /tmp/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 1024
sort_buffer_size = 1024M
read_buffer_size = 256M
read_rnd_buffer_size = 64M
thread_cache_size = 8
#query_cache_size= 32M
join_buffer_size = 12M
table_cache=5000
# Temp directory changed
tmpdir = /usr/local/var/tmp
# Maximum # of connections
max_connections = 51
# Max heap table size
max_heap_table_size = 1024M
tmp_table_size=1024M
query_prealloc_size=16384
query_alloc_block_size=16384
query_cache_size= 0
query_cache_type=0
transaction-isolation = READ-UNCOMMITTED
max_sort_length=2048

# Set the open_files_limit to a higher number(partition with the innodb_file_per_table
# option enabled requires more open files
open_files_limit = 32768

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
skip-networking

# Disable Federated by default
skip-federated

# Replication Master Server (default)
# binary logging is required for replication
#log-bin=mysql-bin

# binary logging format - mixed recommended
#binlog_format=mixed
# number of days to keep binary logs
#expire_logs_days=7

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1

# Try number of CPU's*2 for thread_concurrency
innodb_thread_concurrency = 32
innodb_table_locks = 0
innodb_file_per_table = 1
innodb_data_home_dir = /usr/local/var/
innodb_buffer_pool_size = 4G
innodb_data_file_path=ibdata1:10M:autoextend
# Set .._log_file_size to 25 % of buffer pool size
innodb_additional_mem_pool_size = 32M
innodb_log_buffer_size = 16M
innodb_lock_wait_timeout = 120
innodb_open_files = 32768
innodb_log_file_size = 512M
innodb_doublewrite = 0
innodb_file_io_threads = 16
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_checksums = 0

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Thanks for the help.

Arun Shankar

Options: ReplyQuote


Subject
Views
Written By
Posted
MySQL performance help
2729
January 07, 2012 06:55AM
1007
January 08, 2012 12:04PM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.