MySQL Forums :: General :: MySQL performance on a HP DL385G7


Advanced Search

MySQL performance on a HP DL385G7
Posted by: Gary Richards ()
Date: April 01, 2011 05:30AM

Hi,

I wonder if anyone can make any suggestions on a problem that i'm having with some new hardware that we've purchased. The hardware is a HP DL385G7, 1x 2.33GHz 12 core Opteron 6176 SE CPU, 32GB RAM, 2x 10k SAS disks in RAID1 with 1GB cache and battery backup unit.

On this system i'm running Ubuntu 10.04, but have MySQL 5.1.49 back ported from Ubuntu 10.10.

The issue I have is that this system was bought as a way to consolidate a bunch of MySQL servers used by a dev team. Before we got to that stage we wanted to compare the performance of this system with the current production environment that we run (a HP DL365 G1, 2x 2.6GHz dual core 2218 CPU, 16GB RAM, 2x SAS disks RAID1 with 256M cache, but not battery unit, so no write cache enabled on the storage controller).

For the benefit of testing, we also took another system we recently bought and performed the same tests (a HP DL360G7, 1x 2.67GHz Xeon X5650 CPU, 12GB RAM, 2x 10k SAS disks, RAID1 with 512M cache and battery unit).

Some initial basic tests were done, such as bonnie++ and results suggested that i/o on the two new systems perform at least as well as each other.

After this, we installed MySQL on both of the new systems, configured them with the same my.cnf as on the old system, then loaded a database dump from one of our applications (dump is approximately 700MB in size, so nothing huge). Immediately we noticed that the DL385G7 (from now on referred to as 'the AMD box/server') took almost 50% longer to load the database than the DL360G7 (from now on referred to as 'the Intel box/server'). This seemed strange, so we thought perhaps the dump was in cache somewhere on the Intel box but not on the AMD box. We tried again, after running 'sync; echo 3 > /proc/sys/vm/drop_caches;' on both of the systems. The results were the same. The same database dump, using the same config, loading from the same partition as mysql is installed, takes approx 4mins on the Intel box, but 6 mins on the AMD box.

Unfortunately we were unable to test loading a database into the production system, but we soldiered on (but a little confused by the difference) and pointed a test app server at each on these servers in turn and ran some tests against them. We chose a quiet time to also run the same tests against the production DB server.

The tests are able to indicate how long various actions in one of our applications spend running various SQL statements and waiting for results, amongst other things, such as total request time, non sql time (ie total time - sql time) etc.

Pretty much consistently, every test performed in our application suggested that every test took 50% longer in sql on the AMD server, than it did on the Intel server. non sql time was very similar suggesting that the application itself was performing its tasks in the same amount of time once it got data from the database.

To avoid confusion, when I say 50% longer, an example would be one task spends 30seconds in 'sql time' when the DB server is the Intel box, but the same task takes 45 seconds in 'sql time' when the DB server is the AMD box. We also reran the tests (to ensure that any caching the mysql had done was not an issue of just performing each test once, so most tests were performed 10 times). Comparison of results was similar, often the second and subsequent times than a test ran, its overall time would reduce slightly, but the comparison of teh sql time between the amd and intel boxes was almost always 50% loner spent doing sql on the AMD box. Results from the old server (the one we currently have in production) are similar to that of the Intel box.

I understand that in theory a single 2.3GHz core is not going to be as 'fast' as a 2.6GHz core. However, I would never expect to see the 50% difference that i'm seeing due to a slightly slower clockspeed CPU. And that mysql might not be benefiting from the number of CPU cores available, but I don't think SMP is even to be considered as an issue at this stage, because during testing my systems are not highly loaded or anything like that.

I also understand that there's differences between the Opterons and the Xeons, but that doesn't explain why the system with the Opteron in our current production server performs equally as well as the Xeon system in our new servers, but the Opteron new system doesn't seem to perform anywhere near as well as either system.

My actual problem is not trying to tweak mysql on these systems at this stage, it's to try to work out why there's such a performance difference between the systems before I even begin to try to tweak the new systems.

Other things i've tried:
- Ensuring that my partitions start at a position that's a multiple of the RAID stripe size.
- Recompiling the Ubuntu source package for this version of MySQL with extra optimisations for this specific Opteron CPU, using -mtune=amdfam10. This actually resulted in worse performance?!
- Ensuring that the storage controllers have battery units installed and therefore are enabling the write cache (this was a problem that I came across initially, simply installing the AMD servers vs the Intel servers was taking 3-4 times longer until I realised that although the AMD storage controllers came with the cache, none of it was being used as a write cache by default because no battery unit was in place, installing the battery unit solved that problems, the systems now install in a similar amount of time).
- Testing I/O performance with bonnie++, it suggests that I/O on both systems performs equally as well as the other.
- Monitoring system statistics whilst my tests run, at no point does the AMD system get anywhere near close to maxing out any one single CPU core, suggesting that CPU speed isn't the bottleneck. There's no processes waiting due to blocking on I/O or anything like that, the systems seem to not really be doing any work at all (with the exception of the old server in our production environment, which was still doing mysql replication to a couple of slaves). All they were doing during the tests was serving mysql requests to one single application server, which was serving requests to one single client.
- I've checked standard things like my servers are configured correctly, they're all able to DNS and whatnot.
- Tried a different DL385G7, just in case the one I was using was faulty in some way.

I'm sure there's a bunch of other things i've done too, all of which escape me right now, none of which have helped me at all.

As suggested, the mysql config for all three systems is exactly the same. My goal at this stage it to work out why I get significantly worse performance from the DL385G7 with the exact same mysql config as the other servers.

If anyone has any suggestions or questions then anything is greatly appreciated. All the while that I can't even make this system perform as well (or even almost as well due to the slightly slower CPU) as the other servers, I can't even begin to think about consolidating a huge bunch of developers mysql instances onto this system. Which was the ultimate goal.

If you got this far, thanks for reading the entire post! Any help as always is greatly appreciated.


Thanks

Gary

Options: ReplyQuote


Subject Written By Posted
MySQL performance on a HP DL385G7 Gary Richards 04/01/2011 05:30AM
Re: MySQL performance on a HP DL385G7 Rick James 04/02/2011 11:47AM
Re: MySQL performance on a HP DL385G7 Gary Richards 04/04/2011 03:50AM
Re: MySQL performance on a HP DL385G7 Gary Richards 04/04/2011 04:52AM
Re: MySQL performance on a HP DL385G7 Rick James 04/04/2011 11:11PM
Re: MySQL performance on a HP DL385G7 Gary Richards 04/05/2011 03:21AM
Re: MySQL performance on a HP DL385G7 Rick James 04/05/2011 08:48PM
Re: MySQL performance on a HP DL385G7 Rick James 04/06/2011 06:44PM
Re: MySQL performance on a HP DL385G7 Gary Richards 04/08/2011 08:13AM
Re: MySQL performance on a HP DL385G7 Rick James 04/08/2011 09:11PM
Re: MySQL performance on a HP DL385G7 Gary Richards 04/11/2011 03:14AM


Sorry, you can't reply to this topic. It has been closed.

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.