help understanding log files

[Eric Miller]

At least once a day, I'm getting this in my error log file:

InnoDB: Fatal error: cannot allocate 1064960 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 248356160 bytes. Operating system errno: 12
InnoDB: Cannot continue operation!
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=2093056
max_used_connections=30
max_connections=100
threads_connected=30
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 876143 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

060516 12:19:22 mysqld restarted
060516 12:19:23 InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 2 1264729428
InnoDB: Doing recovery: scanned up to log sequence number 2 1264737530
060516 12:19:23 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 3
1 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 6
7 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 1412821, file name ./ns1-bin.155
060516 12:19:24 InnoDB: Flushing modified pages from the buffer pool...
060516 12:19:24 InnoDB: Started
/usr/local/libexec/mysqld: ready for connections.
Version: '4.0.25-log' socket: '/tmp/mysql.sock' port: 3306 FreeBSD port: mysql-server-4.0.25


We recently had our hosting company perform a hardware test that showed no problems, we added an additional GB of RAM (taking us from 1 to 2), but the problem still persists. Has anyone seen this before that can tell me what's going on and what needs to be done to stabilize the system?