Update to: Bizarre replication delays

[Daniel Mahoney]

This afternoon the slave process on db1 indicates that it is lagging behind gw8 ("show slave status" shows Seconds_Behind_Master: 679). It also shows:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Doing "show processlist" on db1 produces:

| 516 | replication | gw8:47205 | NULL | Binlog Dump | 109547 | Has sent all binlog to slave; waiting for binlog to be updated | NULL |
| 2122473 | system user | | NULL | Connect | 232 | Waiting for master to send event | NULL |
which suggests to me that the replication process seems to actually be running.

So I guess my question then becomes, what could cause a replication slave to lag behind the master? At this point the slave process on db1 is falling an additional second behind every second, so it looks like the slave is not actually processing the replication stream.

Suggestions?