Re: ndb watchdog overslept

[Mikael Ronström]

First of all from the looks of it I would say that you have
hit a bug. So please report the bug and upload any interesting
trace files to it.

I have no idea why tables are lost. But I presume something
must have occurred between 17.56 and 18.26 that can be
seen in the logs.

The bug is that we run out of Page entries for a disk page.
I noted from your config that you use a 64MByte DiskPageBufferMemory.
This is most likely much too small. So would recommend that you
increase this to at least 1 GByte or a few GBytes given the data
sizes you are working with.

The page entries are not supposed to run out. So this is most likely
where the bug exists that you somehow can try to seize more of those
than is possible.

ALL DUMP 11100 and
ALL DUMP 11101
can print some interesting information about page entry usage
while the loading of the dump is going on.

Default config is that you can have 2048 32k pages in disk data
buffer. You can have 20480 page entries.

So somehow your dump load managed to queue up 20480 page requests.

Obviously this should rather be handled as a transaction failure.
So this is the bug.

But I am pretty sure that the workaround in your case is to
increase DiskPageBufferMemory to something like 4000M.