Re: Exporting 5 million records

[Geert Vanderkelen]

First, you might look at non-Python solution: SELECT .. INTO OUTFILE explained in MySQL manual here:
http://dev.mysql.com/doc/refman/5.1/en/select.html

There are restrictions for it, but check it out, it might be much faster.

If you want to go the Python way, you might want to look into MySQLdb's special cursor SSCursor. During connection set this:
.connect( ...
cursorclass=MySQLdb.cursors.SSCursor
)

That should not buffer when you call fetchone()

MySQL Connector/Python, https://launchpad.net/myconnpy , doesn't buffer by default, so maybe you can give Sun's connector a try too.

Hope this helps!

Cheers,

Geert

Geert Vanderkelen
Software Developer at Oracle