There are many places where the problem could have occurred.
> So I check my tables. They are set to utf8 - default collation.
But what about the connection from the client? See "SET NAMES".
More:
http://mysql.rjweb.org/doc.php/charcoll
> If I remove the COLLATE statement, and instead, use BINARY just before the address comparison, then it works - the query runs fine. But it isn't fast enough.
When you do WHERE x = ... and x is a particular charset and collation, and x is indexed, but you are forcing another charset _or_ collation, then the index will not be used.
> :%s/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci/
What charset setting do you end up with at the beginning of the .sql file? It must still say utf8 since that is what the bytes will be saying (because of --default-character-set=utf8)