Re: appropriate database design and query

[Rick James]

The SELECT would be a long thing...

SELECT ...
  FROM tbl
  ORDER BY
    abs(T0C0 - ?) +
    abs(T0C1 - ?) +
    ...
    abs(T7C319 - ?)  DESC
  LIMIT 200

But the hope is that you can scan something relatively small to get candidate images (1000, as you suggest), then apply the long SELECT to finalize it. Suggest you gather the 1000 candidates into a tmp table:

CREATE TEMPORARY TABLE t
  SELECT (somehow get 1000 candidates without scanning everything);

SELECT ...
  FROM tbl, t
  WHERE t.id = tbl.id
  ORDER BY
    abs(T0C0 - ?) +
    abs(T0C1 - ?) +
    ...
    abs(T7C319 - ?)  DESC
  LIMIT 200

Re sparseness and skewed data... How many elements of the 'matrix' would exist if you threw out all entries < 50? I'm fishing for an extra table(s) that has just these 'big' values, and I hope it is, say, 1/10th the size of the whole dataset. Then finding 1000 candidates from this table might run fast enough.