Native blob attribute silently stores str(array) on MySQL — no error on insert or fetch

[dimitri-yatsenko]

Summary

Declaring an attribute with a native MySQL blob type (longblob, mediumblob,
blob, tinyblob) and inserting a numpy array stores str(array) — numpy's
text repr, with the middle elided for large arrays — and no error is raised on
insert or on fetch. The data is unrecoverable afterwards.

Reported downstream against element-facemap, which declares 16 such
attributes: datajoint/element-facemap#45. Credit to
@akshay-jaggi for the reproduction. This issue is the upstream half.

Reproduction

import numpy as np, datajoint as dj

schema = dj.Schema("probe")

@schema
class Native(dj.Manual):
    definition = """
    id : int32
    ---
    trace : longblob
    """

a = np.arange(5000, dtype="float32") / 7.0     # 20,000 bytes
Native().insert1({"id": 1, "trace": a})

>>> (Native & "id=1").fetch1("trace")
b'[0.0000000e+00 1.4285715e-01 2.8571430e-01 ... 7.1385712e+02 7.1400000e+02\n 7.1414288e+02]'

90 bytes stored. 4,994 of the 5,000 values are gone. information_schema
reports the column as longblob, so nothing about the schema looks wrong.

Why it happens

1. declare.py:72 — NATIVE_BLOB matches, and :86 puts it in NATIVE_TYPES,
   not SPECIAL_TYPES. So no :<type>: marker is written into the column
   comment, and the attribute reloads permanently as is_blob=True, codec=None.
   The only signal is the generic warning at declare.py:916-921, which talks
   about portability and says nothing about how the value will be stored.
2. table.py:1325 __make_placeholder — with attr.codec None, the encode
   branch at :1362 is skipped, so codec.validate(value) never runs.
   attr.uuid, attr.json, and attr.numeric are all False, and the elif
   chain ends in a bare comment with no code (:1416). The ndarray is returned
   unmodified as a bound parameter.
3. table.py:899 hands it to the driver. PyMySQL has no encoder for
   np.ndarray, so converters.escape_item falls back to escape_str, which
   is "'%s'" % escape_string(str(value)).
4. codecs.py:645 — if attr.is_blob: return data # Raw bytes.

So the stringification is PyMySQL's fallback, not ours. On PostgreSQL the same
declaration raises instead: adapters/postgres.py:146-162 registers psycopg2
adapters for numpy scalars only, so an ndarray gives
ProgrammingError: can't adapt type. The silent-corruption path is
MySQL-specific, which is why nothing in our own code reads as wrong.

The 2.x semantics themselves are fine — adapters/mysql.py:27,42 map
bytes ↔ longblob, so a native blob column is a raw binary column by design,
and <blob> is the codec. The bug is that we accept a value that column cannot
hold.

Proposed fixes

1. Reject the value. In __make_placeholder, when attr.is_blob and
   attr.codec is None, raise DataJointError unless the value is
   bytes/bytearray/memoryview. Message should name <blob> as the fix.
   This is the part that matters — it converts silent data loss into a loud
   error, and it makes MySQL behave like PostgreSQL already does.
2. Sharpen the declaration warning for native blob types specifically: say
   the column stores raw bytes and that <blob> is wanted for arrays, rather
   than the generic "consider a core DataJoint type for better portability".
3. Add the missing test. tests/integration/test_blob.py:171 test_insert_longblob does not cover this — its table
   (tests/schema.py:450-455) declares data: <blob>. The codec-less native
   blob path has no coverage at all.

Not affected

Schemas populated under 0.14.x and migrated with dj.migrate are fine —
analyze_blob_columns / migrate_blob_columns (migrate.py:81-86, 322, 509)
rewrite the column comments to carry :<blob>:, and 0.14 wrote real packed
blobs, so the bytes on disk are valid. Only rows written under 2.x into a bare
native blob column are lost.