An upload used to be read twice: once at admission with Python’s csv module,
and once more with pandas for de-identification. When pandas refused a file the
first reader had accepted, the de-identifier fell back to column names alone,
and a column of email addresses under an innocuous header could land in storage
as it arrived.
There is one parser now. It reads the file once, every cell kept as text and
empty cells preserved. The column analysis and the de-identification both work
from that same frame, and the identifiers are stripped before the import row is
written. A file the parser refuses is a 400 that names the row and carries no
cell value, and nothing is stored, on the direct upload and on upload by
reference alike.
Two smaller changes ride along. The per-column metadata used to keep up to 50
sample values per column; for any column the detector flags as an identifier,
the samples are now empty and the column carries its category instead. And
duplicate header names are suffixed the way pandas does (Email, Email.1)
rather than kept verbatim. Metadata already stored for earlier imports was
rewritten the same way the night this shipped.