give a model a paragraph and it does fine. give it a column called ref_4 holding 11 character strings and it has nothing. not less signal, nothing. the values are opaque by design and the column name was an abbreviation someone stopped explaining in 2019.
the standard answer is that there is less tabular data in the training mix. i don't think that's most of it. text carries its meaning in the tokens. a table doesn't. the meaning of a column sits in three places outside the column: what it joins to, what the neighbouring columns hold, and what someone who used the app remembers about which screen it was on.
so the model isn't undertrained here, it's undersupplied. parameters don't recover information that was never in the input.
the actual question, which i can't settle:
is this a representation problem, where you feed in the join graph, the cardinalities, the value distributions, and a good enough architecture gets there
or is a chunk of it irreducible, where the column means whatever the person who left in 2021 decided it meant and no amount of compute brings that back
i lean irreducible for some fraction of it and i would like to be argued out of that. relevant bias: we build in this space, and we hit the same wall. our models do well on free text fields and noticeably worse on short code columns, for exactly the reason above, and that gap has not closed as models got bigger. that's a data point against my own optimism, not for it.
adjacent, if anyone has a real answer: text-to-sql agents demo beautifully on a 12 table schema and fall apart around 300. everyone repeats this and nobody has written down where the break actually happens or what causes it. retrieval, context, or real schemas being ambiguous in ways demo schemas aren't.
[link] [comments]