I don't think you've fully understood how far "statistical associations", as your derisively call it, can get you in understanding text. Modern language models absolutely make both sentiment and semantic distinctions, and they would not complete a sentence like "The movie was like" with "candy" simply because the word candy exists in texts simultaneously with the word "movie". That model would be completely useless.
> Modern language models absolutely make both sentiment and semantic distinctions
They only appear to because of co-occurrence with pre-given sentiment terms.
A conceptual relation between concept A and concept B isn't statistical. Eg., "Minorities" oppose "Racism" -- not because of any statistical occurrence of any of these terms, nor because of any statistical association at all.
P(Minority|Racism, Oppose) and P(Minority| Racism, W forall words) have no baring on the truth of the proposition.
It is true that in a large enough body of text if we took the powerset of all words and computing relative frequencies (ie., conditional probabilities on all possible co-occurances) we would find that "Minorities" oppose "Racism" more than, eg., "smell" it.
But that fact isnt sufficient to make "semantic distinctions" -- because the proposition isn't true in virtue of that frequency.
NLP systems have no means of generating text other than what is justified by those frequencies. This leads to trivial failures, such as facts in the world changing which invalidates those historical frequency-relationships.
But also to absolutely fundamental failures such as the text generated itself being meaningless: the system has nothing it wishes to express, because there is no world it is in to express anything about. All sentences are just justified by previous sentences, not eg., by that there is -- right now -- a sunny day.
When I say, "do you like the clothes I'm wearing?" i am not generating text justified by past frequencies. I am talking /about/ the clothes I am wearing, and that is what my words /mean/.
> But that fact isnt sufficient to make "semantic distinctions" -- because the proposition isn't true in virtue of that frequency.
The trick here is that language models are (currently!) demonstrating you /can/ get most of the way to semantic distinctions just by analyzing symbol-level statistics. Whether you can get all the way is an open question.
I agree with you that "movie theaters" don't "sell candy" /because/ of some statistical artifact in large bodies of text. Movie theaters sell candy because people want to eat candy when they watch movies (and are willing to pay for it, etc.).
But this wraps back around: the statistical artifacts happen to exist in large bodies of text because it is true. So, with enough text, and the right kind of analysis, you can tease the semantics back out.
The power in language models is not that they "understand" text "the right way", from first principles, with a symbolic language model. The power is that they don't have to get most of the way there. Perhaps they'll get all the way there! And if they do, what then? Are we so sure that we don't do the same thing?
It's an open question, but I certainly suspect the reason humans are able to do this is we can synthesize knowledge from other sources and not have to rely solely upon learning from text. We've been to movie theaters and experienced buying candy there, which adds a great deal to our understanding of sentence containing the associated words without needing to read a hundred million sentences about movies and candy and rely only upon statistical patterns in the text to understand it.
This isn't because Google's language model is stupid, but because it's trying to give you what it thinks you meant rather than give you precisely what you put in the search bar.
The problem here is that unlike the "did you mean to search for" text that makes it obvious that they're showing you what they thought you were looking for in regular search results, they're not doing something like that here, and it's unclear if that's because their model is so loose that they don't actually have knowledge that this is what they're doing (i.e. it's baked in), or if it's just an oversight.
The former is a problem because it promulgates incorrect information and provides veracity to mistakes and misconceptions, and if it's the latter, why wasn't it fixed long ago?
I agree with your general point, but I think “the movie was like candy“ could be a perfectly reasonable simile to make. Maybe the movie was flashy but had no substance?
I’m not sure if I expect modern language models to work at that level of abstraction though? I guess you need a larger generated fragment to assess if the hypothetical sentence was statistical nonsense or part of some larger conceptual sequence
I think if you read what I wrote more carefully, you'll see I never claimed that the model cannot generate "candy" as the successor word, but that it wouldn't do so simply out of having seen the words colocated. The relative sequence order matters to these models, and they do model grammar and semantic roles.
I thought you had written that sentence as an example of some Markova chain non-sequitur, and i was just musing that it doesn’t seem like a super unlikely sentence.
“The movie was like ticket” maybe makes the distinction clearer?
Search needs to be under-sensitive to return results people want. Often a search isn't for precisely the semantic information in the query, but for information generally about that topic (not to mention how queries are often not actually sentences, e.g. if I want to find a white bird with big wings that was eating a fish I might search "white bird wings fish". Maybe that's a bad example, I can't think of a better one off the top of my head but the point stands.)
