Imagine a world where the human-in-the-loop was deemed to inefficient, and code does not get reviewed anymore. The next generation of worms just need to spread into the data centers that host these models and flip neurons. Would be a pretty cool supply chain hack.
This type of research should put the "do LLMs think" to rest, not to mention the ludicrous claim of consciousness cough Dawkins cough. It is quite clearly a pattern matching algorithm at scale.
It's still narrow AI but massively scaled to look like general AI.
I'm not sure why you think so. We know that very local brain injuries can result in personality changes, sometimes dramatic ones. And that's in a system evolved to deal with constant damage and rebuilding. With LLMs trained to achieve the opposite of that (minimal redundancy, maximum performance), the effect is similar but more extreme.
How does this follow? Isn't it fair to say that what someone's interacting with at the 'personality surface'-level is a form of emergent complexity / emergent behavior, not readily apparent or discernible at the neuron level (just like how human 'thinking' or 'consciousness' seems to work)? I'm not even arguing these models are conscious, I'm just saying that it doesn't seem like we can take the fact that a single neuron flipping their behavior or triggering 'alter egos' is enough to say they don't "think", etc. We might discover that flipping - if not a single neuron but perhaps really tiny clusters of neurons/synapses - in a human brain, might result in similar effects. We haven't had human connectomics studies to really show this (to my knowledge).
It's still narrow AI but massively scaled to look like general AI.
It therefore provides a correct answer in 96% of cases, and displays a sense of humor in the remaining cases.