Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.
What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
Like any other software or dependency. Open or close.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
No, only like any other closed source software or dependency. It’s at the very least harder to hide things like that if all the source is available.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
Surely we all run these agents in isolated environments since months back already, right? I don't know anyone who manually approve each command/change a agent does, and also I don't know anyone who doesn't review the changes the agent ultimately propose tightly, before accepting it locally.
They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.
So what is the proposed "fix" here if there is any?
Btw, if it's just the date (and nothing else) researchers or anyone interested can explore that by just providing future dates and see if anything changes.
Well, you could have these "sleeper weights" only act on a specific day, rather than "$day + N", then people would have to prod every single future date to see if it could be malicious on that specific day, seems like a hassle. But I guess would be the only way really to detect it?
Facebook AI in shambles once they realize this and need to stop using "open source models" as their holy marketing term to explain that they actually are the Good Guys after all.
Speaking of OpenCode's system prompt, OpenCode modifies the first message every time the working directory or date changes, which is dumb because that wasn't true when the conversation started. Transcripts are supposed to be historical records... Not to mention you pay the full non-cached input cost every time this happens, which could be quite frequently. Use a better agent like Pi.
What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
Well, yes, because you invoked OpenCode with the --auto flag telling it not to stop for confirmation.
- https://arxiv.org/abs/2311.14455
- https://arxiv.org/abs/2401.05566
- https://arxiv.org/abs/2410.13722
Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)
So what is the proposed "fix" here if there is any?
Then I was saying to never trust an LM you haven't trained yourself. But can you really?
If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.