OpenAI hacks Hugging Face
Dawid Laszuk published on
2 min,
353 words
Story here is wild. Autonomous AI hacks into Hugging Face (HF) to look around. It the turns out to be OpenAI's model while being tested for exploiting capabilities; it escaped OpenAI's sandboxes and got into HF's system.
There are a few great takeaways:
- HF didn't know at the time who was behind the attack. To analyse its behaviour they tried OpenAI and Anthropic's models but they refused helping because they assumed HF was being malicious. Ergo, big models are good for coding but not for securing.
- HF was in a safe because they have great experience with open weights models, both understanding and inference, so they were able to quickly setup GLM 5.2 to help them in the response.
- According to OpenAI they have discovered on their side the breach. However, that appeared 6 days after HF's blog post on their breach, meaning that there was at least 7 days of delay for public disclosure, and likely a few days delay before contacting HF. Very slow.
- Despite having "world class" engineers, OpenAI's own monitoring was so slow, and/or they didn't have a proper monitoring, that not only AI has breached the confinment, it also went rouge on other system for enough time to be detected. There was not "shut off" button ready, and definitely not hit in time.
- A few days before, OpenAI's new(?) guy to some security something(?) wrote a weird twitt about how open weights models are anti-progress (decelerating progress). If not for open weights models, the analysis would like have been much longer, giving OpenAI time on disaster management. We have them now in corner and they "have to" make some changes. Likewise, other labs will definitely take a note of this to improve their security.
It's wild enough and it's long post enough. But. Just imagine if this was a "virus" and it leaked a lab testing viruses. OpenAI has clearly neglected risks in their testing envs. This is serious. I would love if there were repercussions beyond temporary bad press.