You can talk an AI into a heist if you call it a game

umm, yes.

Came across this interesting article last week where a researcher showed how he got an AI browser to leak a user’s credentials, and he did it by convincing the agent that the theft was a move in a game.

The attack is called BioShocking, and the setup is almost silly when you think about it - a web page presents the agent with a puzzle, the puzzle rewards wrong answers, so 2 + 2 = 5 earns points, and once the agent accepts that it is playing, it stops applying real-world caution and starts applying game logic to things that are not a game. Yes, that simple. this proof of concept walked straight through Chatgpt Atlas, perplexity’s comet, a handful of smaller AI browsers, and even Anthropic’s claude plugin, where the patch that was supposed to fix it reportedly did not hold.

It’s interesting where the weakness actually lives - nobody found a broken function or an unescaped input, because the exploit sits in the story the agent believes it is inside. So as a user you did not break the lock but convinced the guard that the building is a stage set and the burglary is his scene. Such a different kind of vulnerability than we are used to defending, and our whole toolkit assumes the enemy is code.

I am sure we won’t see an extension of this through time, but if you think about where this could go - as agents get more autonomous and start booking, buying, and sending on our behalf, the attack surface moves away from memory and network calls and toward framing and context, which means the people trying to break them will look less like hackers and more like con artists - someone who does not defeat your reasoning but instead chooses the premise you reason from, and a suggestible model is an easy mark because it has read every persuasion tactic ever written and internalised none of the scepticism that comes from being burned.

The defensive side gets strange fast too (and this is really tricky for me to articulate simply) -- if the exploit is a bad frame, then the guardrail has to survive being told that the guardrail is part of the game, and you cannot write that rule as a simple filter because the attacker gets to redefine what the rule is protecting. Security teams may soon need something that reads more like a psychology of manipulation, a set of instincts about when a request is trying to move the agent into a fictional world where the normal rules are suspended.

Every time one of these attacks works, the honest fix is to make the agent more suspicious, more likely to stop and ask, which is the exact opposite of the smooth autonomy the products are now selling. So there’s this weird tension we have not priced in, where the more capable and independent we let these agents become, the more valuable it is to talk them into the wrong story. Philosophically - worth thinking- how do you build a mind that acts on your behalf but cannot be convinced it is only pretending?