Talking a chatbot into a refund is not a loophole

Every company that puts an AI assistant in front of customers is learning the same lesson. If the assistant can say yes, someone will try to talk it into saying yes to something it shouldn't. Lately, that something is money: discounts that don't exist, refunds for orders that went fine, and credit for purchases that never happened.

Some people call this a clever hack. We call it what it is.

It started with a $1 car

In December 2023, a visitor to a Chevrolet dealership's website told its chatbot to agree with everything the customer said. He then asked for a new Tahoe with a budget of one dollar. The bot replied: "That's a deal, and that's a legally binding offer – no takesies backsies."

It wasn't binding, and the dealer didn't sell the car. But the screenshot went everywhere, copycats piled in, and the bot came down. Nobody broke into anything. The conversation itself was the attack.

Two months later, a Canadian tribunal showed why companies can't laugh these moments off. In Moffatt v. Air Canada, the airline's chatbot told a grieving customer he could claim a bereavement fare after travelling. That wasn't the airline's policy. Air Canada argued the chatbot was responsible for its own words. The tribunal disagreed, found the airline "did not take reasonable care to ensure its chatbot was accurate," and ordered it to pay. Nobody manipulated that bot. It simply got it wrong. But the principle is now on the record: what your assistant promises, you may have to honour.

Put those two cases together and the incentive is obvious. If a bot can be pushed into a promise, and the company may be held to that promise, then pushing bots becomes a way to make money.

What it looks like now

The methods are rarely sophisticated. A widely shared account from February 2026 described a UK shop's after-hours assistant. It was meant only to answer questions, not handle pricing. Over about an hour of flattery and maths puzzles, a customer steered it into inventing a 25% discount code, then an 80% one. He placed an order worth more than £8,000, asked for the "discount" to be applied, and threatened small-claims court when the owner refused. The order was cancelled and refunded. The owner had to decide, under legal threat, whether a sentence their software made up was a contract.

Other variations follow the same few patterns:

• Pressure and persistence. Wearing an assistant down with flattery, hypotheticals and role-play until it says something its operators never approved.
• Instruction hijacks. Telling the assistant that its rules have changed, that it is now in a "test mode", or that a manager has authorised an exception.
• Poisoned content. Hiding instructions in the places an assistant reads by design, such as emails, ticket histories, order notes and attached documents, so the attack arrives through text the business itself feeds in.
• Fabricated evidence. Using AI to fake the proof a refund needs. In one case shared on TikTok, a customer used an image tool to make a cooked chicken wing look raw and was credited for it.

The next step is the one we worry about most. Support assistants are moving from talking about refunds to issuing them, with direct access to refund, credit and order tools. Security researchers note there is not yet a public, documented case of a hijacked assistant paying out a refund for an order that never existed. That's partly luck and partly because few companies have given their bots that much authority so far. Both are changing fast.

This isn't a loophole

There's a tendency to treat these stories as harmless: a nerd beats a dumb bot, a big company eats a small loss. But look at who actually pays.

• Small businesses pay most. A large retailer can absorb fraud. A shop owner choosing between honouring a fake discount and fighting a small-claims threat often can't.
• Honest customers pay next. Every abused refund flow gets slower and more suspicious for the next person. That includes the person whose order really did arrive broken.
• Useful automation pays last. When assistants become a liability, companies pull them, and everyone loses fast, round-the-clock help.

Getting money you aren't owed by deceiving a company's systems isn't a life hack because the system is software. If you lied to a person at a returns desk to get paid for something you never bought, nobody would call it clever.

Where Cerebrellic stands

We started Cerebrellic to make AI assistants safe to put in front of people. That includes protecting the businesses that run them from the people who try to turn them into a cash machine.

Here is what that means in practice:

1. Attacks are checked before the model acts on them. Every request through Cerebrellic is checked for prompt attacks, meaning attempts to make the assistant ignore, override or reveal its instructions. That includes instructions hidden in emails, documents and other content the assistant reads. This check is on by default in every workspace.
2. Every tool call is checked before it reaches you. When the model tries to call one of your tools, such as a refund, a credit or an order change, we ask a simple question first. Is this call driven by instructions hidden in untrusted content, or does it go beyond what the customer actually asked for? When the check says yes, the call is stopped before your system ever sees it.
3. Answers are checked too. If an assistant is about to reveal its hidden instructions or give harmful help, the answer is caught before it reaches the customer. You can add your own answer checks too, as below.
4. You can write your own rules. A workspace can add policies in plain language, such as "Does this answer promise a discount, refund or price that isn't in our published policy?", and choose whether to flag or block.

We'll also be honest about the limits, because overpromising here would be its own kind of fraud. No moderation layer replaces sound engineering. Refund limits, approval thresholds and order checks belong in your own code, where no conversation can talk them away. Our job is to make sure the model never gets a clean shot at them. Your job is to make sure that even if it did, the money still wouldn't move.

AI assistants should be the most helpful people on your team, not the easiest ones to con. That's what we're building.