AI Site Chat is a small plugin I built for this site which reads my published posts and pages, sends them to the Anthropic API as context, and answers visitor questions in a floating widget in the corner. There are no embeddings or vector database, and it uses a super cheap LLM model applied to the content on this site. Simple and cute. If you want the longer version of how it works, I wrote about what happens when you ask my site a question back when I first built it.
In Simon Willison’s latest newsletter, I saw he mentioned that Claude Haiku 5.5 was released, and that it’s better and cheaper than Haiku 4.5 that my AI Site Chat plugin was using. Upgrading the model used in my plugin turned out to not be a simple string change in one file, since that would have broken some of the answers on my own site without logging a single error. We definitely don’t want that.
Why it wasn’t a simple string change
Newer models can work through a problem before they answer, a bit like scribbling on paper before writing the final sentence. The API sends that working-out back as its own separate piece of the response, sitting alongside the actual answer. Haiku 4.5 only did this if you explicitly asked it to, and I never asked, so the answer was always the first thing in the response. The plugin grabbed whatever was in the first slot and treated it as the answer:
$answer = $body['content'][0]['text'];Haiku 5.5 thinks by default. The answer is still in there, but the model’s reasoning now comes first, and reasoning has no “text” field for my code to pick up. Some questions would have come back empty, and unpredictably, depending on whether the model decided to think about that particular one. Those visitors would have been shown my generic “Unexpected response from AI service” message.
Nothing would have told me this was happening, either. The API was returning a perfectly normal success response, so my PHP error log didn’t register anything. The plugin would have looked fine from the inside and broken to anyone actually using the site. The fix was to stop assuming the answer comes first, and instead look through the response for the piece that’s labelled as text.
Thinking being switched on also meant revisiting three settings I’d tuned for the old model. The first is max_tokens, a ceiling on how much the model is allowed to write in one reply. Mine was set to 512, because answers appear in a small chat bubble and shouldn’t go on for too long. That ceiling now has to cover the thinking as well as the answer, so a reply could spend its whole budget reasoning and leave nothing for the visitor. So I raised it to 2048.
Haiku 5.5 also lets you choose how hard it thinks, on a scale from low up to max, and it defaults to the middle. I started on low, to keep answers quick. I also pushed the request timeout from 30 seconds to 45, since a model that stops to think takes longer to come back and I didn’t want that looking like a connection error.
When the model refuses to answer
Haiku 5.5 will sometimes decline a question outright, which Haiku 4.5 never did. Anthropic runs a set of safety checks over what gets asked, covering four broad areas: help with malware or breaking into systems, instructions for causing physical or biological harm, help building competing AI models, and a catch-all for anything else that breaks their usage policy. Those aren’t likely to be tried on my site but I did need to plan for somebody noticing there’s a free text box on my site wired up to a real language model, and trying to use it as a general purpose assistant. Anthropic also says the catch-all category can fire on innocent questions, so I can’t treat a refusal as proof the visitor was up to no good.
A refusal is awkward to spot, because the API doesn’t report it as an error. The response arrives with the same HTTP 200 success code as a normal answer, and the only sign that the model declined is a field tucked inside the response body. As far as my plugin could tell, the request had worked and just happened to contain no answer, so the visitor would have been shown my “Unexpected response from AI service.” message. That’s accurate from the plugin’s point of view and useless from the visitor’s, because it reads like a broken website rather than a deliberate decision.
The refusing itself didn’t need any updated attention in the plugin since it’s built into the new model, but handling it properly did need a change. Before my plugin tries to read an answer out of the response, it now checks whether the model declined, and if it did the visitor gets a plain sentence instead: “Sorry, I can’t answer that one. Try asking about something on the site.” Much more useful, and it doesn’t come across as a failure.
There’s one option I don’t have here. With some of Anthropic’s other models you can hand the API a list of alternative models to retry a refused question against, so a decline doesn’t have to be the end of it. Haiku 5.5 doesn’t support that, so there’s nothing to fall back to. The visitor gets told, and that’s where it stops, which is totally fine for the purposes of this site.
The low setting was breaking a rule
Almost all of my instructions to the model are rules rather than goals: use the exact post URL from the list that’s sent, never link anywhere off this site, never answer from anything other than the content I provided, and end every reply with the same closing sentence. To check that the low setting was holding up, I asked four questions against the real API on a local copy of the site: one about my job at Elementor, one about my Lighthouse Scanner plugin, one about the capital of France, and one about enterprise pricing, which this site has never mentioned. The two real questions came back accurate and linked to the right posts, the other two were declined rather than answered from thin air, and all four ended with the right closing sentence. Good enough, so it shipped.
Then I read the full text of those answers rather than just the first part of them, and found the plugin linking off-site to my GitHub repo. The instructions say never link off this site. I asked the same question eight times and six of the answers linked out. Haiku 4.5 followed the rule and never did that.
The reason turned out to be that at the low setting the model wasn’t thinking at all: every answer I measured came back with zero thinking tokens. That’s why it was quick, and it’s also why it was careless about the rule. At the high setting it thinks for a few hundred tokens before replying, and when tested over ten runs it kept the rule every time.
Raising it to high was the fix, and I was concerned about the price increase, but it turns out it barely costs anything. Thinking is billed as output, and this plugin’s spend is about 98% input, because it ships my whole content index with every question and gets back a couple of paragraphs. Input stayed at roughly 51,300 tokens a question either way, while output went from an average of 217 tokens to 498. That works out at about 2.7% more, or something like six cents a year at the traffic this gets. The real cost is time: a substantive answer went from about 2.4 seconds to about 5.2. Short questions are unaffected, because even at high the model still skips thinking on those.
It got cheaper, but not as much as the price list suggests
Models charge by the token, roughly a word or part of a word, and the way this works is lopsided: it sends my entire content index with every single question and gets back two short paragraphs. Nearly all of the cost is on the way in. Haiku 5.5 charges $0.10 per million input tokens where 4.5 charged $1.00, so the new model is theoretically 10 times cheaper. But 5.5 counts text differently, and the same content comes out at roughly 30% more tokens. My index is about 134,000 characters of posts and pages, so the real saving is closer to eight times. At the traffic this widget gets, which is not much, it now costs me somewhere around $2 a year.
While I was working on this upgrade I looked at prompt caching again, which sounds like it would be a great way to save even more. Caching lets you pay to store a chunk of context once and then pay far less each time you reuse it, and a cached read here costs $0.01 per million tokens against $0.10 for ordinary input. Every question I send carries the same content index, so it should be a perfect fit. I’d looked at it back in July and decided against it then, so I wanted to see whether the new pricing changed that.
It doesn’t. Storing something in the cache costs more than just sending it normally, 1.25 times the usual rate, and that extra only pays for itself if another question turns up before the cache expires a few minutes later. My widget gets a handful of questions on scattered days, and rarely do two come in within five minutes of each other. Most requests would pay extra to store content that nothing ever reads back, which works out around 25% more expensive than doing nothing. Those multipliers are identical on both models, so I stuck with the decision to not implement caching.
That’s it! The latest version of the plugin now talks to Haiku 5.5, which hopefully will give visitors a better response, and reduces the cost on my end. It’s win-win!
