The Claude Code Setup That Stops an AI Agent Being Confidently Wrong

Published 2026-09-18 by Dale Taylor

The problem this setup is built to solve

I run Aigentic, a one-person AI automation consultancy, through Claude Code. Outreach, client projects, proposals, this article. I don't write code by hand any more.

Earlier this week I asked the agent to check the plan for a YouTube channel I'm building. It found five things wrong with it, including a line saying Aigentic builds automation for construction and HVAC firms. It doesn't. An earlier session invented that months ago, wrote it into a file, and every session since had treated it as fact.

That's the actual risk of running a business on an AI agent. It isn't that the agent can't do the work. It's that it states a wrong answer with the same tone as a right one, and there's nothing in a plain conversation that tells you which you're getting. Everything below exists because of that one property, not because the setup needed to look impressive.

The brief: one file, kept short on purpose

Claude Code reads a file called CLAUDE.md at the start of every session. It's the induction pack: who the business is, what it sells and doesn't, which database belongs to which client, how the business writes to clients. Mine runs to about 490 lines.

The mistake I made early on was letting it grow. Every time something went wrong, I added a paragraph explaining what happened, right there in the file the agent reads every single time. It got long, and a long brief gets skimmed, by people and by models.

So now the rule lives in the brief in one or two lines, and the story of why it exists moves to a separate incident log. That log has seventeen entries. The agent only opens it when it's considering changing a rule, not on every session. The brief stays short enough to actually be read, and nobody, including me, can delete a rule without first seeing why it's there.

Skills: only build one after doing the job by hand

A skill is a saved set of instructions for one recurring job. Type a slash and a name, and it runs without you re-explaining the task. I've got one that writes a LinkedIn post and designs the header image to match the brand. One that checks saved YouTube videos, scores them for relevance, and writes summary notes into a vault. One that drafts client proposals.

The one I use most is called roast. Give it an idea and it sends five separate agents at it, a contrarian, a buyer, a researcher and two others, then a judge returns a verdict. I ran it this week on a side hustle I'd seen on YouTube, buying badly listed items on Facebook Marketplace and reselling them properly. It scored 3.8 out of 10, average of five separate scores. The verdict was that my edge sits on the wrong side of that particular deal. Fair enough.

The rule I hold myself to with skills: build one only after doing the job by hand a few times first. A skill built before that just automates a guess about what the job needs.

Memory: delete the old note when a new one replaces it

Claude Code can keep notes between sessions, roughly 120 of them for my setup. Who I am, what's been decided, what went wrong last time and why. It's genuinely useful, and it's also exactly where the YouTube channel plan went wrong.

A memory note is true on the day it's written. Then something changes and the note doesn't. Two disagreeing notes sitting in the same memory store is worse than no note at all, because the agent has no way to know which one is current, so it just picks one and states it with full confidence.

The rule now is simple: when a new fact replaces an old one, the old note gets deleted, not left alongside it. And anything carrying a date gets checked against the live system before it's repeated, rather than trusted because it's in memory.

Checks: the part that actually built trust

This is the piece that matters most, and it's the one most agent setups skip. The agent will tell you something is done, missing, live or fixed, and it sounds exactly the same whether that's true.

Two examples from my own setup. Once it told me a header image didn't exist. It did, the agent had only looked in one folder. Separately, I ran a full review of my own setup and asked for improvement suggestions. I checked every specific claim in the report against the actual system. About one in four held up. It invented a setting that didn't exist and suggested installing a tool that was already installed, twice.

Two changes followed from that. First, any time the agent claims something doesn't exist, it has to show where it looked: a line starting "Searched:" naming every folder or system it checked. If the search was wrong, that's visible immediately instead of costing another hour of manual checking. Second, I built a script holding more than twenty of the claims the agent makes most often, things like "the digest ran" or "the email sent with its attachments". Each one has a check that tests it against the live system, and the agent has to run that check before it's allowed to report success.

That single change did more for how much I trust the output than any model upgrade has.

Hooks: enforce, don't ask

An instruction in a brief is still only a suggestion. Models skip steps, especially deep into a long session. The rules that actually hold are hooks, small scripts that run automatically when the agent tries an action and can block it outright.

Mine stop three things. It can't send a client an email with attachments unless a check has passed in the last thirty minutes. It can't commit code straight to the main branch. And it can't leave em dashes in anything a client will read, because I don't use them and the models reach for them constantly.

The principle is straightforward: if something actually matters, don't rely on the agent remembering to do it. Enforce it in a place the agent can't talk its way past.

Fifteen jobs that run without me

The last piece is scheduling. Fifteen jobs run automatically on my Mac: an overnight sweep that checks pipelines for known failure patterns and leaves a report, a morning brief, a video digest among others. One honest failure: a job missed a night because the Mac had gone to sleep. The tempting fix was to rewrite the job's logic. The actual fix was a power setting.

Is it worth it

Mostly, yes, but I wrote a line in my own roadmap a couple of weeks back that's still true: the tooling is ahead of the pipeline it serves. Building the system is more enjoyable than selling, and it's easy to hide inside it.

So the practical version of this, if you're setting up something similar: start with a short brief, two or three skills, and one check for whatever would hurt most if it quietly went wrong. Add anything past that only once something actually breaks, not in advance of it. And get a second model to review anything that matters. Two of my own safety checks shipped with the exact hole they were meant to close, and a review from a separate model caught both.

Start with the Sample Five, five leads in your niche with a named decision-maker and a verified work email.