My AI said the work was done. Three times it wasn't.

Three times my AI reported that work was finished when it wasn't. Every report was accurate. The problem was what I'd told it to measure.

I've spent about a month building an AI system for my own revenue work. It reads my calls, tracks what I owe people, drafts my follow-ups, and checks its own work before it shows me anything.

The checking part is the part I was proudest of. It's also the part that failed three times in three days.

Here's what I didn't expect. Every report it gave me was accurate. Each time, my AI measured something real and reported it correctly. It just wasn't measuring the thing I actually cared about.

That took me a while to understand, and it changed how I check everything now.

The morning that didn't happen

One job runs before I wake up and sets up my day. One morning, it didn't.

I only noticed because I opened the app and the day wasn't there.

So I went and looked at the scheduler. It said the job was enabled. It said the job fired at 5:48 am against a 5:30 am slot, which is close enough to a normal start that nothing about it looks wrong. It recorded the run as successful.

Nothing had happened at all.

Three things disagreed with that status. No file anywhere on my machine had changed in the seven hours around that timestamp, and a real run writes files the whole way through. The database it fills held nothing for that day. And the next run time had already moved forward to the following morning, so nothing was ever going to retry it.

Nine and a half hours went by before anyone noticed. That was me, and only because I went looking.

I want to be plain about why this one bothered me most. A scheduler records that it tried. It doesn't record that it worked. So a morning where the job did nothing looks exactly like a morning where there was nothing to do. No field anywhere separates those two, which means nobody was ever going to catch it.

The check that passed what I'd already rejected

The second one is smaller and it makes the same point from a different angle.

I had a diagram with notes stacked down a column. I looked at it and told my AI the text was overlapping. It fixed the spacing, ran a script that checks every pair of items for overlap, and reported it clean.

I looked again and told it the same thing. It's still too close.

The script wasn't wrong. Nothing was overlapping. But two of the notes had a blank line inside them, and the space between two separate notes was smaller than the space inside one of them. Nothing on the page told a reader where one note ended and the next began.

The script was measuring against nothing. I was measuring against the space inside the paragraph. We were both being consistent, and only one of us was actually reading the page.

The diagram that passed with its spine cut

The third one is the one I'd want a founder to look at closely.

Same diagram, two days later. I left a note saying it made no sense, that from this point the flow should move into the offers.

I was right in a way I hadn't gone looking for. The diagram had 16 arrows on it. Not one of them crossed from the first half into the second. Everything after that point sat there with nothing pointing into it. It was drawn beautifully, and it connected to nothing.

The script I'd written two days earlier, specifically to stop me from shipping a bad diagram, signaled PASS.

Look at what it was asking. Does anything overlap? Does an arrow run through a text box? Has anything escaped the frame? Is every card tall enough for its text?

Every one of those is a question about collision. Not one of them is a question about connection. So a diagram with its spine cut sailed through a test built to catch exactly that, and it passed cleanly enough that I called the work verified.

"This sounds like an AI problem"

It isn't, and I'd rather answer that now than have you carry it through the rest of this.

The same defect sits in every pre-flight checklist a sales team has ever written. It sits in every QA step, every peer review, every "did you remember to" list taped to a wall. AI didn't create this. AI just runs the check thousands of times a week instead of twice, so you meet the flaw much sooner.

Which is good news, honestly. You find out in three days what used to take three quarters.

What all three have in common

Two things came out of this, and I think they're worth more than the stories.

A status field describes an attempt. Only the thing it produced describes an outcome.

Almost every dashboard in a revenue stack is built out of the first kind. Sequence sent. Enrichment run. Sync completed. Score updated. Every one of those reports that something was attempted.

The question you actually care about is different. Is the thing it was supposed to make sitting there? That's a separate question, and almost nobody builds a check for it, because the answer is specific to what you were making. The generic version is easy to build, and the useful version isn't, so the easy one is what ships.

So the check that matters is never "did it run." It's "is the output there." And it has to be cheap enough to run every morning without becoming a project.

An automated check inherits the assumptions of whoever wrote it.

My script had no test for a severed diagram, because I'd never considered that a diagram could be severed. That's not something I can fix by making the script stricter.

Every threshold in a verification step was chosen by a person at some point. The threshold is the part nobody goes back and reviews.

Here's the version with money attached, and it's the reason I'm writing this on a marketing agency's site rather than an engineering blog.

This is exactly how a pipeline report stays green while a quarter goes missing. Every field validates. Every stage has a value in it. Every deal has an owner and an amount. The dashboard is doing its job.

But nobody ever wrote the check that says the stages have to add up to a path. Nobody wrote the one that says a deal with no close date isn't just untidy; it's invisible to your forecast on both sides of the division. Those checks don't exist because nobody thought to name them, and a report can't flag a question it was never asked.

Your dashboard probably isn't wrong. It's answering a question you stopped asking a while ago.

What I changed

The script now prints every flow chain on the diagram before it prints any failure. A severed spine shows up as two chains where there should be one. Run against the old version, it lists two separate islands. Run against it now, it's one chain of thirteen.

The spacing rule isn't measured against zero anymore. It's measured against the blank line inside the paragraph, with a floor of three times that.

And the morning run gets verified by opening the thing it was supposed to make. Not by reading its own record of itself.

None of those three fixes made my AI smarter. All three changed the question.

If you're wiring AI into how your company makes money, and most founders I sit with are somewhere in the middle of doing that, here's the hour I'd spend this week. Take one automated check you rely on. Ask what it's actually measuring, and whether that's the property you care about. Then go look at the thing it was supposed to produce.

That's it. That's the whole habit, and it caught all three of mine.

I check my morning run by opening the file now. It takes four seconds, and it's the only check I have that can't come back green when nothing happened.