We built the agent for ourselves first

Written by Michael Carlson | Aug 18, 2026, 3:53:43 PM

Everyone is an AI company now. It takes an API key. Here is a harder test.

There is a claim being made across marketing services right now that costs almost nothing to make. A firm signs up for a commercial model API, wraps a prompt around it, and describes itself as AI-driven. I do not think most of these firms are being dishonest. I think the bar for the claim has collapsed, and buyers have no reliable way to tell the difference between a company that built something and a company that subscribed to something.

So let me offer a test that is harder to fake: did they run it on their own work first, and what happened when it was wrong?

The problem we had

A client came to us with market intelligence scattered across inboxes. Conference attendee lists, partner introductions, event registrations, all of it arriving as email attachments in a dozen formats, all of it needing to end up as clean, deduplicated records in a CRM that already held roughly fourteen thousand contacts. The existing process was manual, slow, and error-prone in the specific way manual data entry is always error-prone: not dramatically, but steadily, in ways nobody notices until a campaign goes to the wrong people.

We built a system to do it. Document AI to parse the attachments, a matching and deduplication layer to decide whether an incoming person already existed, and the CRM as the single source of truth on the far side.

On a two-hundred-and-fifty-record conference list, it produced a clean, enriched segment with zero errors. One hundred and forty-six net-new contacts, twenty-one updates to existing records, one hundred and forty-one company associations.

The interesting part is not the AI

I want to be precise about what was hard, because it was not the parsing. Document AI is good now and getting better, and anyone can call one.

What was hard was deciding what the system was allowed to do when it was uncertain.

Consider the actual decision. An incoming record resembles an existing one. Same last name, similar company, different email. The system can merge them, create a duplicate, or stop and ask. Merging wrongly destroys human-entered intelligence that a salesperson spent a year accumulating and nobody can reconstruct. Creating a duplicate quietly corrupts every count downstream. Stopping and asking costs a human thirty seconds.

We chose to stop and ask, at a threshold we set deliberately and can defend. We also put a hard checkpoint in front of anything irreversible or anything that spent enrichment credits, and we built the system so that human-entered fields could not be overwritten by machine-inferred ones. Not because a rule required it. Because we had watched what happens when automation is allowed to be confidently wrong at scale, and the cleanup costs more than the automation ever saved.

That is the part you cannot get from an API key. The model is a commodity. The judgment about where the model is permitted to act is the product, and it comes from having been responsible for a CRM that other people’s commissions depend on.

Why “we ran it on ourselves first” is the real credential

We build this class of system for our own operations before we offer it to anyone. That is not a values statement, it is a risk position. When the system is wrong on our data, we absorb the cost and learn something. When it is wrong on a client’s data, they absorb the cost and we have learned something at their expense. Those are not the same arrangement and no amount of confident language makes them the same.

It also produces a specific kind of knowledge that does not survive being read about. You learn which failure modes are loud and which are silent, and it is the silent ones that hurt. A parser that crashes tells you it failed. A parser that misreads a column and writes plausible-looking wrong values into four hundred records tells you nothing at all, and you find out in six weeks when a campaign lands badly.

The question to ask any vendor

CMOs are now putting an average of 15.3 percent of marketing budget toward AI, while only about thirty percent describe their organizations as ready to scale it. That gap between spending and readiness is where money goes to die, and it is where a confident vendor is most useful to a nervous buyer and most dangerous to one.

So when someone tells you they have built AI for marketing, do not ask what model they use. Ask three questions.

What does your system do when it is not sure? If the answer is a shrug, the system has no uncertainty handling, which means it is guessing and reporting the guess as a result.

What can it change without a human? If the answer is “anything in the CRM,” you are being offered a liability.

Where did you run it before you ran it here? If the answer is “you’ll be the first,” that can be fine, but it should be priced and scoped as what it is.

Everyone is an AI company now.

Far fewer are willing to tell you what their system does when it doesn’t know the answer — and that, not the model, is what you are actually buying.

Michael Carlson is the founder of Pertinacity, a growth agency that builds the AI and the systems underneath the marketing it runs.

Sources