If You’re Not At Least Spot Checking AI Results, You’re Nuts

I know we are pretty far down the AI rabbit hole at this point, but we do still have functioning brains. And we should absolutely be using them!

I am astonished at the sheer number of people who really and truly think that today’s AI (or more accurately LLMs) can reason, analyze and think. I hate to be the one to break it to you, but they cannot actually think. What they can do is ingest data and spit out very plausible and confident sounding results to prompts. And one of the biggest problems is that most people just stop there. They take what Claude or ChatGPT or Gemini or CoPilot spit out as correct and hit the ground running with it.

Have any of you actually spot checked results or analysis or strategy that an AI produced? I was talking with an industry friend today who was telling me how an AI report had mixed up the click through rate and conversion rate columns of data. WHAT???? Think about that for just a second more. What would you take from that result if you saw it and did not do any type of verification or double checking?

You might think:

OMG my conversions have been WAY undercounted – how did my agency miss this?

Is our CRM data wrong? We did not see that many conversions internally?

Or I thought our click through rate was good, why is it so low? Are our ads bad? Targeting way off?

And maybe you go to your agency or consultant or in house PPC person and ask. Or maybe you just run with what the AI told you and start making decisions based on extremely incorrect data.

This is one of the big things that is currently driving me absolutely insane about the whole AI discourse. It can’t think. It really, really can’t. It guesses on words based on what it has in its training dataset relative to the prompt you entered. It isn’t even as sophisticated as Excel or Google Sheets. At least those applications perform actual calculations or functions from the data you enter into them. LLMs do no such thing. But that does not stop them from presenting strategy or data figures as if they do!

And for those who will say it’s just about inputting the perfect 1500 word prompt string, please miss me with that nonsense. If you have to spend that much time massaging prompts, maybe you should just do the actual analysis?

At a bare minimum, someone needs to spot check critical data points in AI/LLM results. It is truly the bare minimum and it could save you a lot of headache down the line. Because the truth is, it can take months for bad AI recommended strategy changes to wreck your account. It is often not instantaneous. Neither are the remedies.

So next time you want to shortcut it by dumping your performance data into an AI/LLM for strategy, ask yourself how you will know what it tells you is correct. And what might be the consequences if you don’t bother to check.