The Reckoning: AI Infrastructure Challenges for CMOs
The last four years have been a wild ride for everyone. In 2022 chatGPT burst onto the scene and gave everyone superpowers at the tips of their fingers. All the virus experts and bitcoin bros suddenly transformed into AI experts and pushed LLMs for everything.
And let's be honest, at first glance, it looked amazing: The chatGPTs of this world performed campaign planning, analyzed data, and let CMOs dream of an unseen increase in productivity and drastically lower costs.
"AI offerings" boomed, and it took longer to test all the new shiny tools than it would have taken to do the work manually.
AI Hangover
After the euphoria came the hangover. The results were... questionable, and much more expensive than originally thought. Suddenly, hallucination was the word of the minute, and LLM bashing started, along with calling out the reputable and dignified em-dash, a surefire sign that a text was created by an LLM. Rule of three followed, and users were quickly fed up with AI slob on every website and in every ad.
Admittedly, some parts of generative AI are here to stay. ChatGPT is not bad. It helps to speed up ideation, can outline an article for editors, and image and video-generating AI is upheaving whole industries.
Agentic AI
History likes to repeat itself. When we just thought we had come through the first storm of AI hype, a new buzzword emerged from research and academia: Agentic AI.
Finally, an AI that is supposed to do things, not only talk about it. The peak of the hype cycle was ClawdBot, Moltbot, Molty, and OpenClaw. You install it on your computer, and it does EVERYTHING.
Especially charging your credit card, deleting your hard drive, publishing sensitive information, and tore open security issues beyond good and evil.
These are not only spotlights, but issues that persist in most agentic applications. Even Microsoft, not new to security scrutiny, can't get Co-Pilot to stop leaking sensitive information.
In the end, it does not seem to be that easy to get up and running with AI after all. Paraphrased from cryptocurrencies:
Agentic AI = Everything you do not understand about AI + everything you do not know about CyberSecurity.
The AI stack for 2026 and beyond
Let any one of you who is without technical debt be the first to deploy AI agents on legacy systems. Or whatever the Bible says.
I think it is fair to observe that AI, and especially agentic AI, uncovered every single technical debt an organization accumulated in the past. Forgotten permissions? Gotcha. Unsecured file vaults, that one hurts. Piles of documents, no one is really sure about the content, oh my..
Organizations learned the hard way that technical debt does not simply vanish if you polish it with agentic AI, and that the tech stack and infrastructure remain the breaking point.
Garbage in, garbage out is truer than ever. Generative AI amplifies bad data, and, unlike discriminative AI, it does so confidently sounding, even if the LLM does not pretend to be a lawyer.
Cover your bases
Currently, it is en vogue to bash AI initiatives. McKinsey, BCG, Fortune - Pretty much everyone honed in on the MIT study, citing with glee that Gen AI fails at the staggering rate of 95%.
Oh, how the mighty have fallen! At least it makes for a good headline when everyone and their mom is pumping billions into Gen AI.
So why are we still talking about Gen AI? Either it wants to be a lawyer, or it fails, case closed. For one thing, when you actually read the studies and not only salivate over the headlines, the basic challenge emerges pretty clearly: Data.
Of course, every organization drowns in data. Such and such Zillion bytes are created every day, brave new world. But this does not make the data actionable, let alone AI-ready.
Data - Source and Solution to all problems
Worse than tech debt is data debt. Having no data sucks. Having misformatted, missing, and messy data is worse. You have the Damocles sword of "some day someone will clean up this data" over your head. Suddenly, an AI provider asks you for data, and here it is again. Perhaps you make a half-assed attempt at cleaning up some Excel sheets, but it never ends. Suddenly, there is the request for more data, which is referenced in the first Excel, and so on and so on. Then the numbers don't add up anymore, somewhere is an error, but it cannot be traced, because, correct, the data is too damn messy.
Hand to the heart. What have you done in such situations? Most of the time, you decide the data is now "good enough". The problem with AI is that barely good enough is not good for AI; quite the opposite, it is bad.
Not surprisingly, 64% of organizations cite data quality as the main challenge, 77% of organizations rate their own data quality as average or worse. Gartner calculated that these cost organizations between $9.7 and $ 15 million annually due to operational inefficiencies and flawed decision-making.
So what is the solution?
In pre-AI-Hype times, data lakes and data warehouses were all the rage. One unified data pool that is correct, easily accessible, and fully integrated. As the current challenges show, the data lake/warehouse initiatives were not that good in the first place.
Salesforce estimates the annual costs of data silos in the millions, not accounting for the hundreds of work hours lost searching for data.
It's the integration, stupid!
MuleSoft came up with interesting numbers. On average, organizations have 897 applications, but only 29% are integrated. If you are wondering how data silos are born, this is it.
Of course, there are good reasons not to integrate an app. Without having a study at hand, security concerns will probably be the most mentioned answer. If your employees are using Shadow AI, your developer tried GitHub Co-Pilot, and marketing installed ClawdBot, this is the most pressing concern; we got you!
The truth is, it's more about hard work. You have to think about data integrations, pipelines, all the boring ETL stuff we thought was behind us, because we have shiny LLms now. Come on, we're over your computer science, type safety, and all that boring stuff.
Data Access Limits Intelligence
We are reading about grounding AI in truth because LLMs don't know what truth is. But if your data is the problem in the first hand, it gets pretty hard, pretty fast to find the truth, even in your own data.
The hard truth for many AI initiatives is that DataOps is the battlefield on which the AI wars are won or lost. You cannot win a NASCAR race with your old Toyota Supra, even if it is a nice drive to work. Data governance is no longer a compliance checkmark, but a strategic tool to leverage all the promised AI benefits.
Shameless plug: We have developed a data hub as a service and battled with all the data integrations, locked-in syndromes, and data unification challenges. That is not for the faint of heart.
If you read til here, take a moment to think about it. Why did we do this? Because we were tired of subpar AI results, the data plainly sucked. Every client has data problems, no shade, but our AI tools can only do that much lifting.
If you use our data hub, the one from Salesforce, or Funnel does not matter. What matters is that you get a grip on your data before you dive headfirst into AI projects.
Tokens, everybody gets tokens
Let's say you have your data in check and found a way to manage your clients' data if you are an agency, and suddenly, you, the CMO, have a new thing lurking. Just like Bruce in the waters of Amity Island, there is a shadow circling around, waiting to take a huge bite out of your budget: Tokens.
Suddenly, everything is a token. And while the ads say you are charged cents per million tokens, your budget is vanishing fast.
Let's say 100 employees consume 1 billion tokens per year, at $2 per million tokens. $2000 sounds manageable. But now you have your data in order, new apps get integrated, and the usage jumps to 20 billion tokens. Now you are facing a $40,000 invoice from the AI provider of your choosing. Not that easy to stomach anymore.
The numbers look worse for large enterprises, where the number of employees runs into the tens of thousands.
As a CMO, you have to calculate the AI usage, no matter if you are a small agency or a large enterprise.
The economic logic of token consumption can be modeled by the total cost of ownership (TCO) for AI operations:
Math, how fun.
Suddenly, you have to think about usage-based, outcome-based, or per-seat pricing. An AI running wild can quickly diminish your expected returns and make a good deal look bad.
And finally, to stay with the good people of Amity Island, you need a bigger boat!
Model Lifecycle and LLMOps
As regular as tax returns, every couple of months, LinkedIn and Reddit fill up with posts about an AI becoming stupid. Suddenly, the LLM does not answer as expected. "Last week it was better". You can exchange the LLM provider; everyone gets called out. Often, this happens at the tail end of the lifecycle of an LLM. All LLM provider are in the red offering their systems at a loss. After the big release of a new model and after everyone has run their tests and the model is at the top of the leaderboards, the models get distilled.
This process involves training a large model to train a smaller model, in the hope of making it cheaper to run while losing little quality in the outputs. If that does not move the cost needle enough, reasoning gets compressed. While the ads still talk about trillions of parameters, only a couple of billions are doing the hard work. This leads to a decline in quality, especially in corner cases or niche topics.
At this point, you have cleaned your data, done the math on token costs, and know the model is declining. To top it off, after the new shiny model is released, you have to redo your cost calculation because providers are pushing their new model and new pricing, quickly sunsetting old models.
CMO-CIO-CISO Alliance
We hope you have great co-workers. As a CMO moving forward, you have to talk to a lot of them and find common ground. The easy task of using AI in marketing turns out to be a cross-functional, cross-department challenge. Suddenly, you have a shared roadmap with all the tech guys. Security, data quality, algorithmic integrity, and governance are increasingly becoming your bread and butter.
Not all those who wander are lost. Deep roots are not reached by the frost.
What sparked a glimmer of hope in Frodo should be your guiding star if you are a CMO facing multiple challenges in getting AI up and running for your marketing teams.
The often-cited Digital Transformation is here to stay, and you'd better tackle it now, rather than in a year. What we have learned over the last five years when it comes to integrating AI into legacy systems of all sizes and makeups:
- Start small: Find a precisely and small described problem you want to solve with AI. You have to learn to walk before you can run.
- Make sure you have data. Ensure data is in place and in a format that can be used for AI and your problem. If you want to predict housing prices, you need numerical data, not an essay about Victorian houses in Outer Sunset.
- Partner with experts: More likely than not, you do not have the AI expertise in-house you need. No worries, most likely your competitors do not have it. Find a partner who has solved your problem before and has a proven track record. Be careful with wrapper infrastructure. If someone is selling you ChatGPT in a new UI, you buy all their tech debt.
- Measure, measure, measure. We cannot stress it enough. Measure the outcomes. Is the new AI tool really an improvement over the legacy way, or is it making things complicated in a new exiting way?
- Scale: If the first four steps worked well enough, and you measured a real return, scale. Roll the tool out to new departments or new problems. And keep on measuring.
