AI Is Not the Product. The Product Is the Product.

Setting aside for a moment "agentic" (which I address later) and other buzzwords du jour, by 2025, business conversations about AI all converged on one dominant idea: large language models as chat interfaces. Nearly every executive had by then seen a demo of someone typing a question into a box and receiving a beautifully formatted answer, and many organizations immediately tried to replicate that experience inside their own systems. In the rush, teams bolted LLMs onto legacy software or, in some cases, created entirely new "AI tools" that floated in isolation. The excitement of the early ChatGPT years had encouraged people to equate "AI" with a chat window, leading to architectures that offered novelty but not transformational user value.

This post is my rough attempt to articulate a clearer, more durable way of thinking about AI in B2B SaaS products that serve enterprises (some of these thoughts also apply to AI products more broadly). It traces how we got here, why chat-first thinking has constrained product vision, and why the future of enterprise AI will depend on hybrid, contextual, governed systems—not just bigger models or smarter interfaces.

How We Got Here: Three Generations of Enterprise Computing

Why was the business community so obsessed with plugging these into their organizations at the first chance they got?

For most of the past seventy years, enterprise computing has evolved in recognizable stages. The framing is oversimplified, but it helps explain the moment AI now occupies.

The first generation—the ERP adoption era that stretched from the mid-20th century through the early 2000s—was about encoding business logic into systems of record. These systems captured transactions, standardized processes, and gave organizations their first digital memory. But the latency between something happening and someone knowing about it was enormous. Weekly, monthly, and quarterly reporting cycles were the norm. The central challenge of that era was simply getting the data right.

The second generation, beginning around the mid-2000s, was defined by dashboards and BI tools. With the rise of Tableau, R, PowerBI, Looker, and similar technologies, leaders could finally see data in close to real time. Dashboards became the primary lens through which performance was interpreted. But dashboards brought their own constraints: users had to know where to look, how to interpret the visualizations, and what actions those insights implied. The data was fresher, but the burden of understanding still fell entirely on the human. You had to navigate, infer, decide, and then manually act. This era was fundamentally about visibility.

Then came 2023 and the emergence of ChatGPT, which kicked off the third generation: the belief that systems could not only show data, but understand it. Suddenly, pattern recognition, narrative explanations, and decision suggestions seemed possible through natural language alone. Retrieval through RAG promised to eliminate navigation entirely, finally relieving high cognitive load. And once the idea of LLMs connected to organizational data took hold, it was only a short leap to imagining "agents" that could take actions on a user's behalf.

But much of this imagination rested on assumptions that didn't survive contact with reality, and many enterprises started to realize the problem with the popular plug-and-play, LLM-into-chat construct of AI product.

The AI Hype Cycle and its Limits

By 2025, "agentic AI" had become the buzzword in every boardroom. The term "agent" became so overused that it lost definition. For some, an agent was a workflow executor. For others, it was a conversational assistant with API access. For still others, it was an autonomous operator capable of completing entire multi-step processes without human involvement. These interpretations were incompatible, yet often discussed as if they were interchangeable. Whenever people talk about agents, I always ask them what exactly they mean.

Businesses quickly learned how overestimated the capabilities were (especially where LLM inference was involved with poor guardrails)—and how underestimated the cost, complexity, and governance challenges would be.

Now to return to problems with the LLM-intensive approach in integrating AI into products (dominated by chat interfaces)

Non-deterministic nature

LLMs, impressive as they are, remain stochastic systems. They hallucinate. They cannot be relied upon for row-level data analysis or anything requiring arithmetic precision. That created immediate friction with enterprise contexts, where rules, constraints, and safety boundaries are explicit.

Unsustainable cost structure at scale

In general, many companies learned that leaning on a chat interface meant:

• Inference heavy process
• Natural language input allowed users to ask questions not related to anything at work
• Users would ask the same questions again and again
• Long-winded response

All of these drove token usage through the roof, making costs prohibitive such that these solutions were not scalable across an enterprise in a sustainable way for the business.

Vendor lock-in

The potential of strong lock-in with OpenAI (or the other major providers) presented single-point-of-failure and future economic risk

Regulatory constraints, data privacy and cybersecurity concerns

Really, we're going to give OpenAI access to all our data? What if a bad actor puts a malicious query into the chat? Is the chatbot's API access circumventing previously implemented access privileges to safeguard sensitive data?

Surely everybody thought about and perfected this tech overhead and complexity beforehand /s

How AI Chatbots Became a Mental Trap

I believe that many enterprises, and their PMs shipping AI products, have been artificially constrained by the chat construct, limiting themselves from thinking more broadly and holistically about core customer needs and the value that is really provided by chatbots. For example, someone might say "when I'm sitting in a sales update meeting, I have to go into the CRM and pull information in real time so that I can talk through it to provide updates." That person, or a PM, might envision a solution as a chatbot that allows salespeople to ask questions and receive updated information via LLM. In reality, maybe the sales update meeting could have been altogether eliminated if the sales team's manager was kept apprised in real time regarding sales efforts, through some AI-enabled solution (this could take many forms).

And, for that use case, is the idea that business users should ask questions to retrieve every piece of data they need throughout the day? Isn't that getting to the same place as custom reports and dashboards in a much more compute intensive way?

From a logistical point of view, how will the chats be stored, organized, and referenced in the future? Will it become a new navigational nightmare to recall past conversations where valuable work or thinking might have taken shape?

Effective "AI" Uses Determinism

In my view, much of what people seek to accomplish with AI must be built on a thoughtful blend of deterministic logic and AI inference in order to be scalable, reliable, and cost-effective. This means aggressively reducing unnecessary LLM inference—offloading where possible through smart caching, deterministic rule engines that encode business logic, and lighter-weight ML or SLMs running on edge-friendly hardware (obligatory shoutout @ Cisco AI-ready UCS/XEON/Hyperflex nodes). Not every AI experience should default to a chat interface; products should expose intelligence in the form that best fits the user's workflow.

In many cases that means command palettes, assisted form-fill, inline predictions, explainable alerts, queue triage systems, custom generated workflows, or agentic back-office automation—not conversation. I strongly feel that the pinnacle of AI product is reached when the intelligence is seamlessly embedded in the experience, quietly providing value without loudly drawing attention to itself. Integrating AI into products quietly should remove friction and elevates outcomes in ways that feel intuitive. Recommendation systems are the clearest example: they’ve been around for decades, yet nobody really thinks about them. Despite what some might hope for, no user thinks to themselves, “I am so impressed by this cutting edge AI model that just improved my movie night”; they just enjoy better choices with less effort. The highest form of AI product makes users feel that, “this just works, and it works for me.”

The real task is designing systems in which deterministic components handle the predictable, safety-critical, and latency-sensitive paths, while model inference is reserved for the parts of the workflow where uncertainty or creativity is actually required—ideally governed by carefully curated model context, scoped to each use case and even personalized at the user level.

All of this ultimately rests on the bedrock of great data architecture.

Architecture as Enabler of Agility in the AI Era

As I was editing this post earlier in the fall, during UN Climate Week, I asked Equilibrium Energy's founder whether he pushed engineering to invest in architecture upfront, or built use cases first and sorted out the foundations later. His answer was that he gathered everyone into a room and vigorously encouraged them to throttle back their obsession with velocity and the weekly-MVP treadmill of use cases. The message was simple: you cannot move fast if you don't first build the substrate that makes speed sustainable. The company has since shipped a number of impressive new AI-enabled products (at high velocity!) that are impressively performant and tailored.

This is where people often get confused: architecture is not incompatible with Agile. In fact, the Agile Manifesto explicitly states that "continuous attention to technical excellence and good design enhances agility." Good architecture is exactly that technical excellence. And it maps directly to the PM's responsibility: you cannot "satisfy the customer through early and continuous delivery of valuable software" unless you deeply understand how users interact with data—where latency matters, what context is required, how decisions flow through their domain, and what invariants govern their workflows. Architecture is not anti-Agile; it is the enabler of agility.

The Manifesto's insistence that "business people and developers must work together daily" captures the core of PM work on AI systems: collaborating with architects and engineers so that schemas, caching layers, routing rules, and pipelines reflect real user needs and scale across many use cases rather than being bespoke one-offs. And when Agile praises "simplicity—the art of maximizing the amount of work not done," it is directly advocating for shared infrastructure—systems that avoid duplicating work and prevent the proliferation of ad-hoc pipelines that slow teams down. In other words, building coherent data architecture is the Agile thing to do.

For AI specifically, PMs carry an additional responsibility: they must understand user needs at the deepest most abstracted level and anticipate the attributes (and attribute combinations) that will drive the majority of use cases. They must partner closely with engineers and architects to ensure those attributes are present, standardized, and governed in the data layer that their AI products are based on. This work is essential not only for enabling high-granularity use-case definition—critical for MCP configuration, feature construction, insights surfaces, and agentic behavior—but also for ensuring that a single, coherent architecture supports data stewardship and incorporates existing well-founded guardrails (e.g., RBAC) in the high-risk domain of AI. Without this backbone, "AI product work" becomes a series of brittle demos and tech debt bonfires instead of a scalable system capable of delivering real value.

Closing Note