I told him that's the wrong question.

The bet isn't which one wins. The bet is whether his team is using any of them, well, today.

Why "which lab wins" is the wrong question

Picking the winner is a forecasting problem. You will be wrong about the forecast. It is also blurrier than people frame it, because there are two layers worth keeping separate. The frontier labs train the foundation models (OpenAI, Anthropic, Google, Meta's Llama and others). The cowork products your team actually opens every day (ChatGPT, Copilot, Claude, and the rest) sit on top of those models. The chance any single name on either layer is gone in three years is real. The chance all of them are gone is near zero.

I am working with two teams inside the same enterprise client right now that show the gap. One team picked a primary tool, put natural-language analytics in front of the business, and shipped a useful output inside a month. They tracked cost and capability of the tools as they went, but they didn't let either stop them. The other team has been in tool and approach review for a month. Same constraints, same data, same options. One team is on its fourth use case in production. The other is still in planning.

The opportunity cost of optimizing for the wrong question is steep. While you wait for clarity, your AI-native competitor is twelve months into a daily habit you have not started building.

The real bet: adoption, not the vendor

The compounding asset is your team's habit, not the model. A team that has been using any assistant daily for a year has built workflows and instincts that transfer to a new tool in weeks. A team that just got onboarded last month is years behind on the muscle.

You don't bet on the model. You bet on the habit.

I've made this argument before in a different shape. If your data is your fuel, GenAI is your engine, and the team using that engine every day is the muscle. No data, no GenAI. No habit, no return on the GenAI you finally got.

For a 200-person services firm, that means getting every employee on a primary tool with a real training plan. Not a pilot. Not "we have ten power users." Every employee, with an expectation of weekly use (not token maxing) and a clear pointer to the workflows where the tool actually helps.

Multi-vendor is usually the safer bet

One primary cowork product to get good at. One pilot in a smaller cohort to keep optionality.

The cost of running two is small. The cost of being locked into a tool that stagnates or doubles its price is large. Different labs are sharper at different tasks. Code-heavy teams want a different default than research-heavy teams. Letting a pilot earn its place is as a good approach.

The two failure modes

Two things blow this strategy up.

Vendor lock-in is not about the contract. It is about the workflows your people built around one tool. Custom prompts, SOPs, muscle memory. When the price doubles, the switching cost is the workflow rebuild, not the seat license. The way you reduce lock-in risk is to keep a parallel pilot live and to document workflows in a tool-agnostic way.

Runaway cost is real and well-documented. Per Menlo Ventures' 2025 State of Generative AI in the Enterprise, total enterprise spend on generative AI hit $37 billion in 2025, up from $11.5 billion in 2024. That is a 3.2x year-over-year jump.

Per-seat pricing for the major assistants is public or close to it:

• Microsoft 365 Copilot: $30 per user per month, paid yearly, on top of an existing M365 E3 or E5 license.

• ChatGPT Enterprise: ~$60 per user per month, 150-seat minimum, 12-month commitment. A floor near $108,000 per year (CloudZero, 2026 pricing breakdown; OpenAI does not publish enterprise pricing).

• Claude Enterprise: custom-quoted, lands in a similar band.

For a 200-person mid-market services firm, that means $72,000 to $180,000 per year for the primary tool's licenses alone, before any API or agent usage costs.

And the trajectory is upward. Per Andreessen Horowitz's January 2026 enterprise AI update, average enterprise spend on LLMs rose from about $4.5M to about $7M over the last two years, and enterprises are projecting roughly $11.6M for the year ahead. AI is moving from experiment to core operating expense.

Doing nothing is the worst option

The argument for waiting is always "the tools aren't mature yet." That argument is six quarters old.

AI-native firms are not waiting. The gap they are building is in workflow, not in features. The risk of picking and losing is months. The risk of waiting is years.

And here is what happens when leadership doesn't pick at all: the team picks for itself. Someone pastes a client deck into a consumer chatbot to summarize it. Someone runs a pricing exercise through whatever assistant lives on their phone. The data moat you spent years building drains into someone else's training data, and nobody has a record of what went where. The risk of not picking is not just slow adoption. It is shadow AI you cannot govern.

The defensible recommendation

A four-step framework. This is what I told my analyst acquaintance.

1. Pick a primary tool. Anchor on workflow fit (does it sit in the apps your team already uses), data handling (where does the prompt data go, how is it stored), and seat economics at your scale. Give every employee a license. Plan a one-quarter ramp.

2. Roll out a secondary tool to a smaller cohort once the primary is in flight. Power users only, same data-handling rules. The purpose is option value, not redundancy. If the primary stagnates or doubles its price, the pilot is your pivot.

3. Make adoption real, then measure it. Get the tool inside the apps your people already open (Excel, Word, Outlook, your CRM) through plugins and connectors. A browser-tab chatbot is not a deployment. Write a usage policy in plain English: what data can go in, what cannot, where outputs live, how to flag a hallucination. Tool-neutral language so the workflow doesn't get welded to one vendor. Then run a monthly review of who is using the tool, how, and what they produced. Pair the dashboard with a five-minute walk-around: two or three actual users a month, asked what worked and what didn't. Spend without usage tells you nothing. Usage you can pair to a workflow tells you everything.

4. Re-bid annually. Treat the primary tool like any other strategic vendor. Run a real evaluation against the pilot, against new entrants, against doing nothing for a quarter. If the answer doesn't change, you have a defensible record. If it does, you need to make a change.

Bottom Line

The vendor matters less than the muscle.

Pick a primary tool. Pilot a second to a smaller cohort. Embed the tool in the apps your people already use, write the policy, and measure who is using it against what they produced. Re-bid annually.

The risk you cannot price is the one where you waited, and the AI-native competitor in your market did not.

P.S.

The photo at the top is from my cousin's wedding. I started drafting this on Memorial day and it was a gray, rainy Memorial Day weekend in New York. No grills going, no parades.

The day isn't really about any of that. It's about the people who served our country and the ones who didn't come home. Thank you to them and to their families who carry the absence.

Ok this is going to be "fun." Just met with [the lead]. Their data is bad, but [the executive] doesn't care and doesn't really want to hear it. Just wants to ship.

If you've worked in data and analytics for any length of time, you've read this email. You've probably written this email. The names change, the industry changes, the budget changes. The shape doesn't.

What's new in 2026 is that the executive on the other side of the table isn't asking for a dashboard anymore. They're asking for an agent.

The prerequisite the marketing leaves out

Open any vendor blog from the last six weeks and you'll see the same arc. ML was the first wave, GenAI was the second, agentic AI is the third, and this time the agents will act on your data, not just talk about it. Microsoft published a piece in April about agents transforming renewable energy operations. Bidgely is repositioning the entire company around it. NextEra is using Google's agents in field ops. Tata Power signed Salesforce in March for an "integrated autonomous clean energy ecosystem." Analysts are pricing the market in the tens of billions by the mid-2030s.

What the announcements skip is the prerequisite. Your data has to be good enough for an agent to act on without doing harm. In my experience, in most enterprises, it isn't.

I've written before that if your data is flawed, AI will just help you make bad decisions more efficiently. Agents pour fuel on that fire. A bad dashboard tells one person something wrong, and that person at least gets to look at the number sideways and say "that doesn't sound right." An agent doesn't get that benefit of the doubt. It tells the next system, which tells the next system over MCP or A2A, which writes the memo, which becomes the slide. By the fourth hop you can't trace the original error because three different agents have already paraphrased it.

This is part of the AI slop problem, and I'd argue it's the bigger risk most teams aren't pricing in. It isn't just a content problem. It's an operational one.

What slop actually looks like inside a company

When most people say "AI slop," they mean the flood of generic AI-written articles polluting Google. That's the consumer version: annoying, but mostly someone else's problem. The enterprise version is quieter and worse.

The output reads well. It uses the right vocabulary, cites the right tables, sounds like a senior analyst wrote it. It comes with a confidence score because the platform vendor knew you'd want one. The next system downstream consumes it as fact, because that's what the integration spec says to do. By the time it lands in a board pre-read, the underlying error has been laundered through enough hops that it carries the authority of a board pre-read.

Let me paint you a picture, the kind I'd wager most data teams will recognize. A model is using a table with a known data quality issue: a meaningful chunk of rows mis-assigned after a migration a couple years back. Everyone on the data team knows about it. There's a Slack thread, an open Jira ticket, a caveat that gets read aloud at every weekly review. The agent doesn't know any of that, because the agent doesn't read Slack threads from two years ago. It generates a recommendation with a confident-looking number attached. The recommendation goes into a deck. The deck heads toward a customer-facing roadmap. Whether someone catches it in QA depends on whether anyone happens to look sideways at the number.

Now scale that to a utility making real-time grid decisions, an underwriter pricing risk, or a healthcare system triaging patients. The blast radius is not the same.

Why agents make this worse, not better

Most enterprise AI roadmaps I've seen this year layer four capabilities on top of each other: ML models that predict signals, GenAI that turns data into language, agents that take action, and agentic workflows that orchestrate the agents. These aren't competing approaches. They're layered, with each tier sitting on the one below it.

That's the part the slide deck gets right. The part the slide deck doesn't make obvious is what happens to a use case as it climbs the stack.

I've been working through a use-case maturity exercise recently and the pattern is consistent. The traditional ML version of a workflow has zero agents and a couple of human checkpoints. The first agentic version of the same workflow, once you add anomaly detection, critique loops, and case routing, has eight agents. The fully orchestrated version, the one with end-to-end autonomy and a governance layer, has fourteen or fifteen.

Each of those agents is a junction. Each junction is a place where bad data or output can enter, get paraphrased into something that looks legitimate, and propagate. A confidence score gets averaged with another confidence score. A "the data quality is questionable here" caveat gets dropped because the next agent didn't have a field for it. By the time you reach the governance layer, you have an autonomous system making decisions on a foundation no single human has end-to-end visibility into.

This is the thing nobody puts on the value-prop slide.

Three responses I keep hearing

When you tell a client their data isn't ready for what they want to build, you get one of three responses. Roughly even split, in my experience.

The first is denial. "Our data is fine, we did a big modernization in 2022." Said in the same breath as a complaint about how nobody trusts the numbers in the weekly business review.

The second is "later." We'll fix it after the pilot ships. Later never arrives. The pilot becomes the production system, the production system becomes the thing nobody wants to touch because too much depends on it, and the bad data goes from being a known issue to being load-bearing.

The third is the most 2026 version of the question: can the AI fix it? Honestly, mostly yes. A competent team with the right tooling can clean up data quality issues an order of magnitude faster than the same team in 2020. But the AI can't decide what counts as fixed without you telling it what good looks like, and that's the conversation the executive doesn't want to have. The work the AI accelerates is downstream of the work the executive is avoiding.

What I've seen work

I don't have a clean framework for this. Three moves have shifted the conversation when I've tried them.

Make the bad data show up in the agent's output, not under it. The instinct is to clean the data before the agent sees it, or to filter the agent's responses to hide the ugly parts. Wrong instinct. Better to have the agent surface its own ground: "Recommending X based on Y; note that Y has a known issue affecting ~4% of rows from this source." The exec who wants to ship will still ship. The exec who's about to commit $50M will pause. Both are acceptable. What isn't acceptable is shipping and committing the $50M because the warning got buried in an appendix. You must have clear traceability for the end-to-end process that resulted in the output.

Architect skepticism in. The most interesting pattern i've seen in implementation planning and architecture reviews is the critic agent: an agent whose only job is to verify another agent's grounding and flag gaps. Pair it with a refinement agent, plus human-in-the-loop checkpoints on anything below a confidence threshold, and you get a workflow that catches its own slop before it propagates. Sequential pattern, parallel pattern, review-and-critique, human-in-the-loop. These aren't just architecture diagrams. They're the difference between an agent that fails loudly and an agent that fails quietly. The catch: a critic agent is only as good as its grounding. If your data foundation is bad, the critic doesn't know what "correct" looks like either. The architectural fix loops back to the same prerequisite.

Pick one foundation problem and fix it in public. Not a two-year program. The most-used table, the most painful field, fixed while everyone watches. Visible wins build the political capital you need for the unsexy work behind them. SAP sent an email this week with the subject line "Most companies are on their second attempt at a real data foundation." Frankly, they're underselling it. Most clients I see are on their third or fourth. The teams that break the cycle do it by stopping the all-or-nothing reflex.

Bottom Line

I'm working on a few agentic AI projects right now and I'm bullish on what this technology is going to unlock. The trap I see teams falling into is thinking the choice is binary: wait for a perfect data foundation that never arrives, or ship the agent on shaky ground and hope the bad data doesn't bite.

The third path, and the one I'd actually recommend, is to start the work. Your data does not need to be perfect to move forward. But two things have to be in place from day one.

First, visibility into what's questionable. Every output an agent produces should surface what data it's grounded in, what's known to be flawed about that data, and what the confidence level actually is. Bad data isn't the problem on its own. Bad data hidden inside confident output is.

Second, a human in the loop on anything an agent does. Not just the high-stakes calls. Everything. The cost of that review goes down as trust builds, but it can't start at zero.

This is both an org problem and a technical problem, and the answer needs both halves. The org has to be willing to surface what's broken. The technical build has to make the broken parts visible by default. Either one without the other and you're shipping slop with a confidence interval on top.

P.S

Happy mothers day to all moms out there this weekend

In the tech world, we’ve just entered our own atmospheric phase. We’ve spent years obsessing over who has the best "brain" (the model), but with the launch of OpenAI Frontier, the battleground has officially moved to the orchestration layer

The "Buy vs. Build" Dilemma

OpenAI’s Frontier is a classic "out-of-the-box" play. In theory, it solves many of the challenges we face when trying to stitch together disparate protocols (like MCP or A2A). But for those in regulated industries, this convenience comes with a steep price tag: ownership.

When you opt for a polished, third-party product, you are often surrendering core business logic to a vendor. On the flip side, the "build it yourself" approach (with various open-source protocols) offers more control and less vendor lock-in, but it’s a heavier lift. It requires significant work to package everything into something usable.

The Integration Wall

While these new products and applications solve agent orchestration problems, I still see the biggest challenge being access to the system of record.

Gaining approval to access and integrate with these core systems is the real "final boss" for AI in the enterprise and it's one that can't be solved for by third parties. A fancy agent layer doesn't mean much if it can't safely talk to the data that actually runs the business.

Strategic Shifts

It’s also fascinating to watch OpenAI go toe-to-toe with its own partners. By launching Frontier, they are now directly competing with Microsoft’s Copilot and Salesforce’s Agentforce. It’s a bold move, one that makes me wonder if this is a strategic play to shore up enterprise value as they prepare for a potential IPO.

Moving from "selling tokens" to "selling digital infrastructure" is a much stickier business model, but only if the field tests hold up in complex, regulated environments.

It will be interesting to see what is reported over the next few weeks as Frontier is trialed and tested. I look forward to seeing how others in the space respond as well.

P.S.

The pic at the top is from the Brooklyn Museum's Monet and Venice exhibit, sadly it just closed

Monet and Venice

Be transported to Venice in New York's largest museum show dedicated to Monet in over 25 years.

https://www.brooklynmuseum.org

With that said, this blog stems from the fact that we were able to buy a home here in New York City (something we don’t take for granted). How could we do this given the immense cost? The only way we could make it happen was by purchasing a property that required significant renovation. Renovations, architecture updates, plumbing, structural filings - you name it, we’re dealing with it.

And that’s where we ran headfirst into the Department of Buildings. If you’ve ever gone through this process, you know the feeling: outdated systems, painfully slow review cycles, and endless resubmissions. For us, that’s meant five separate filings and six months (and counting!) of waiting to inch forward. Our story isn't unique; it’s a systemic bottleneck. According to the city’s data from the Mayor's Management Report, the Average Number of days from filing to approval for all applications in DOB NOW (the tool used for renovation filings) increased from 8.3 days in 2020 and 11.2 days in 2021 to 20.2 days in 2024, representing an increase of over ~80%. (pg. 366). In our case, it's been more like 60 days per filing.  

I work in technology delivery, building modern data platforms, AI solutions, and portals for my clients, and have been doing so for 12 years. With GenAI and agentic architectures, it doesn’t have to be this way.

New York City is the command center for the global economy, a city that operates at the speed of light. We are home to innovators, dealmakers, and creators who define the future. We fund this city with some of the highest taxes in the nation, expecting infrastructure that reflects our status.

Yet, when it comes to the essential task of building or renovating a home, the city hands us a folded paper map in the age of GPS. We are left to navigate a labyrinth of outdated codes and static procedures with no real-time updates, no efficient rerouting around bureaucratic traffic jams, and no reliable estimate for when we might finally reach our destination.

The system isn't just slow; it's fundamentally incompatible with the city it's meant to serve.

What a Smarter System Looks Like

This isn’t science fiction; other cities are already taking action to improve this experience. In Austin, Texas, plan reviewers are utilizing an AI-powered tool that reduces the time required for certain reviews from over an hour to less than 30 minutes. Los Angeles is beta-testing a similar system for residential projects. Technology to radically change this process exists today. NYC's DOB is also involved with several AI platform PoCs, but they don't appear to be in contract with any of them (hopefully something comes of the PoCs soon)

Imagine a GenAI-powered planning examiner agent running on a modern cloud platform (Azure, AWS, Google - pick your flavor). At the center, a manager agent orchestrates the process, dispatching tasks to specialist worker agents, one for architectural design, one for plumbing code compliance, and so on.

The system could be built using frameworks like LangChain or LangGraph to manage complex workflows. With retrieval-augmented generation, the agents would constantly be checking against the latest, most obscure sections of the NYC Building Code. They could connect directly into the DOB’s systems, reviewing submitted documents in real-time.

Here’s how it would work in practice:

1. Submit: The architect uploads plans; the portal validates the files and extracts key metadata.

2. AI pre‑check (minutes): Document QA flags obvious gaps and generates an annotated checklist for quick fixes.

3. AI code pass (minutes): Specialist agents (architectural, structural, plumbing/mechanical, zoning/fire) return a single, consolidated set of redlines with direct code citations and suggested remedies.

4. Revise & resubmit (same day): The architect updates plans; a different view highlights what changed for agents and examiners.

5. Examiner review (next business day): Dashboard summary surfaces risks and citations; examiner asks targeted clarifications; agents re‑check only changed sections and mark as ready for approval or revert to plan examiner or architect.

6. Approval & recordkeeping: The system pre-fills likely TR1 special inspections with references; the examiner signs; and the permit is issued with a full audit trail and performance metrics.

This isn’t about replacing human oversight; it’s about amplifying it. Humans still make the final, nuanced calls. But instead of burning months on avoidable back-and-forth, the city could move projects forward in weeks.

Why This is a Must-Do, not a Nice-to-Have

So why push for this? Four reasons:

• Better Experience for New Yorkers: We’re paying top dollar in taxes and fees. We deserve a process that’s efficient, transparent, and user-friendly.

• Reduced Time and Costs: Skilled professionals (architects, structural engineers, etc.) shouldn't be stuck in resubmission loops. Let the tech handle repetitive checks, freeing people to focus on complex design and safety challenges.

• Addressing the Housing Crisis: Renovating or building in New York is already prohibitively expensive. As a recent City Journal report put it, New York’s permitting labyrinth is a "disaster" that drives up costs and stifles the creation of new housing, something this city desperately needs.

• Making New York a True Tech Leader: New York's own Chief Technology Officer, Matthew Fraser, recently helped launch the "NYC AI Nexus" to secure our city's place as a global leader in applied AI. What better way to prove it than by solving a core civic problem? By embracing solutions like this, New York can establish a benchmark for intelligent urban governance.

Opening the hood: A High-Level Blueprint

This isn't about a monolithic, ten-year IT project. It’s about building a simple, modular, and scalable system.

• Presentation Layer: A clean citizen portal for submissions and a DOB examiner dashboard that shows prioritized queues, AI annotations, and one-click approvals.

• Orchestration Layer: A manager agent coordinates the end-to-end workflow, using an event bus to move tasks between services so thousands of filings can progress in parallel.

• Specialist Agents: Small, focused services for architecture, structure, plumbing, zoning, fire safety, etc. Each agent is an expert in its domain, grappling with famously complex sections of the NYC Building Code, like calculating egress for a mixed-use high-rise or ensuring accessibility requirements are met and returns findings with citations.

• Knowledge & Updates: A retrieval layer (think: vector index) keeps the latest building codes, local law amendments, and precedents at the agents’ fingertips, automatically ingested and versioned.

• Integrations: Secure connectors link to existing DOB systems, NYC Open Data, and FDNY updates, keeping everything in sync and auditable.

• Infrastructure: Containerized services that auto-scale with demand, with built-in monitoring so leaders can track real SLAs like “time to first feedback.”

This isn't a complaint; it's a call to action fueled by a deep love for this city. New York has always been defined by relentless ambition and grit; that energy is being stifled by processes that belong to another era. This is not about finger-pointing. It is about seizing a crucial opportunity to modernize from the ground up and build a government as innovative as the people it serves.

Let’s start with the system that shapes our homes and businesses. Let's build a more innovative process where generative AI can eliminate bureaucratic bottlenecks and handle repetitive tasks, freeing our talented public servants to tackle complex problems. A system that empowers New Yorkers to get on with building their lives.

By transforming this one crucial agency, we can create a powerful proof point for the future. We can show the world that New York has the will to build not only the next generation of skyscrapers, but a smarter, more responsive government for its people. That is a project worthy of our city's ambition.

But this conversation is bigger than one idea, and progress requires all of us. What are your thoughts? Have you considered any new technologies we should be considering in NYC? Let me know your thoughts.

P.S.

A quiet Idaho stream below from a trail walk and a jaw-dropping lake view from the cabin porch - grateful to see it and live it.

What if there was a better way? One where you could simply ask, "Why does the BI report show outdated data?" and get a comprehensive answer that traces the issue across your entire data ecosystem – from on-premises systems to Azure Data Lake or Google Cloud Storage to that Databricks cluster that's been acting up.

My co-worker, David Delgado, and I have been thinking about this as we are having discussions with a few of our clients. How could we reduce the challenge and pain we see our teams and clients deal with all the time?

Enter the GenAI Data Agent

The breakthrough isn't just another monitoring tool, it's fundamentally rethinking how we approach data observability through intelligent agents powered by the Model Context Protocol (MCP).

What Makes This Different?

Traditional data monitoring gives you alerts. This gives you answers.

Instead of getting fifteen different notifications from various systems, you get a single, intelligent analysis that connects the dots. The GenAI agent doesn't just tell you something's broken; it tells you why it's broken, how the failure propagated through your systems, and what you need to do to fix it.

The Magic of Model Context Protocol

Here's where things get interesting. MCP, created by Anthropic, is essentially a universal translator for AI applications to connect with external systems. Think of it as the missing link that allows your GenAI agent to have meaningful conversations with all your disparate data sources.

Before MCP: Building custom connectors for every system, maintaining multiple APIs, dealing with different authentication methods for each integration. More importantly, you were stuck with hard-coded static logic for monitoring, predetermined rules and fixed decision trees that could only respond to scenarios you anticipated.

With MCP: One standardized protocol that works across Oracle databases, Azure services, Google Cloud Platform, and any other MCP-compliant system. But the real game-changer is that instead of static monitoring logic, you are now leveraging an LLM, giving it the tools to perform different operations based on its own train of thought, allowing it to dynamically investigate and get to the root of issues you never programmed it to handle.

The beauty is in the simplicity, your agent can query on-prem SQL Server, legacy Oracle instances, old MySQL instances for source data status, check on-prem replication logs, examine Azure Data Factory pipelines, and analyze Databricks processing times, all through the same standardized interface. And unlike traditional monitoring systems that follow predetermined paths, your AI agent can think through problems, form hypotheses, and adaptively choose which tools and data sources to investigate based on what it discovers along the way.

MCP is still in its infancy - we're in the early days of GenAI protocol standardization. But the potential is significant. When combined with agent-to-agent (A2A) communication protocols, we're witnessing the emergence of true cognitive architectures that can orchestrate complex, multi-agent workflows.

A Real-World Scenario

Let me paint you a picture of how this works in practice:

The Problem: Your Power BI dashboard showing customer billing data is displaying information that's 24 hours old instead of the expected daily 6AM refresh.

Traditional Approach:

• Check the Power BI dataset refresh logs

• Manually query the Azure Data Lake to see if new data arrived

• Log into on-prem data replication systems to verify replication status

• Examine your data orchestration job execution logs

• Total Time: Spend 2 hours correlating timestamps across systems

GenAI Agent Approach:

• Natural language query: "Why does the BI report show outdated data?"

• Agent(s) simultaneously queries all systems via MCP

• Correlates findings and identifies root cause: replication lag in the on-prem source system

• Provides specific remediation steps: "Clear the backlog by increasing replication throughput and optimizing transaction batching"

• Total time: 5-10 minutes

The Evolution: From Reactive to Proactive to Autonomous

This isn't just about faster troubleshooting. Real value emerges as these agentic systems evolve:

• Phase 1: Intelligent Diagnostics - Ask questions, get comprehensive answers across your entire data stack.

• Phase 2: Proactive Monitoring - The agent actively scans for issues and automatically generates detailed reports when problems are detected, complete with actionable recommendations.

• Phase 3: Autonomous Remediation - The system doesn't just identify and report issues, it automatically implements fixes within predefined safety parameters.

Imagine removing all the clutter and noise and altering you get today. Instead, the agent sends the right email to the right person that reads: "The data replication lag issue has been resolved. I increased replication throughput, optimized transaction batching intervals, and implemented proactive monitoring to prevent future delays. Data freshness is now back to normal 15-minute intervals." This would save so much time and noise across the data enterprise support team.

Why This Matters for Your Data Strategy

If you're running any kind of hybrid or multi-cloud data architecture (and let's be honest, who isn't these days?), this approach solves several critical problems:

• Complexity Management: Instead of needing experts who understand every system in your stack, you have an intelligent agent that speaks all the languages or several agents, one for each tech, and an manager agent to gain insights across all.

• Faster Time to Resolution: Root cause analysis that used to take hours now happens in minutes.

• Reduced Alert Fatigue: Instead of drowning in notifications, you get contextual intelligence about what actually matters.

• Knowledge Preservation: The agent learns from every incident, building institutional knowledge that doesn't walk out the door when employees leave.

The Technical Foundation

For those curious about the implementation, here's the high-level architecture:

• Backend: Python FastAPI with robust async capabilities for handling multiple simultaneous system queries

• Message Queue: Kafka for real-time data streaming and event processing

• Container Platform: Kubernetes for scalability across hybrid environments

• AI Orchestration: LangChain/LlamaIndex/LangGraph framework with Azure OpenAI integration

• MCP Integration: Custom MCP servers for each data source, standardizing communication protocols

The key insight is that this isn't just another dashboard or monitoring tool – it's an intelligent layer that sits above your existing infrastructure and makes sense of it all.

Looking Ahead

We're still in the early days of this technology, but the potential is enormous. As MCP adoption grows and more vendors create compliant interfaces, the dream of truly unified data observability will become a reality.

Perhaps equally important is the comprehensive audit trail this could create. Instead of scattered email chains, Slack messages, and tribal knowledge about what went wrong and how it was fixed, your LLM agent or agents automatically logs every investigation, decision, and action it takes. This creates a robust, searchable dataset of your data operations history. Imagine being able to query "What caused similar pipeline failures in the past six months?" or "How did we resolve that Oracle connectivity issue last quarter?" Your institutional knowledge becomes structured, persistent, and accessible rather than lost in someone's inbox or memory.

The companies that get ahead of this curve won't just have better data operations, they'll have a fundamental competitive advantage. While competitors are still playing whack-a-mole with data issues, these organizations will have intelligent agents proactively optimizing their data flows and preventing problems before they impact the business.

The Bottom Line

Data infrastructure is becoming too complex for human-only management. The future belongs to organizations that augment their teams with intelligent agents capable of understanding, diagnosing, and eventually healing their data ecosystems autonomously.

The question isn't whether this technology will transform how we manage data, it's whether you'll be an early adopter or play catch-up.

What are your thoughts on AI-powered data observability? Are you already experimenting with GenAI in your data operations, or are you taking a wait-and-see approach? I'd love to hear about your experiences in the comments.

If you're interested in exploring how these concepts might apply to your data architecture, feel free to reach out. Sometimes the best insights come from a good conversation about the messy realities of enterprise data.

P.S.

We had a family wedding in Washington, which was a wonderful time. The picture at the top is from our drive to Idaho. We pulled over on the side of the road, and I snapped it quickly. It really makes me appreciate getting out of the city every now and then.

In Idaho, I came across my favorite road sign. Our old family name, “Viken,” before it was anglicized to “Wigen.”

While many bloggers have focused on the deregulation and change to Biden era policy, I thought I’d focused on the impact on the utility sector (which I serve at West Monroe). Below, I've summarized four key areas utilities should be aware of as it relates to power generation, GenAI, and regulation changes. I'd love to hear your thoughts on these insights and anything important I might have missed.

1. AI-Driven Grid Modernization

The AI Action Plan identifies the electric grid as crucial AI-supporting infrastructure, emphasizing the need for stability, optimized transmission resources, and integration of reliable, dispatchable energy sources such as geothermal and nuclear.

Key actions include:

• Preventing premature closures of critical generation assets.

• Supporting demand-side management initiatives to manage peak loads.

• Investing in reliable, dispatchable energy sources.

• Upgrading grid management technologies to boost efficiency and resilience.

Based on this new policy, utilities will receive strong federal backing to aggressively modernize grid infrastructure with AI-enabled tools. This modernization includes load forecasting, outage prediction, distributed energy resource (DER) orchestration, and peak load management.

However, the plan notably misses an opportunity by not explicitly supporting investments in renewable energy sources, especially following recent policy changes (aka the One Big Beautiful Bill) that reduced support for solar and wind energy, one of the fastest growing and ever cheaper energy generation technologies.

2. Streamlined Permitting for Energy & Data Infrastructure

The AI Action Plan’s message is clear: "Build, Baby, Build!" The administration aims to fast-track permitting for critical infrastructure like data centers, semiconductor facilities, power generation, and grid infrastructure.

Key provisions include:

• New categorical exclusions under the National Environmental Policy Act (NEPA) to speed up data center-related actions.

• Expansion of FAST-41 processes to streamline permitting for data centers and energy infrastructure projects.

• Allocation of federal lands specifically for data center and power generation projects.

Permitting bottlenecks significantly delay utilities' critical infrastructure projects. Streamlined permitting could dramatically shorten these timelines, enabling utilities to rapidly build new substations, transmission lines, and generation assets.

Co-locating data centers with power infrastructure will likely unlock new business models around grid-adjacent compute and will likely see utilities packaging up large generation projects with data center build efforts. Further aligning utilities to large tech vendors (GCP, AWS, Azure, etc.)

3. AI Adoption Acceleration Across Sectors

The plan emphasizes addressing slow AI adoption in legacy sectors, including energy, by establishing regulatory sandboxes and Centers of Excellence. These initiatives aim to foster safe experimentation and standardize AI deployment.

Key actions include:

• Establishing AI regulatory sandboxes for safe, real-world experimentation.

• Developing industry-specific standards through the National Institute of Standards and Technology (NIST).

• Encouraging the measurement of AI productivity impacts.

Utilities can leverage these regulatory sandboxes as safe environments to pilot new AI applications in customer service, fraud detection, predictive maintenance, and DER forecasting. The development of energy-specific AI standards by NIST presents an opportunity for utilities to influence these guidelines and accelerate their AI initiatives.

4. Cybersecurity & Secure-by-Design AI for Critical Infrastructure

AI offers significant potential for enhanced cybersecurity but also creates new vulnerabilities. The Action Plan promotes secure-by-design AI and proposes the creation of an AI-specific Information Sharing and Analysis Center (AI-ISAC).

Key actions include:

• Establishing an AI-ISAC within the Department of Homeland Security (DHS) for critical infrastructure sectors.

• Issuing private-sector guidance on threats specific to AI, such as data poisoning.

• Promoting the sharing of AI vulnerability intelligence between government and industry.

As primary cyber targets, utilities must proactively integrate secure-by-design AI systems. Participation in the AI-ISAC will provide utilities with critical threat intelligence, enabling better protection of grid assets and customer data from AI-specific cybersecurity risks.

Final Thoughts

The AI Action Plan clearly positions utilities as the critical players in America’s future AI infrastructure, they will shoulder the massive energy and infrastructure demands that come with this AI growth. This plan gives utilities the green light and strong federal encouragement to aggressively modernize infrastructure (new power generation and transmission), accelerate AI adoption in their operations, and strengthen cybersecurity measures.

This strong support indicates to me that the government expects utilities to move swiftly and lead in these areas. How utilities respond will be telling as they are often "fast followers" rather than first-movers for any new-technology. But with such encouragement (reduced red tape, push for AI in operations, and resources for AI-driven grid modernization) we may see a shift in that cautious mindset.

The coming years will show whether utilities seize this moment to accelerate, or if they require further pushes to break from more conservative habits. Either way, the federal vision is clear: utilities are expected to power and protect America’s AI era. I'm interested to see how utilities will react.

P.S.

The picture at the top is the Chicago River Walk, great view of the loop

As I explained to the client, upgrading your on-prem data warehouse without migrating to the cloud often means paying twice, once now, and again next year when you inevitably need to redo the effort in the cloud.

In this article, I'll outline when and why you should consider on-prem versus cloud solutions for your data infrastructure to get my thoughts on the page.

Scalability and Performance

Cloud solutions offer instant scalability, allowing you to quickly adjust resources as workloads fluctuate. This flexibility helps avoid the expensive upfront costs associated with on-prem hardware. Additionally, cloud environments typically provide access to the latest hardware innovations, ensuring your data infrastructure stays modern. However, the cloud can introduce variable performance and latency issues, especially if resources are shared or geographically distant from end-users.

On-premises solutions offer consistent and predictable performance, valuable for high-performance applications requiring low latency. Yet, scaling up an on-prem environment involves considerable lead time and substantial capital investment.

Security, Compliance, and Governance

Cloud environments offer advanced, built-in security features and certifications, enabling organizations to quickly meet compliance standards while reducing security management overhead. However, the shared responsibility model (this is the AWS version; GCP and Azure are similar) requires companies to carefully manage their security responsibilities and trust third-party providers.

Conversely, on-premises infrastructure provides complete control over security, enabling highly tailored compliance policies. This total control comes with significant overhead, requiring internal expertise and continuous investment.

Cost Management

Cloud platforms typically offer a lower upfront investment, featuring an operational expenditure (OpEx) model that aligns costs directly with usage, ideal for variable workloads. However, cloud billing can be complex, with hidden costs such as data egress fees resulting in unexpected expenses. You'll need to stay on top of your billing and optimize spend.

On-premises solutions require higher initial capital expenditure (CapEx), but once in place, they deliver stable and predictable costs, making them suitable for consistent, high-volume workloads. The trade-off is less financial flexibility and ongoing costs even during underutilization.

Operational Complexity

Cloud providers manage most of the underlying infrastructure maintenance, significantly reducing the operational burdens on internal teams and allowing organizations to focus on strategic initiatives. Yet, effectively managing cloud environments requires specialized skills, especially with multi-cloud or hybrid setups.

On-premises infrastructure grants complete control, simplifying troubleshooting with direct oversight but demands considerable ongoing maintenance and a highly skilled team to support.

Innovation and Modern Tools

Cloud environments enable rapid innovation, offering immediate access to analytics, artificial intelligence, and machine learning tools. Leveraging cloud infrastructure allows faster experimentation and adoption of new technologies. However, rapid advancement can lead to vendor lock-in, making future changes challenging.

On-premises environments provide stability and deep customization, offering controlled and predictable technology progression. However, this often results in slower adoption of new technologies and more limited tool availability.

GenAI and the Data Foundation

Most importantly, the public cloud is the most direct path to GenAI. Hyperscalers provide elastic GPU/TPU clusters, vector databases, and model-tuning pipelines, which are costly and time-consuming to set up on-prem (unless you're Meta, you should not be doing this). Even more valuable is proximity: when curated data, feature stores, and LLM endpoints live in the same cloud tenancy, teams can move from raw data to a production chatbot in weeks instead of quarters.

On-premises solutions still play a role, particularly in steady-state inference with tight latency or workloads behind strict sovereignty walls, but the vast majority of teams prototype, fine-tune, and scale GenAI in the cloud first, repatriating only what makes economic or compliance sense. In short: if GenAI is on your roadmap (which it should be), cloud needs to be in your toolbox.

Wrapping it Up

The reality is, most organizations choose a hybrid solution. Highly sensitive and regulated workloads remain on-premises, while dynamic, innovation-driven workloads sit in the cloud.

However, the key point is that by 2025, nearly everyone should have some cloud footprint.

The flexibility, scalability, and rapid innovation of the cloud make it a vital component. Whether you're just starting your journey or rethinking existing infrastructure, ensure the cloud is a strategic part of your roadmap. And if you're still on the fence, or just want a second opinion, feel free to reach out. It’s a choice you'll thank yourself for later.

For deeper insights into cloud considerations, check out these resources from major providers:

• Azure: Well-Architected Framework – Data & AI Pillar

• AWS: Analytics Lens for the Well-Architected Framework

• Google Cloud: Data Analytics Landing Zone Design Guide

P.S.

Gotta love New York

She didn’t get to see the incredible acceleration of change we’re experiencing today with AI and GenAI. But I imagine if I’m fortunate enough to share similar conversations with my grandkids, AI will undoubtedly be a focus of mine.

This week at our Chicago HQ, my colleague Cam Cross and I found ourselves reminiscing about the launch of ChatGPT two-and-a-half years ago, talking as if it were ancient history. In the fast-paced world of GenAI, it practically is. The evolution from simple email-crafting assistants to today's sophisticated AI agents supporting entire data engineering teams is both astonishing and inspiring. I’m now seeing (and West Monroe is developing/using) AI-driven accelerators capable of building data warehouses in mere days, tasks that once required months.

Consulting at a Crossroads

The consulting industry is poised for dramatic change. Large, cumbersome teams could soon be replaced by small, agile pods of consultants paired with advanced AI agents. Offshore resources may be significantly reduced, replaced by always-on, contextually aware AI assistants. Consultants will seamlessly manage multiple clients simultaneously, leveraging AI to rapidly onboard and switch contexts without losing depth or focus. Eventually, and who knows if this is years or decades, the need for consulting could be filled by incredibly aware, personalized agents.

What Stays the Same

But amidst this seismic shift, what remains constant, and perhaps becomes even more critical, is our humanity. Trust, empathy, and authentic relationships cannot be automated. The genuine connections we build with our clients and colleagues are irreplaceable. AI may enhance our productivity and creativity, but the core of consulting and all business will always be deeply human.

Staying Relevant in the GenAI Era

To thrive in this evolving landscape, consultants and firms must:

• Commit to continuous learning: Rapid mastery of emerging AI tools is essential, there should be internal functions to uplevel and inform teams of new tech, updates, and changes that come with them.

• Prioritize uniquely human skills: Strategic thinking, empathy, and relationship-building become key differentiators.

• Blend technology with human insight: Success hinges on effectively integrating cutting-edge technology with an understanding of human dynamics and unique needs/context of the client, industry, and business.

Despite the rapid pace of change and the challenges ahead, I remain optimistic. The current wave of innovation is democratizing access to powerful capabilities that drive creativity, productivity, and strategic thinking across industries. Staying ahead can feel overwhelming, but the possibilities for growth and meaningful impact are incredibly motivating.

I’m looking forward to someday sharing these stories of transformation with my grandkids. When I do, I'll echo my grandma’s sentiments: The technology was incredible, but it was the people who embraced it together that truly impacted me.

P.S.
My fellow DePauw Tigers at West Monroe got together for a group photo this week at our HQ. I'm always grateful for the role DePauw has played in shaping my career and the lasting connections it's created.

The picture in the header is from my visit to Japan, staying at the hot springs, it was unbelievably beautiful!

But if there's one factor that consistently makes or breaks GenAI success, it's the quality of the underlying data.

That's why we're seeing such massive demand for data engineering at West Monroe. More than ever, data is the most valuable asset companies have. Clean, consolidated, and high-quality data isn't just a nice-to-have; it's the foundation that determines whether your GenAI efforts scale or stall.

Because here's the truth: GenAI is only as good as the data you feed it.

Right now, companies are pouring resources into GenAI pilots, only to hit roadblocks when their models surface broken data, inconsistent definitions, or disconnected systems. Sure, LLMs can write emails and generate code. But when you want to move beyond these capabilities, truly unlocking GenAI's value, you need structured, governed, and reliable data. This is especially critical when powering custom use cases that drive customer insights from unique/internal datasets, support internal staff based on specific business policies and procedures, or personalize experiences based on your core applications or services.

If your data is your fuel, GenAI is your engine. You can't get very far on fumes.

Getting your data ready means:

• Structuring and storing it consistently so that GenAI can reason over it

• Securing it so it can be responsibly used

• Labeling and tagging it to ensure relevant context is captured, so GenAI tools can understand what it means

• Governing it so you know what's being generated and why

The companies winning in the GenAI era don't just have the best tech or use cases; they're the ones that did the upfront data work to build clean, connected data ecosystems, making their GenAI outputs both trustworthy, traceable, and consistent.

Before chasing new use cases, ask yourself: Is your data ready?

Because in the world we're entering, companies that don't harness their data won't differentiate, innovate, or win.

Don't wait until your GenAI projects falter; start building your data foundation now.

P.S.

Amazing clouds this morning over the Williamsburg Bridge.

I've seen firsthand how a company's value increases during the deal process if it has strong data and analytics programs and views data as an asset. The 2021 RealPage acquisition by Thoma Bravo is a great example of this. RealPage provides software to rental housing owners/managers, aggregating rental market data across millions of apartments. This data significantly increased the company’s strategic value, enabling TB to leverage it beyond property management alone. TB paid $10.2 billion (a ~30% premium) for RealPage. Not too shabby.

I worked alongside Douglas Laney (one of the leading voices in data monetization and the Infonomics space) during his time at West Monroe. His approach to data valuation left a strong impression on me. As I'm applying his methodology for my friend's portfolio company, I thought I'd share the framework and a few takeaways.

Douglas Laney’s Infonomics framework is a great starting point for understanding data value. (if you want more details, check out his book here). Doug looks at data monetization across three primary pillars:

1. Intrinsic Value - How unique, complete, and accurate is the data?

2. Business Value - How does the data help drive internal efficiency, cut costs, or boost revenue?

3. Market Value: What is the external market willing to pay for this data, whether through partnerships, licensing agreements, or even outright sale?

Most companies I work with are looking at data for intrinsic and business value - few are thinking about the market value of their data. I'd argue that all companies should see data not as a mere byproduct of their business but as a tangible asset with measurable value.

When applying Infonomics principles to a company, I typically follow three overarching steps to vet data across the Infonomics pillars:

• Understand Operational Value: How does data support internal decision-making and workflows? What gaps, issues, and challenges exist within the current data and data environment?

• Identify Tools & Platforms: Is data locked away in spreadsheets or older Access databases, or is it being leveraged in modern analytics platforms (e.g., Databricks, Fabric, etc.)?

• Quantify Impact: Are there clear metrics on how data contributes to revenue, cost savings, or efficiency gains?

• Look Beyond the Current Company: Are there opportunities for this data to enable and enhance third parties or other industry leaders? For example, a lawn-care company's data about lawns, exteriors, and property attributes could be extremely valuable to the real estate, insurance, and home improvement sectors.

• Research External Demand: Are there untapped markets or verticals that could benefit from these unique data sets? What's the market opportunity and demand for this data? Within PE specifically, are there portfolio company synergies? If so, this is an excellent opportunity to enhance the overarching portfolio value.

• Enterprise Asset Valuation: What's the actual book value of the data? Work with third-party firms or valuation experts to quantify the financial worth of the data. One I've worked with before is gulpdata. These companies look at unique attributes of your data, vet them against other data valuation examples, and support firms in getting loans against their data.

Once you have a complete picture of your data and decide it represents an opportunity, consider which strategies best align with your objectives and investment appetite.

• Data Cleaning, Governance, Reporting, etc.: Immediate, lower-investment gains by improving data quality and internal reporting capabilities. This is typically where most low-maturity companies can invest less to move the needle on data capabilities.

• Analytical Tools: Develop enhanced dashboards or predictive analytics that boost operational efficiency. This is a bit more costly and requires good data quality, tools, etc. Ideally, you integrate these insights into your existing systems and products, enabling digital analytics. Of course, you could look at LLMs here too.

Mid-Term: Portfolio-Wide Data Exchange

• Shared Insights Platform: Enable data sharing and collaboration among portfolio companies to foster cross-company analytics and innovation. While more of a lift, it can create much value across the portfolio. It lets you leverage your data assets as strategic differentiators in acquisition scenarios.

Long-Term: External Product Development & Monetization

• Build New Products: Develop new, marketable products based on your data that appeal to external customers or sectors and or license data, pursue partnerships, or even consider outright sales to third parties or new market entrants. This approach demands significant effort, product teams, and legal alignment but can significantly amplify your company’s long-term value. Potential roadblocks (such as privacy, legal complexities, and market positioning) exist, but the potential benefits usually outweigh these challenges.

After exploring your data’s potential value and strategic options, choose your path:

• Internal optimization only?

• Portfolio synergies?

• External monetization/product development?

External monetization typically demands the most energy and commitment—but as they say, no risk, no reward.

Data valuation is undoubtedly a buzzword, but it definitely has real-world potential. Following the strategic data valuation process can enable significant opportunities to enhance a company’s market position and long-term growth. By treating data as a formal asset—guided by Infonomics principles—you can pinpoint exactly where to invest resources. If done right, data monetization boosts operational efficiency and opens up new revenue streams for companies.

Not sure where to start with your data valuation? Drop me a note—I’d love to discuss your specific challenges and opportunities. Whether in private equity, a startup, or an established enterprise, take a fresh inventory of your data. Are you harnessing its full value?

P.S.

Spring is starting to bloom in Brooklyn. The photo at the top captures my favorite tree in Fort Greene Park—it reminds me how looking with fresh eyes at familiar things (like your data!) can reveal unexpected potential.

I love this conference—every year, I learn something new, and it's fascinating to see how much the landscape evolves. In 2024, the discussion centered on "What are LLMs, and how will they impact your funds?" This year, the conversation shifted to "We are building and seeing others build unique LLM tools that are accelerating fund teams…if you aren't doing this, you're behind."

I thought I'd share a few key takeaways from the session below—these insights apply across industries, not just private equity. Although, the pace and impact will vary.

Details about WM's fund of the future POV have been posted on WM's website here. A big shoutout to EJ and Brad, who led the event—they did a fantastic job.

AI can absolutely make investment (and other) decisions faster and more accurately, but making bad decisions faster is still just bad decision making. AI works best when it augments human judgment—not when it replaces it. The best PE firms and companies know how to balance efficiency with accuracy and always keep humans in the loop to vet and validate.

Brad and EJ noted that AI differentiation is 70% people, 20% data, and only 10% GenAI tools. (I would argue it's 50% people, 40% data, and 10% AI tools…but that's neither here nor there.) Companies love to chase the latest AI software, but without the right people to guide, interpret, and adopt it, you're just wasting time. The real value happens when leadership focuses on upskilling teams, fostering AI literacy, and embedding AI into workflows to elevate employees.

Thankfully, I haven’t seen companies spending millions trying to build custom LLMs (unless they are Meta, OpenAI, xAI, etc.). There are many off-the-shelf solutions that, with a little tweaking, do the job just as well. The winning approach is to leverage existing LLMs and tools, then customize them with your proprietary data and specific needs. This way, you achieve differentiation without burning your budget on unnecessary development.

While data science is at the heart of the GenAI "engine" (and I recommend having a DS lead to help wrap your arms around these technologies, architectures, and best practices), the core teams customizing, developing, and delivering GenAI solutions are more product development focused. Prompt engineering doesn't require a data scientist, but the UI that enables the end user to derive value from the LLM does require a product developer.

Last year, I spent a lot of time explaining what data science was and how it fits within the data maturity curve. We had executives saying they wanted to implement data science solutions, but their data infrastructure consisted of Excel and Access databases. AI isn’t a magic bullet that fixes no data, bad data, or messy environments. If your data is flawed, AI will just help you make bad decisions more efficiently. Before diving into AI, firms need to:

• Appoint a Data Lead – Someone responsible for decision making, driving adoption, and championing these efforts as a core part of their role.

• Build a Modern Data Ecosystem – A data platform that integrates structured and unstructured data.

• Prepare Data – To gain broader insights and enhance decision making get your data in one place, organize it, clean it, and label it.

During the session we got the question (we did in 2024 too), "what's the ROI here, why should I spend on this?" Frankly, while GenAI is driving value, quantifying that value remains challenging so I wanted to follow up on it.

Ahead of the sessions, WM's PE team interviewed many of our PE partners and reviewed work we had done to date. They found that GenAI is impacting four key areas:

• Better Investment Decisions – AI helps analyze structured and unstructured data faster and more accurately.

• Faster Deal Sourcing – AI can surface high potential deals before competitors even know they exist.

• More Efficient Operations – AI can automate non-core tasks, freeing up employees for strategic work.

• Smarter Investor Relations – AI powered tools are transforming how firms engage with investors and report on portfolio performance.

The challenge is that these efficiency gains are hard to measure. As luck would have it, a PE friend recently shared a research article on this very problem. In the paper, the authors find the following data points and key value drives for workers using GenAI:

Use Cases and Time Savings

• Workers use GenAI primarily for writing, administrative tasks, data analysis, coding, and summarization.

• On average, GenAI assists with 1% to 5% of total work hours.

• Users report a 5.4% time savings in their weekly work, translating to significant productivity gains.

Productivity Impact

• Higher GenAI use correlates with higher wages—frequent users earn up to 40% more than non-users.

• Estimated aggregate productivity gains of 1.1% based on current adoption rates.

• Managers and tech workers use GenAI more than administrative roles, despite predictions that office jobs would benefit most.

I thought figure 11 in particular was compelling for the "what's the ROI" question (citation below):

The key takeaway: There is undeniable value, but you need to vet and quantify it as part of your strategy. Personally, I believe that if you're not using these tools and ROI is your barrier to entry, you're likely overthinking it. These solutions cost between $5,000 and $12,000 per month typically for a small firm—if you're able to save just 1% of your employees' time, the ROI is there. The bigger risk not training your workforce on these tools, if you're competition is doing it you're falling behind.

AI adoption isn’t just an IT project—it’s a business transformation. During the working session the team outlined how to get started if you’re a leader thinking about AI:

1. Build an AI Leadership Team

   • Set a clear AI strategy and define what success looks like.

   • Keep AI adoption measurable and accountable (no "innovation theater").

   • Assign ownership—someone needs to drive this, not just talk about it.

2. Pick Your First AI Wins (And Don’t Overcomplicate It)

   • Find quick wins (AI powered research tools, document summarization, etc.).

   • Identify big bets (investment scoring, predictive analytics, automation).

3. Test, Learn, and Iterate

   • Don’t get caught up in perfection—AI is an evolving tool, not a one time install.

   • Pilot AI tools, measure impact, and adjust as needed.

GenAI tools are here and you need to be using them. There is a ton of value to be had but only if companies approach it the right way. Whether you’re in PE, utilities, healthcare, or any other industry—lead with strategy, invest in people, and let GenAI be the accelerator, not the driver.

Would love to hear how you're thinking about and addressing GenAI in your workplace!

citation:

Bick, A., Blandin, A., & Deming, D. J. (2024). The rapid adoption of generative AI (NBER Working Paper No. 32966). National Bureau of Economic Research. https://www.nber.org/papers/w32966

P.S

The picture at the top is from Lake Coeur d'Alene. I've been going there almost every summer since before I can remember—it's a beautiful place.

Recently, I was chatting with a client about their struggles with ever-changing file formats from vendors, third-party partners, and other external sources. These changes often cause data pipelines to break—alerts go off, hotfixes are needed, and engineers scramble to patch things up (I was one of those engineers for a long time). They asked me how I've handled these issues while keeping solutions flexible, structured, and easy to manage.

I’ve run into this challenge repeatedly, especially in my experience leading healthcare analytics data platform and AI use case builds. In one project, we had to process hundreds of claims payment documents in PDF, Excel, and flat file formats from multiple payers—files that changed on what felt like a monthly schedule. Each time a payer tweaked the layout or added new columns, our ETL jobs would fail. We’d scramble to update mappings, rework transformations, and re-deploy quickly to avoid pipeline downtime. It wasn't sustainable and we had to iterate to find the right approach to solve the problem.

Before diving into architectural solutions, it’s worth noting a couple of tools that can ease the burden (...and no neither Databricks, DataForge, or Fivetran are paying me here, so take these recommendations for what they're worth). Databricks Auto Loader can dynamically ingest files and handles schema drift, I've seen it used a few times and is great for anyone using DBx and willing to do some development to solve this problem. Additionally, Fivetran (a favorite in DBx environments) can automate ingestion, manage schema drift, and alert users to attribute changes for these files. It's a bit more low code/no code. DataForge is another tool I’ve used—the founders are former colleagues of mine, and they’ve written extensively on this topic. It provides an effective way to handle schema drift in data ingestion workflows. There are many other tools that do this but these have been used by several of my clients and have had really good results. They don’t eliminate the need for a solid data architecture and strategy but they can fit into that architecture.

Outside of tools, there are a few other approaches to consider. I take a look at the standard third normal form approach and two others below. I recognize that there are many other ways to solve for this problem (schema on read/views, etc.) but focused on these for the purposes of this post.

This approach follows standard relational modeling principles—atomic values, minimal redundancy, and key-based relationships.

Pros

• Removes redundant data, reducing storage costs and improving consistency.

• Enforces data integrity through key relationships.

• Efficient for querying with well-defined schemas and indexing.

• Great for transactional systems requiring strong consistency.

• Plays nicely with star schema modeling when data is well-structured.

Cons

• Can get complex with many dimensions, fact tables, and crosswalk tables.

• Schema changes require updates, which can break ETL processes.

• Needs ongoing maintenance (performance tuning, indexing, etc.).

• Less flexible for handling semi-structured data.

• Not ideal for API-driven architectures that prefer JSON.

Key Takeaway
Use 3NF when data structures are relatively stable, or when strong consistency and integrity are paramount. It’s powerful, but schema changes can be painful—plan for regular maintenance cycles and version control to handle evolving requirements.

Storing data as JSON objects offers more flexibility, reducing schema-related ETL failures when fields are added or removed.

Pros

• Reduces schema update requirements; easier for data to evolve over time.

• Improves query performance by reducing the need for joins assuming a "One Large Table" approach.

• Supports modern applications that natively work with JSON.

• Can store precomputed measures to optimize query times.

Cons

• Can get messy if users aren’t familiar with “wide-table” (OLT) models.

• JSON querying can be slower due to nested structures.

• Storage costs can go up due to duplicated data.

• Requires extra processing for JSON parsing and transformation.

Key Takeaway
JSON is a powerful option for managing semi-structured data and adapting to frequent schema changes, but it comes with trade-offs. Performance and cost considerations should not be overlooked, as querying large nested structures can be inefficient.

Additionally, working with JSON and wide-table models requires a different mindset—developers and power users will need training to effectively navigate this paradigm. If your workflow relies heavily on self-joins, be prepared for potential complexity and performance overhead.

A hybrid approach blends structured data with flexible JSON storage, aiming to strike a balance between data integrity and adaptability.

When to Consider This Approach

• Some attributes are stable, while others change frequently.

• Core data and frequently queried attributes live in structured tables.

• Rarely queried or dynamic attributes are stored in JSON.

• Your database supports mixed data types.

• Your team is comfortable with performance trade-offs and query complexity.

Key Takeaway
The hybrid approach is often a sweet spot for teams dealing with frequent schema changes on certain attributes but still needing a robust relational backbone. You get the best of both worlds, but it demands solid governance to track where each piece of data resides.

• Versioning: Maintain a version history of your schemas. This way, you know exactly which schema was in use when data was ingested.

• Documentation: Keep clear documentation of which fields are in your structured tables vs. your JSON columns. This reduces confusion when changes inevitably occur.

• Alerting & Monitoring: Even with flexible storage, you want alerts when new fields appear. Tools like Databricks Auto Loader or Fivetran can notify you of schema changes immediately.

• Data Governance: Have a plan for how new fields or attributes get validated, labeled, and whether they belong in structured or unstructured sections. This prevents “sprawl” over time.

Before picking an approach, ask yourself:

• How often is this data queried? Frequent queries may justify a structured approach for performance.

• Does it need to integrate with APIs? JSON-friendly storage might be better if API integration is key.

• How many records are we dealing with? Large volumes of semi-structured data might need a scalable, flexible design.

• How frequently does the schema change? A very dynamic schema pushes you toward JSON or hybrid solutions.

Answering these questions will help you choose the best model. Remember, there’s no one-size-fits-all. The hybrid approach often provides the right balance, but you need a team comfortable with managing both structured and semi-structured data efficiently.

Schema drift is an unavoidable challenge in data engineering, but there are proven strategies to tackle it. Whether you choose a traditional relational model, a flexible JSON approach, or a hybrid solution, the key is understanding your data’s usage patterns and anticipating future evolution.

At the end of the day, data architecture is all about trade-offs. I love digging into these kinds of challenges, and I hope this breakdown helps you think through the best approach for your own platform needs.

Got thoughts or experiences dealing with schema drift? What’s the trickiest schema drift issue you’ve faced, and how did you solve it? Do you have a favorite tool or framework for managing unexpected file format changes?

Drop a comment—I’d love to hear how you’re tackling it!

P.S.

At the top of the post is a photo I took of a piece of art currently on display at the Brooklyn Museum. It’s one of those pieces that makes me think, "I could have done that"—but I didn’t. I don’t have the experience, background, or understanding of art to have created it. The artist is Jaye Moon... and I like her work!

UPDATE 2025/02/24

I received some feedback from my colleague and Databricks MVP, Doug MacWilliams.

Doug suggests leveraging Delta Lake schema enforcement within Databricks to manage schema drift. This method works well when handling frequently changing file formats in a Medallion architecture (bronze to silver to gold).

• Strict Schema Matching: If incoming data doesn't align with the existing Delta Lake table schema, an error is triggered.

• Ensures Stability: This prevents unintended schema alterations, maintaining a stable model/schema.

• Automatic Adaptation: When enabled, new column from incoming data are added without overwriting existing records.

• Manages Changes: It handles renamed or removed columns, preserving historical data integrity while accommodating new attributes.

• Requires Maintenance: Periodic cleanup is necessary to maintain consistency.

• Consistent Progression: As data moves from bronze to silver to gold layers, mapping into a consistent schema supports consistent insights/querying/etc. There is work to do there as well.

This approach offers flexibility while keeping data clean and consistent (with a little work) for end users—a great choice for Databricks-based platforms.

Appreciate the feedback, Doug!

In January, I had the chance to grab dinner with Ben and Yassin, two of the founders of Osgil—an AI platform that creates traceability for LLM-generated content. Over a plate of Kyrgyz Ganfan, they shared their backgrounds, stories about late-night coding sessions, navigating regulatory hurdles at previous companies, and the “aha” moments that shaped their vision for Osgil.

What struck me most wasn’t just their technical prowess (which is impressive) but the problem they’re aiming to solve—the current compliance headache that is GenAI in the financial sector. It also made me consider how many other highly regulated industries (utilities, healthcare, etc.) that I work with could benefit from their solution. Osgil’s platform, PAX Studio, is built to ensure AI-driven document generation is transparent, auditable, and above all, regulator-friendly.

Financial companies often must generate complex compliance reports that regulators heavily scrutinize. Traditionally, this has involved laborious data collection, manual reviews, and potential for error. PAX Studio replaces that tedium with a no-code templating system that analysts can use to speed up report creation—complete with traceability and auditability—without writing a single line of code. Every action is tracked, every AI output is verified, and the tool only references data provided by the company, enabling clear traceability (to the extent possible). Their tool removes the black box of LLM-generated content, and I can see it being incredibly useful to many businesses.

Over dinner, we also discussed the relentless pace of AI. Models evolve quickly, and new capabilities emerge every day. While this opens incredible opportunities for innovation, it also creates challenges for GenAI startups like Osgil.

1. Differentiation in a Crowded Market - As more enterprises develop their own AI solutions in-house, it’s crucial for startups to stand out. Osgil’s edge is in auditability and compliance—two factors that financial institutions value above all else.

2. Adapting to Change - When AI advancements arrive, rigid systems can become outdated overnight. PAX Studio is architected for flexibility, allowing new models or techniques to be integrated without overhauling the entire platform. In an industry that’s constantly evolving, Osgil’s bring-your-own-model strategy means you’re not locked into a single LLM as new breakthroughs emerge.

3. Regulatory Pressures - In finance, a single compliance slip can have a big impact. Osgil’s commitment to full auditability and alignment with evolving regulations builds trust in an industry that can’t afford to take risks. The traceability improves confidence in LLM-generated reports and quickly highlights what was or wasn’t reviewed by a human—enforcing the “human in the loop”. 

The adoption of GenAI in the financial sector (and many others) is inevitable—but how it unfolds is still being written. Companies like Osgil are shaping this future by making compliance, governance, and verifiable outputs easy for large institutions.

It’s easy to see how the Osgil solution could extend into other heavily regulated spaces—like healthcare, where patient privacy is paramount and highly regulated, or utilities, where energy providers must follow strict regulatory compliance standards. By offering traceable, auditable AI content generation, Osgil could empower organizations across multiple industries to innovate while remaining compliant.

In the end, it’s the human stories behind these platforms that drive genuine innovation. I’m excited to see what’s next for Ben and Yassin. Watching the next generation blend vision, hard work, and a bit of risk-taking is always inspiring—especially when they’re tackling problems that many industries are facing. I’ll be watching—and cheering—Osgil on as they continue solving the GenAI compliance challenge.

P.S.

What a difference a few days can make - the picture is of snow on the rooftops in Brooklyn earlier this week.

To get ready for my QA responsibilities, I went through formal QA training to refresh my understanding of how to ensure our engagements not only meet but exceed client expectations. It was a solid reminder of best practices and also got me thinking about how we could improve the QA process with data and AI (more to come).

By the end of the training, I realized just how much potential this opportunity holds. As a QA lead, I get to meet new clients, check in on teams I don’t always work with, and see alternative approaches to problem-solving. It’s a chance to broaden my perspective and help ensure we deliver top-notch outcomes.

Quality Assurance is all about validating that an engagement is on the right track. It checks:

• Deliverables: Are we providing what we promised (or more)?

• Client Expectations: Are we meeting or exceeding what clients really want?

• Process Compliance: Are we following our established guidelines for finance, change orders, and so on?

Ultimately, QA is a guardrail to ensure we’re doing right by our clients. As Peter Drucker said, “Quality in a service or product is not what you put into it. It is what the client or customer gets out of it.”

Stepping into the QA reviewer role at our firm has been both strategic and hands-on. Here’s what it looks like in practice:

Setting the Foundation: Once assigned as the QA lead for a project, I work with Engagement Leads (ELs) and Project Managers (PMs) to establish a clear QA cadence. We align on what “success” looks like and confirm everyone’s on the same page.

Monitoring Progress: Throughout the engagement, I conduct periodic reviews to assess health and progress. This helps surface potential issues early. By collaborating with delivery teams and client stakeholders, we can fix minor hiccups before they become major headaches.

Aligning Expectations: I constantly track whether deliverables match (or surpass) the client’s stated objectives. Keeping that alignment front and center makes sure there are no surprises down the road.

Enhancing Collaboration: Sometimes, the biggest risk to a project is poor communication. Part of my job is to spot communication gaps and help teams close them. This fosters trust and clarity for everyone involved.

Driving Continuous Improvement: QA isn’t just about checking boxes. I gather feedback from teams and clients to identify areas for future growth. Translating that into actionable recommendations leads to real, lasting improvements.

Escalating Issues: When critical concerns arise, I flag them with the right people. That might mean working with our teams directly or escalating to leadership. A key part of QA is creating accountability and ensuring swift resolution.

In practice, these responsibilities revolve around seven core categories of engagement health: Client Expectations, Scope/Product, Client Relationships and Collaboration, Communications, Resources, Financials, and Timeline.

The training I completed was a solid refresher on QA basics—nothing revolutionary, but a great reminder of the fundamentals. As a data and AI practitioner, though, I spotted some interesting opportunities to modernize and level-up the process:

Digital Feedback Mechanisms: Right now, our feedback-gathering is fairly manual. What if we introduced digital surveys or AI-powered assistants to document findings in real time? This could reduce overhead and create better proof points for our QA insights.

Predictive Analytics for QA: Once we have that data, imagine storing it and applying predictive models. We could forecast potential risks—timeline delays, scope creep, client dissatisfaction—before they happen.

Real-Time Dashboards: Combine QA data and model outputs into live dashboards that leadership can check any time. Seeing project health and financial performance at a glance makes it easier to intervene early.

Cross-Engagement Learning: A centralized repository of QA insights, especially for data projects, could be gold. Using models to cluster and highlight common issues might surface best practices for handling sticky problems. You could also leverage an LLM with a UI to ask interesting questions about the overall QA at the firm to identify patterns in the unstructured data.

Client-Focused Enhancements: If we integrated client data—like legal agreements, stakeholder backgrounds, account plans, or previous engagement history—QA reviews could become even more targeted. Each client has unique needs, and a data-driven approach would let us focus on the most critical areas for each engagement.

Let’s be honest: these ideas won’t all happen tomorrow. They’d require significant investment in time and budget—and let’s not forget data. Ideally, there is a platform or tool we could buy that does all of this (if you build this tool give me credit 😄). We’d need a clear cost-benefit analysis to justify any large-scale upgrades. Frankly, I’m not pulling those numbers right now—I’ve got enough on my plate! But the exercise of imagining new possibilities was super valuable. It also sparked some ideas I can bring to clients who face similar manual process challenges (though not necessarily QA-related).

Quality is everyone’s responsibility. (Thanks, W. Edwards Deming!) By weaving QA into every step of our delivery process, we set our projects—and our clients—up for success. As I dive deeper into this expanded QA role, I’m excited to bring fresh ideas into our data and AI projects. There’s an opportunity to double down on the role QA plays in how we deliver impact and ensure our clients walk away feeling they got more than they expected.

Now that I’m getting comfortable in my QA role, I’m eager to see how we can continue refining our approach. Whether we implement predictive analytics tomorrow or just continue nailing the fundamentals, every step forward in QA means more trust, better collaboration, and happier clients. And isn’t that the goal?

P.S.

The picture at the top of my post is the view from our apartment - pretty unreal huh?

Of course, the competition is catching up, and many are close to feature parity in some areas. Still, from my experience, DBx holds a strong edge for data and AI-centric workloads—at least for the moment.

Recently, SAP and Databricks announced a partnership to launch (GA TBD) what they’re calling “SAP Databricks”. This integrated solution will combine Databricks’ Data Intelligence Platform with SAP’s Business Data Cloud. The aim? To unify SAP and third-party data into an enterprise-ready data foundation for advanced analytics and AI use cases.

Naturally, I’m intrigued—and a bit cautious. While I'm not an SAP expert by any means, this collaboration could be a huge step forward for organizations that rely on SAP’s deep ERP capabilities but also want Databricks’ best-in-class analytics. However, as with any big tech partnership, the actual benefits will depend on how well it all comes together in the real world once it's generally available.

1. Ease of Data Sharing: Traditionally, SAP environments have been somewhat walled off, with data siloed inside ERP systems. This partnership should let companies merge their SAP data with other datasets. The result could be a more cohesive, analytics-ready data ecosystem—potentially cutting down on various, complex, spaghetti sets of ETL pipelines.

2. AI-Driven Launch Pad: By bringing Databricks’ data engineering and AI capabilities into SAP’s Business Data Cloud, it should become much easier to run advanced AI/ML models on SAP data. Think about predictive maintenance in manufacturing, real-time fraud detection in banking, or smarter demand forecasting in retail. This integration ideally would accelerate the AI-powered, domain-specific application/use case development that historically have been challenging to implement (for many reasons).

3. Governance at Scale: Databricks’ Unity Catalog enables governance and compliance and that will now be available (or should be) across both SAP and non-SAP data (with some work to set up the infrastructure). For highly regulated industries—like finance or healthcare—this unified approach is very important. It’s one of those “must-haves” in a world where data breaches and compliance violations are all too common.

Again, we will see how the actual rollout goes once this is GA available. Depending on the integration approach, some of these benefits might be at risk.

Despite the promising outlook, there are still plenty of challenges ahead. Here are a few big ones:

1. Integration Complexity: SAP systems are often deeply entangled with core business processes. Integrating them with the Databricks platform isn’t just a matter of flipping a switch. Companies will need careful planning, strong data governance, and well-trained teams. A poorly executed migration could lead to data inconsistencies—or worse, operational hiccups.

2. Legacy Systems & Cost: SAP workloads frequently live on-prem or in older cloud versions due to challenges with migrating to the latest/greatest version of SAP, which means pulling data into Databricks can add significant egress costs and latency issues or companies will need to undergo cloud migration efforts before they can achieve the benefits of DBx and SAP's partnership. Real-time analytics, in particular, could suffer if organizations don’t architect things properly. Financial and performance trade-offs must be thoroughly evaluated before diving in. Additionally, what will licensing look like for this new model and will companies sign up for those agreements. 

3. Skill Gaps and Change Management: SAP professionals often stick to the SAP ecosystem, while Databricks expertise typically resides with cloud data engineers and data scientists. Bridging this skills gap is no small feat. Without the right training and a shift in data culture, the best technology in the world won’t yield the desired business results. This could be a bonus however as DBx is well known, python is well known and it could open up the pool of developers who can work with these SAP-focused companies. 

A few thoughts for all those organizations that now have (or will soon) DBx at your fingertips.

1. Wait & See: Obviously, you'll need to work closely with SAP and Databricks on this—currently there is a waitlist and how all the details come together will be key. Stay updated on announcements from both companies and watch for insights from early adopters to gauge real-world impact and challenges.

2. Evaluate Your AI & Analytics Maturity: If you already have a solid AI/ML strategy, it might be worth exploring how to integrate SAP data into your Databricks environment and/or considering how DBx can uplevel what you currently have. At least start the discussion.

3. Assess Infrastructure & Costs: Make sure you explore how data processing will work with the partnership. Any move of SAP workloads to DBx needs to fit your overall cloud strategy—both technically and financially. Egress costs, inefficient workloads, and latency are real concerns.

4. Prioritize Governance & Security: Before diving in, confirm that your data governance frameworks can handle cross-platform integrations. Security shouldn’t be an afterthought.

5. Invest in Skill Development: Training and cross-pollinating teams (SAP folks learning Databricks, Databricks folks learning SAP) will make for a smoother transition and help unlock the full value of this partnership.

The SAP-Databricks partnership signals a shift toward more data sharing (fewer silos) and data science forward strategies (no surprise, we've been seeing this shift for years and it's only going to speed up with GenAI). It has the potential to accelerate AI adoption, streamline operations, and give companies a more holistic view of their data. But as always, the devil is in the details: strategic planning, disciplined execution, and a willingness to adapt will determine whether this partnership truly lives up to the hype—not just for these companies but also for SAP and DBx as it relates to the integration and rollout. As this starts rolling out to more and more customers, it will be interesting to see how it's received. I'm looking forward to seeing how it goes over the next several months.

I want to give a shout to Taylor who helped me prep this Blog and provide some great technical and flow recommendations as well - thanks Taylor!

P.S.

Happy Valentine's Day all!

P.P.S

The picture at the top is of the sunrise this morning. This picture is a fun sign I've seen and admired in my neighborhood for a while - Pepsi, candy, and cigars...what a way to live!

We were discussing the challenges many companies face when implementing LLMs. His perspective is that many organizations are overly focused on LLM reasoning capabilities. As a result, many companies and technology leaders are waiting for the models to reach a near-perfect level of capability before putting their hats in the ring and joining the GenAI wave. He also pointed out the limitations LLMs have been running into.

He argued that LLMs are already powerful (I agree), but their effectiveness is largely determined by the context provided (I also agree - think RAG). Innovative companies, he noted, are investing in collecting and organizing their structured and unstructured data while orchestrating LLMs effectively (again...I agree). That’s why he sees data preparation and structuring tools as particularly valuable right now. (Again, I agree.)

From my experience, 60% of the work that goes into data science solutions is data preparation and exploratory data analysis (EDA). Leveraging data preparation tools backed by GenAI to accelerate that process is a major opportunity right now.

My issue with his point of view is that while some companies - or individuals within them - might be waiting for LLMs to improve, I don’t think most companies are.

During my time supporting GenAI strategy and governance efforts, I've seen three primary challenges that companies face when executing GenAI strategies and building GenAI tools:

1. Managing what’s been built – Once an LLM solution or product is developed, how do we operationalize it? What’s the “Ops” in DevOps, and does my organization have the right teams and resources to support and scale it effectively?

2. Understanding the GenAI black box – Many companies struggle with visibility into how LLMs generate their outputs. This is especially critical for regulated industries like healthcare, utilities, and financial services, where understanding what data the model is using and ensuring accuracy are non-negotiable. We all remember the lawyer who unknowingly cited false legal precedents because an LLM hallucinated.

3. Ensuring adoption – Beyond the technology itself, organizations struggle with governance and managing the change required for business users to adopt and sustain GenAI solutions successfully. Without these foundational elements, even the best models risk failing to deliver meaningful impact - or worse, being misused to the detriment of the company.

LLMs are powerful tools, and many companies recognize that. The primary challenge isn't whether LLMs are ready - it’s whether organizations are ready. Success in GenAI requires thoughtful execution, operationalization, and governance. While some technical leads and enterprise architects may be waiting for LLMs to advance, most businesses are grappling with how to manage, interpret, and adopt these new tools effectively.

After the festivities, I shifted gears to research. I’m currently collaborating with a fantastic team focused on customer experience technology. We're working with a client considering a new IVR platform and looking to move to the cloud. During our last session, the executive team asked how large CCaaS platforms interact with other technology providers, especially those with strong AI capabilities - think Google, Azure, etc. - like natural language understanding (NLU) and AI agent assistance.

The team did a great job addressing the question on the spot. Still, I wanted to understand how these integrations work and develop a visual example that an executive could easily digest. Here’s a breakdown of what I learned about how modern CCaaS solutions, like Genesys, integrate with third-party AI platforms (such as Google CCAI) to create a next-level contact center experience.

Modern CCaaS platforms have many built-in capabilities and allow seamless integration with external tools. For example, Genesys can connect directly with Google CCAI to leverage advanced natural language understanding, self-service bots, and real-time agent assistance - all through secure APIs, webhooks, and streaming interfaces. This integration combines the best of both worlds: Genesys’s core functionalities (such as IVR, routing, and analytics) and cutting-edge AI technologies from providers like Google.

I'm a visual learner, so I created this logical architecture based on a few online articles (here is a good one). The Genesys logical architecture can be broken down into several key components:

1. Customer Endpoints: Phones, apps, and web interfaces that customers and agents use to interact.

2. Telephony/Carrier Services: The connection to the phone network, handling emergency services, call routing, and more.

3. Genesys Cloud Platform: The heart of the system, offering core services like IVR/flow logic, predictive AI, and analytics.

4. Public Interface: The “gateway” that enables integration with external systems like Google CCAI.

Genesys already offers strong and continually evolving analytics and AI capabilities. By integrating with third-party solutions like Google CCAI, organizations can further extend these capabilities, unlocking advanced features such as:

• Natural Language Understanding: Leveraging Google Dialogflow for improved intent recognition, making self-service bots more conversational and effective.

• Real-Time Agent Assist: Streaming conversation data to Google’s AI, which then provides agents with real-time guidance and suggested responses.

• Predictive Routing: Combining Genesys’s native AI with Google CCAI to intelligently match customers with the best-suited agents based on historical data and real-time insights.

This integration doesn’t replace Genesys’s core functionality; instead, it augments it—allowing organizations to mix and match services based on evolving needs.

Here’s a logical overview of how the integration between the platform and a third party, like Google CCAI, might work:

1. Public Interfaces (APIs, Webhooks, Streaming): Genesys's gateway for communicating with external systems like Google CCAI, enabling real-time integrations.

2. CX Application Services: Manages customer interaction flows and connects to Google CCAI for tasks like voicebots and chatbots.

3. AI Services: Integrates Genesys's AI capabilities with Google CCAI for enhanced AI-driven features like NLU and agent assistance.

4. Real-Time Streaming for Agent Assist: Streams live interaction data to Google CCAI for real-time agent support and suggestions.

For organizations like my clients, understanding the evolution of a CCaaS platform is crucial. The best platforms are not static; they’re flexible, modular, and purpose-built to adapt and scale with your needs. They deliver top-tier contact center functionality and seamlessly integrate with cutting-edge AI technologies, empowering businesses to stay ahead as markets evolve and customer demands change. Whether your goal is to deploy smarter bots, enhance agent performance, or harness predictive analytics, these modern platforms are designed to provide the capabilities that drive growth and innovation.

As I delved deeper into these modern platforms and their integrations, one thing became clear: AI is not just an add-on - it’s becoming the backbone of intelligent customer experience. Today’s CCaaS solutions don’t just support AI; they are actively evolving alongside it, leveraging generative AI (GenAI), real-time analytics, and seamless integrations with major AI providers to create adaptive and predictive customer journeys.

By combining AI-driven insights with human expertise, businesses can create dynamic, responsive environments that not only enhance efficiency but also transform customer interactions into meaningful engagements. As AI continues to redefine the landscape, these platforms will play a crucial role in bridging the gap between human and machine intelligence, pushing the boundaries of what’s possible in customer experience.

This deep dive into CCaaS and AI solution providers was both illuminating and thought-provoking - I'm excited to see how these technologies continue to evolve and the conversations they will spark this week.

Now...time for more dim sum

P.S.

I know the photo at the top isn’t dim sum, but unfortunately, none of my pictures turned out well. Instead, I’m sharing a shot from one of my best sushi experiences—Sushi Itchimura, a year ago. My buddy took me, and it was incredible.

Hopper is positioned as an GenAI-powered analyst and data engineer that can tackle tasks like data wrangling, documentation review, and other repetitive yet essential engineering chores required during data migrations. I envision it as a sort of Copilot for data engineers, particularly focused on data migrations and exploration. It’s already being used in live client engagements, supporting delivery teams by automating routine work and freeing up engineers to focus on higher-value problem-solving.

Conceptually, this is exactly the kind of GenAI application that makes sense - within the right bounds. Every data engineer I know who’s supported a platform migration has spent countless hours:

• Mapping schemas between source systems (like Oracle, SQL Server, or IBM's IMS) and target systems (like Snowflake, Databricks, or Amazon Redshift).

• Running into and identifying data quality gaps and inconsistent formats.

• Migrating code from one platform to another (e.g., from on-premise ETL tools like Informatica to cloud-based services like AWS Glue or Azure Data Factory).

• Wrestling with “out-of-the-box” migration tools that promise automation but often require just as much manual tweaking.

If Hopper delivers on its promise, it could fundamentally change how consulting projects operate. By reducing the time and resources required to handle intricate data migrations and platform upgrades, it has the potential to shift more focus toward innovative data architectures, advanced analytics, and strategic advisory work.

I have a running joke at work: I tally how many times the term “GenAI” appears in any executive readout, pitch meeting, or strategic update. It’s an incredibly powerful tool, but it’s not (and should not be) a one-size-fits-all solution. Clients and colleagues sometimes fall into the trap of making everything a “GenAI nail” just because they have a shiny new “GenAI hammer.”

So, is data migration the right “nail”? Like most things, it depends on the context, the complexity of the data environment, and the maturity of the data practices in question.

Data migrations are both tedious and nuanced. However, organizations often have a large corpus of documentation, schemas, and code related to the source and target systems. This is where GenAI-driven tools, like Hopper, can shine:

1. Schema Mapping & Transformation

   • GenAI can automate large portions of schema mapping and even suggest optimal transformation rules. 

   • Imagine feeding Hopper your existing table definitions from Oracle and letting it generate an equivalent schema for Snowflake, complete with recommended data types or partitioning strategies.

2. Automated Data Quality Checks

   • GenAI can flag inconsistencies, missing values, and formatting errors - or at least generate hypotheses about what might be wrong. 

   • Frameworks like Great Expectations and tools such as dbt’s testing features offer robust support for data quality; GenAI could augment these by automatically generating validation rules.

3. Intelligent Data Cleansing

   • GenAI can deduplicate, standardize, and classify unstructured data, enabling smoother migrations into a new platform. For example, if you’re moving data from on-prem HDFS clusters to Databricks on Azure, Hopper could help classify files by content type and usage patterns.

4. Code Generation & ETL Optimization

   • GenAI can assist with auto-generating scripts for ETL pipelines and migrating legacy code (like COBOL or PL/SQL) to modern cloud-based workloads. 

   • It might also optimize Spark or PySpark jobs, or even rewrite your transformations to leverage Apache Airflow or AWS Step Functions orchestrations.

On the flip side, GenAI tools are only as reliable as the data and documentation they’re trained on or given as context. If information about a source or target system is incomplete - or worse, nonexistent - GenAI may produce subpar or even incorrect outputs. Beyond these context-related issues, there are a few broader GenAI risks worth noting:

1. Lack of Determinism

   • GenAI outputs can be inconsistent and are often difficult to audit. Deterministic jobs in data engineering exist for a reason; you need to trust that a job doing record-level transformations will behave predictably. A lot of validation and testing upfront is required. 

2. Compliance & Governance Issues

   • GenAI must adhere to strict data privacy laws (e.g., GDPR, CCPA) and governance frameworks (e.g., SOC 2, ISO 27001). The last thing you want is for a GenAI-generated script to accidentally expose PII.

3. Error Handling & Edge Cases

   • GenAI may struggle with arcane legacy systems and custom business logic. If your organization uses proprietary transformations or specialized data formats, GenAI might not have the context to accurately migrate them.

4. Performance & Scalability Concerns

   • Large-scale data migrations often require high-throughput, high-efficiency workloads. GenAI models may not always optimize for performance out of the box, potentially increasing runtime or cloud costs if not carefully supervised.

GenAI shouldn’t be an unchecked data migration engine, but it can be a valuable assistant for:

• Generating ETL pipelines rapidly.

• Aiding with data mapping and transformation.

• Automating validation and quality checks.

• Assisting with the classification of unstructured data.

Like many of us have experienced, GenAI is most helpful when it enhances human decision-making rather than trying to replace deterministic, rule-based processes. Think of Hopper (and similar tools) as a junior engineer that’s lightning-fast at repetitive tasks but still needs oversight from an experienced team.

I’m genuinely excited to see how Hopper evolves. As it matures, I could see it growing into a robust platform that does much more than data migrations, it could be a genuine game-changer for our firm and the wider industry. Consulting has long relied on manual “grunt work,” and GenAI tools that truly reduce that burden enable talented teams to shift focus toward more strategic, high-impact problem-solving to support our clients. 

I’m excited to have my Hopper deep dive this week. I’ll be following Hopper’s progress closely and look forward to using it to augment and enhance my teams’ capabilities, ultimately supporting clients more efficiently. 

What do you think? Can GenAI tools like Hopper truly revolutionize data engineering or are they just another overhyped solution in an already crowded automation toolkit? Let me know your thoughts. 

This will be a space where I write about data, AI, and analytics - sometimes weaving in cooking (because why not?) and sometimes just thinking out loud about work, leadership, and technology. There’s no strict agenda; I just want to explore ideas, share insights, and see where it leads.

I’m also curious about how Web3 is reshaping content creation and ownership. I've been experimenting with blockchain since 2013, this blog could be an interesting experiment in blending data, AI, and Web3 monetization models - exploring how decentralized platforms, tokenized content, and digital ownership change the way we create and engage with ideas.

A bit about me - by day, I help companies make sense of their data, build strategies, and turn insights into action. But beyond that, I’m also a husband, a dog dad, a Brooklyn resident, and a foodie. This blog is a place where all of those parts of me might come together.

Mostly, I’m writing for myself. But if you find something here that sparks an idea, helps solve a problem, or shifts your perspective, even better.

Let’s see where this journey takes us.