The Savvy CIO: Episode 3 – “Why do AI Buildouts Fail?”

Skylar Roebuck: I think in previous technology waves, there’s a fixation on the what, and that was kind of okay. You could kind of have that transformative idea that changed an entire company, but now the how is embedded in the what. You’re really starting to think about changing the fabric of how the entire company operates. Agent is like the new fabric across your whole company. How does it interact across all of these different business units with each other to deliver all of this new functionality? And once people start seeing its capabilities, you start to get a lot of aha moments.

Bradd Busick: You’re listening to the Savvy CIO, Modernized Wisely by Park Place Technologies. I’m your host, Bradd Busick. This is a show where we talk about budget pressure, AI adoption, security, and the art of keeping the lights on without burning everything down with real people who are tackling these problems every single day.

This episode, we’re saying the quiet part out loud, everybody, and I mean everybody is so excited about AI build- outs. You literally can’t go on LinkedIn or X without hearing about something cool that someone’s doing, but hardly anybody is talking about the fact that most of these things are failing. We’re not talking about pilot projects here. We’re talking about proof of concepts that almost always work in the controlled demo, which is obviously what it’s designed to do, but something happens between the phase of, hey, this looks really promising, and dang, we’re now live and running at scale where things start to go wrong.

Before you know it, the project’s over budget, you’re behind schedule, the CFO is blowing you up, and it probably already has one foot in the grave. So it either wimpers out to a quiet death or it’s rapidly clawed back into something that’s way smaller and far more basic than the big swing innovation that you would actually plan for.

Skylar Roebuck knows this struggle better than most. As a CTO of Solvd, his entire job is helping their enterprise clients build production grade global platforms, what they call the last mile of engineering. They do the hard part of making the AI ambition meet infrastructure reality. Skylar, welcome to the Savvy CIO.

Skylar Roebuck: Happy to be here. Thanks so much for having me.

Bradd Busick: You bet. I’m really excited to talk with you about this personally and professionally. Probably like most, I’ve got my own kind of side projects happening on Saturdays and Sundays where I’m playing around with models and then of course professionally doing this as well. But before we get elbows deep into the main issue of why AI build- outs fail, I’d love to get to know you a little bit better. You have such an awesome background. How did you get to this point where you’re actually standing on the leading edge for AI engineering?

Skylar Roebuck: I think I have a somewhat maybe unconventional background. I started my career in computer engineering and I ended up kind of moving into more machine learning and machine vision towards my master degree, but caught the bug for entrepreneurship pretty early. And that is something that has been really woven in deep to everything I do from day one, the last 15, 20 years, however long it’s been. But I’ve kind of used the same pattern for how I’ve actually approached helping companies because it was about 15 years ago I helped start a company called Mobiquity. And that’s where I really galvanized my approach around building services companies around emerging technology trends. Because when you’re building a services company, you’re always kind of faced with a, why now? Why should I care about this thing now? And I find a lot of comparisons even from then to now because these new technologies come out and companies are challenged with, what should I do with this?

At the time we did actually Panera’s first iPad point of sale system, which was well before anybody probably should have done an iPad POS system, but it changed their business pretty substantially at the time. And it was because they had the conviction really to transform their organization using this technology. It was a really big deal. And from there we went to full medicine adherence systems and all sorts of things with mobile.

And so from where I’ve kind of fit in into this is I’ve been starting these companies or participating in these companies that have been really around these emerging technology trends and trying to figure out how do I bridge the gap between this technology has come out? How do I make it successful within large enterprise organizations? Right?

Bradd Busick: Yeah. That’s awesome.

Skylar Roebuck: And that’s really where I’ve focused my attention, either mobile, the next was IoT, and now we’re really in AI at this point.

Bradd Busick: Well, it feels like you’re placing chips on the table and making bets at really interesting times. It feels like the parallel of you being an entrepreneur is allowing you to see around corners and maybe frankly realize the value in things that other people can’t quite get their arms around yet. Is that a fair assessment?

Skylar Roebuck: I think so. I think part of maybe what makes me a bit different is having this kind of entrepreneurial spirit within an enterprise organization is actually a somewhat unique thing in terms of how to adapt this technology to very meaningful use cases. But part of, again, what I think makes a lot of AI initiatives succeed or fail is really around how to navigate the organization itself, how to allow and give room to these ideas to be able to grow and create the right momentum and be able to navigate the organization in the appropriate way so that it has a chance to actually breathe. And one of the terms I like using is organizational antibodies. When you introduce these things, have such big possibilities of change and transformation, you always will get all these people saying why it’s impossible, why it’s never going to work, why legal’s never going to improve it. The number of reasons to shoot down an idea is endless.

Bradd Busick: Yeah, that’s right.

Skylar Roebuck: And so you have to really be headstrong. And then I think, what is it? Amazon that said the stubborn digital vision?

Bradd Busick: That’s right.

Skylar Roebuck: It has to be the thing that maintains your momentum into these ideas.

Bradd Busick: Now I consider myself a student in organizational change management, and yet it’s only recently that we start talking about the importance of OCM as a practice. And I’m thoughtful follow the market. You’ll hear things show up in academia and you’ll hear authors like Cotter and others come forward and say why we need to have change. But then you’ll see some wild thought leaders like Jeff Hiatt and creating Prosci with a framework like ADKAR that says, this is how you actually measure change. And then you’ll see the big five consulting firms create practices and charge everybody millions of dollars about how they implement change. Give me a double click on with that lens, why are you seeing AI build- outs fail and how often are you seeing these things fail?

Skylar Roebuck: Boy. Well, that’s maybe addressing a couple of those things separately. So when I start thinking about change management, certainly the large consulting companies have made that a big term. Right?

Bradd Busick: Yeah.

Skylar Roebuck: It’s this big thing that the big guys will help manage for a company, and that is its own art form for sure. One thing that’s interesting is in previous technology waves, you could maybe separate out the transformative ideas from organizational change, but this is one where you really can’t anymore. The fundamental nature has changed in terms of what you’re building. And that forces everything you do to have an organizational change component to it. And so if you’re starting to talk about one of the biggest use cases that we focus on a lot, and honestly a lot of people do, is just customer service. What is the future of customer service? And the reason why is it’s the nature of creating that hyper- personalized care is impossible with humans just alone. There’s no business model in the world that will allow that level of personalization with just people that’s economical.

Bradd Busick: Agreed.

Skylar Roebuck: But with AI agents, it’s possible. But the challenge when you start rolling out these new systems where you’re having agents that can do multi- turn engagements with customers to not just be a chatbot that says, ” We understand your problem,” but in fact, to resolve that issue, you start to get a lot of consternation from the organization itself. We have thousands of customer support representatives. What’s the plan for them? How are we going to upskill them? How they’re going to participate in this future? And so it’s hard even to think, I mean, certainly there are, I think what I would say is AI engineering initiatives that maybe can be just software solutions. But I think the day of thinking of these build- outs as just traditional software, it’s kind of over. I mean, you have to think of this as something very different.

Bradd Busick: Yeah, I agree. I mean, with your lens, specifically in the customers you serve, I mean, quantify this for me. What percentage are you seeing AI efforts going down range and bomb? I mean, are we over 50%? Are we above 80% or is it 10%?

Skylar Roebuck: So MIT did a study and I went to AWS re: Invent and every vendor in the world, it was crazy. It was using the same numbers. 95% of AI initiatives fail.

Bradd Busick: I saw that.

Skylar Roebuck: Everybody’s done it. You look at the study and you’re like, okay, well, there’s some flaws in maybe how they run the study.

Bradd Busick: It was flawed.

Skylar Roebuck: Even I would say, us, we ran our own survey. We were like, okay, let’s put some meat to this as well. And we were still close to 80% in self- reported failures as well.

Bradd Busick: Wow.

Skylar Roebuck: But the challenge is that AI as a tool in technology is not just what you were building, but how you were building it. If someone were to say, what is the number one killer use case of AI? It would be software development right now.

Bradd Busick: Sure.

Skylar Roebuck: That is ultimately the strongest thing, the biggest thing that is changing. And so the nature of that means that we are getting more and more and more POCs. And so traditionally I would call this random acts of AI, things that are popping up, one- offs here and there. And that means that you get a lot of failures. The volume of failures from that kind of motion is extremely high because you’re trying a whole bunch of stuff.

Bradd Busick: Sure.

Skylar Roebuck: But our goal when we work with our clients is because we kind of focus on the full life cycle of trying to bring these things to market for large enterprise companies, we have to challenge our clients pretty head on.

And we have to be sold personally on the value of one of these ideas because we know the cost at scale.

Bradd Busick: That’s

Skylar Roebuck: Right.

Bradd Busick: Yeah.

Skylar Roebuck: And ultimately, if we’re not sold, then. And I think there is a problem with people not thinking big enough, to be honest, when it comes to these things. If it’s small and we don’t see a path to something big, we know it’s going to be like flash in the pan. It’s going to be a short thing that doesn’t necessarily move the needle.

Bradd Busick: So you hit on, I mean, you use one of my favorite words, the V word, value, value creation. As you look ahead at all of the pilots that have kicked off, the pile of pilots that are in the graveyard, how do you actually evaluate what is ” working” and what is actually creating true value and success versus what is what we call an IT CBNU? Cool, but not useful. How do you delineate the thinking between those two things?

Skylar Roebuck: I think sometimes the infatuation of new technologies starts to distance people from the reality of the business. So people lose sight that the initiatives that you are implementing ultimately have to drive the value for the company that the company sees and respects ultimately. And this is different for every company and companies exist at different levels of maturity for these things. For example, a company that has been maturing their customer service for a really long time knows the correlation between, let’s say, NPS scores from customers, their general happiness with this, and they can correlate that to retention and correlate that to value. But other companies who aren’t as refined and maybe they don’t see customer service as a value creating function for their business, they see it as just a cost center. Like, oh, we just got to have this thing. They might have outsourced it in fact, and they don’t care if it’s that laborious for customers to go through. And it depends. Maybe that’s the right decision, maybe that’s the wrong decision. But in their eyes, they might see the win as just cost- cutting.

And I think generally speaking, the thing philosophically I’m always trying to challenge clients to do is think about the value beyond cost- cutting because that’s just the easiest and sometimes the laziest way of thinking of the value proposition for these initiatives. But the bigger game- changing ideas are somewhat transformative, and those ideas are sometimes difficult to put a pure measure to that’s immediate.

I mean, I have some examples of how I think about these things. There definitely hasn’t been a silver bullet metric that I would say has driven value across the board. And I often feel like anytime someone uses ROI, it feels small for some reason.

Bradd Busick: Right. So you had mentioned this idea of cost, and I would love to get your lens on this. I mean, we’re all being sold as IT leaders that AI is just getting cheaper and faster. What’s the actual cost reality that you see when you’re running at scale with your current customers and people you’ve worked with?

Skylar Roebuck: It’s interesting because people might fixate on the cost of the models and running those models, but we have to think about also the volume of additional information that you’re leveraging to actually create the outcomes you’re expecting. So let’s take, for example, multimodal search. The idea of going to a website, let’s say it’s a marketplace of some kind. They have 100, 000 dresses. You can’t just search dress. So the idea of what we can do now, multimodal, introducing voice, introducing image videos as a way of trying to understand what a user is looking for. We are now introducing huge amounts of new data into this ecosystem that we are now processing, which should not be ignored. It’s adding tons of cost that you have to basically plan on when you’re building out these solutions. And again, this should be built into the business model from the very beginning is understanding the full cost of ownership. So you have that side of it. You also have how these things are being built.

And so another good example here is, let’s say we have, for simplicity, some kind of virtual try- on or you’d have clothes and you want it to see on an avatar or something like that. There’s a balance of quality management because we’re starting to deal with non- deterministic outcomes and cost. And so when you start thinking about how would you guarantee that a user does not see an image that is not worthy of the brand?

Well, what you do is you engineer it to generate more images, and then you build things that go through these images and just eliminate the ones that don’t fit. And so if your threshold as a business is like, we will never have this image, we’ll never have a problem, then you might be generating 10 times more images for that guarantee, which increases cost by X amount. So how you engineer these things become a major component as well. I think the areas where we’re starting to get much more savvy is things like memory management and context management and things like that. Lots of companies coming up, lots of open source projects happening as a result of helping companies be able to manage this. One of the things I was thinking even that whole context layer for businesses as well is another area where people are investing in quite a bit. And there’s a big difference between people who are doing it well or not well, or I think the worst is waiting.

Bradd Busick: Well, and that’s actually my question. I mean, we have so many listeners across the globe, Skylar, that are in one of two boats. For a CIO who’s been told to hold off, or they have a CEO that thinks that they’re the chief strategy officer or a president that all AI decisions have to come through me. And so they’re basically on the starting block waiting for the gun to go off. What risk are they facing if they’re too slow and they can’t get out of the gate?

Skylar Roebuck: I think existential.

Bradd Busick: Agreed.

Skylar Roebuck: We are in a world where this is, you can say what you want around maybe the economics of training these models, building these massive billion and billion dollar data centers, and potentially that has a challenge to it. But the models themselves, the technology itself, I mean, it is unmistakably powerful in terms of what it can accomplish. Your ways of working are changing. The SEO that was super effective in the past, nope, no more. People are using these new tools for navigating these things. It’s crazy. And so it’s funny because even just in the last week, I was talking to one client who we were going through a lot of possibilities of art of the possible. And we got, well, we’re about to start this PIM project. It’s such a traditional thing. We’re going to introduce this PIM and it’s going to take a year. So why don’t we start thinking about use cases in a year from now when our data is cleaner?

The ship has sailed at that point.

Bradd Busick: It’s out the gate. That’s exactly right.

Skylar Roebuck: A year from right now, I mean, can you imagine the amount? Claude Code-

Bradd Busick: Insane.

Skylar Roebuck: … really kind of fully debuted or it really came into its own November of 2025.

Bradd Busick: Yeah, that’s exactly right. It’s a quantum leaf. It’s quantum. And it’s funny. I love that you’re pushing customers to think in no longer years, but maybe months or weeks. And what’s funny about this is we’ve talked about the CIO that’s going to be slow out of the gate, that’s waiting for a strategy committee to bless something. Those days are gone. And I actually think we’re seeing the CIO move the other direction where he or she is the thought leader in that organization. And to your point, needs to have the intersection of both what and how to move people together towards the goal. So let me ask you that same question with a different lens. We’ve got the laggard. What about the CIO that has been out of the gate, that is literally on the bleeding edge? What risks are they facing? Or what should they be thinking about if they’re three miles ahead of the business already?

Skylar Roebuck: I think the risks, I mean they’re similar risks to us because as a company, what we’re doing is we’re bringing frontier researchers, very, very smart engineers to the table, and we’re trying to identify these really high value use cases. And when we find things that we think apply to an entire segment industry across industries, we want to repeat it, try to get better at it. Challenge with this industry changes so fast.

Bradd Busick: Literally.

Skylar Roebuck: So the things that we did that were groundbreaking maybe six months ago, some of these things like model improvement in that amount of time alone, we’re like, oh, we’ll just use that. You don’t need to maintain and orchestrate eight unique models to do this. There’s another model that’s done that. But the thing is the act of going through this process and starting to build the muscle around these things is a thing that keeps you ahead.

Bradd Busick: Yeah, that’s exactly right.

Skylar Roebuck: And I like how you put it, which was the what and the how. I think in previous technology waves, there’s a fixation on the what, and that was kind of like, okay, you could have that transformative idea that changed an entire company, but now the how is embedded in the what. You’re really starting to think about changing the fabric of how the entire company operates. Agent is the new fabric across your whole company.

Bradd Busick: It is.

Skylar Roebuck: How does it interact across all of these different business units with each other to deliver all this new functionality? And I mean, what we’ve also seen is just by going through the motions of doing it does create its own momentum. People are saying, okay, well, we’re going to build a service agent that’s invoicing that’s multi- turn. And once people start seeing its capabilities, you start to get a lot of aha moments for sure.

And maybe that wasn’t a groundbreaking thing, but it starts to build up.

Bradd Busick: The flywheel. Yeah.

Skylar Roebuck: Yeah.

Bradd Busick: The flywheel.

Skylar Roebuck: It is definitely the flywheel. But along the way, certainly the risk of trying new things and being ahead is eventually you might have to pivot to something. Someone does it better, you embed that, and you just have to continue to stay agile on these things. But I’m definitely in the camp of start moving today. Yeah, start moving now.

Bradd Busick: Let’s land the plane here. The problem with being a realist, not a pessimist mind you, but a realist is that everything we’ve talked about sounds like, man, it’s bad news. And yet there’s little glimmers of hope and light in some of the things that you’ve learned and I’ve learned and our listeners have learned. If there’s a CIO today that’s hearing what we’re talking about, they’re in a tough spot. They’ve got legacy tech debt, they’re out of budget already, they burned through their tokens, or they have a leadership team that is waxing about an AI strategy except they can’t spell AI. What’s the one piece of advice that you’d give them?

Skylar Roebuck: Oh my gosh, all of those things you just mentioned, I’d probably give individual advice to each one of those. But I think of these investments as a kind of diversified portfolio of investments. You end up with your quick wins that create this momentum. You have these steady bets that are pretty solid for the company, and then you have these game- changing ideas. And a lot of times people just fixate on the quick wins. They don’t go anywhere. So you really have to challenge yourself to identify, I would say, those trends of this will never happen. People have to try and close in a brick and mortar store to buy them. These kind of trends that you think won’t break because they will break. And those are opportunities I think to really win. So coming up with a well- rounded investment strategy around AI is really, really solid.

Bradd Busick: It’s good.

Skylar Roebuck: I think a lot of the challenges around legacy modernization, honestly, I think AI is the answer for a lot of those problems. Legacy modernization is actually something that we’re seeing pop up as a very mature business in terms of helping companies move off of these large systems. But now you have agentic software development that’s improving speed and reducing cost, and it’s making things possible that weren’t possible before. So for the first time we’re hearing companies saying like, ” Oh, we have a 15- year Ruby on Rails system. Maybe now is the time we can move off of it.” And it’s actually cost- effective now. It’s no longer a complete. We thought we were going to die with this thing, like the banking industry with their underlying core platforms.

Bradd Busick: And you can swap out banking for government or for academia. I mean, it’s all of those. And I think if anything from today, our listeners are walking out with this notion and maybe a call to action of the time is now, it’s the worst it’s ever going to be.

Skylar Roebuck: It is.

Bradd Busick: And that’s an exciting proposition.

Skylar Roebuck: I would say that’s exactly correct. Get started today and really think about the how as you start to deliver on these bigger ideas. Start to build out the infrastructure, start to build out the context layers, start to build things where you can kind of build on these foundational layers rather than these one- off solutions.

Bradd Busick: I’d say this here in closing, best of luck to you. I love hearing what you’re doing-

Skylar Roebuck: Thank you.

Bradd Busick: … and the change that you’re making. It’s been a pleasure to have you with us today, Skylar.

Skylar Roebuck: Absolutely. Thank you for having me.

Bradd Busick: Now let’s review the takeaways from my awesome conversation with Skylar today. First, it’s really around the core thesis. Most AI build- outs aren’t failing because the technology doesn’t work. As a matter of fact, it absolutely works. They’re failing because organizations are treating AI as a technology problem when actually it’s an organizational readiness problem. The equation of people, process, and technology, they’re all wildly important, but man, people can kill a project faster than most other things. The second, the failure pattern is really consistent. Companies are skipping the boring infrastructure work that it takes, and we know that proof of concept can sometimes get swapped out proof of value, and they’re two very, very different things. And then finally, today’s savvy CIOs, they’re caught in this two- sided trap. If you move too slow and you’re falling behind on capabilities and talent, you’re going to get burnt. And if you’re moving too fast without the foundation that you’re burning budget on, you’re also trapped. Neither is comfortable. Embracing this notion of getting people on board so that the technology and process can adapt is the difference between good and great.

That’s all for today. Thanks so much for listening in everyone. Please follow us so you don’t miss an episode. This has been the Savvy CIO brought to you by Park Place Technologies. If you want to learn more about Park Place, go to www. parkplacetechnologies. com.

Skylar, one more question before we let you go. Because this is the savvy CIO, I’ve got to ask you, what is the savviest CIO choice you’ve made in your career to date?

Skylar Roebuck: I think it is to transition the idea of operational improvement, company improvement to the team rather than keeping it in kind of a silo at the top. It’s too slow to have it siloed and you have to leverage your team to be able to keep up these days.

Bradd Busick: Love it. I hope you hear that clapping sound because that’s our entire audience clapping in their cars and on their runs and on their walks today. That’s solid wisdom.

I’m your host, Bradd Busick. And as always, IT shouldn’t just be at the table, IT is the table.

Skylar Roebuck is the Chief Technology Officer of Solvd, where he drives AI-powered engineering and digital transformation for global enterprises. A Carnegie Mellon–trained engineer and serial entrepreneur, he brings over 15 years of experience helping Fortune 500 organizations translate emerging technologies into measurable business outcomes. Skylar has driven billions in digital revenue across healthcare, retail, and financial services, and is known for advancing agentic engineering and applied AI at scale. His work focuses on enabling CIOs & CTOs to accelerate innovation, and build resilient, future-ready technology organizations.