The Savvy CIO: Episode 4 – “The Hidden Cost of AI Readiness”

Roy Illsley: The biggest thing with AI that people just really do not understand is A, where is it in the organization? If we don’t monitor and measure it, we can’t then understand the costs and then we can’t work out what the value is. You might start AI and find this brilliant use case which costs X, but as soon as you scale it, the scaling costs may actually outweigh the benefits that you’re going to get in. We haven’t done their maths. We don’t know how it works.

Bradd Busick: You’re listening to the Savvy CIO, Modernized Wisely by Park Place Technologies. I’m your host, Bradd Busick. And this is a show where we talk about budget pressure, AI adoption, security, and the art of keeping the lights on without burning everything down with the people that are tackling these problems every single day. Right now, companies are pouring millions of dollars into AI initiatives. They stand up pilots. We hire consultants, we purchase platforms, and then months go by and we’re all left scratching our heads when it feels like nothing has actually changed. The latest data is telling us that the hidden costs of AI isn’t actually infrastructure or data cleanup. It’s not even the models. It’s the investment into something that doesn’t get utilized properly or sometimes gets creatively overutilized in ways that the organization didn’t anticipate.

Roy Illsley has had a front row seat seeing this pattern unfold across industries and budgets both big and small. He has over 40 years of experience in the IT sector, working for a variety of consultancy and end user companies with experience in defense, utilities, automotive, retail, and fast- moving consumer goods industries. I’m speaking with Roy today about what separates the companies getting real value from their AI platforms versus the ones that get stuck in sunk cost purgatory and how you can avoid that quagmire yourself. Roy, welcome to the Savvy CIO.

Roy Illsley: All right, thank you very much, Bradd, and thanks for having me.

Bradd Busick: Hey, your background is so awesome. To kick things off, could you share a little bit about yourself, why you’re exactly the right person to ask about the hidden costs of AI readiness?

Roy Illsley: Well, one of the key things as an analyst that we are very fortunate and we get is we have access to all the key leaders of the technology industries. We also in Omdia have access to vast amounts of data. We have 75, 000 end user budget information. We have thousands and thousands of permissioned end users that we gain and seek views and opinions from. So as an analyst, you sit in a very privileged position. And as a previous practitioner doing the day job that the listeners are doing, it does sometimes feel like I don’t have to put the fires out. What I’ve got to do is help you avoid those fires because when I was doing that job, I knew what it felt like. And today I got access to all this information. I can focus on that information. I can hone it down and give the advice to people that are using it rather than the end user having to sort all this out themselves.

Bradd Busick: Every time all of us as CIOs or IT leaders get asked to go submit our roadmaps, our budgets, how much do you spend on a given platform? Roy gets to see all of that across all industries. I’m assuming with all that info, you start to see patterns develop. What’s the thing that you notice in the first conversation with a new client that says, ” Hey, this is where we’re at. This is our current level of maturation. This is our sponsorship that we have from leadership.” What are some of those things that distill for you that allow you to quickly assess where a company might be at on their journey?

Roy Illsley: Well, a lot of it is actually down to their maturity in how they react and respond to the situations. If you take AI for example now, AI is a big hype. You go back 15 years, we have the same sorts of thing with cloud. So what you look for when you’re talking to customers is those customers that have been on this journey and go, right, I know all this is good, but I know last time we went through this, we got to this point and suddenly discovered that we actually needed to do this. So those people are the ones that you go, ” Yes, that’s what you need to do this time.” Or, ” No, this technology negates that. You’ve got to think of this.”

They will get that and they will go away and work on their mitigations, their responses to it. Those that aren’t so advanced in that thinking or don’t have that history tend to rely on you to give them the guidance and ask them the questions so they can go back and sort of come back to you with, ” Now we understand what you’re talking about. Can we ask you X, Y, and Z?” So there is a wide remit of maturity, but it also goes into, there’s a wide remit of industries and the geographies of how they use technology, where they use technology.

Bradd Busick: I want to maybe pivot to this notion or phrase that I think you and I both hear all the time, and certainly our listeners do, of what it means to be, quote, unquote, ” AI ready.” You can’t open up an article without that being mentioned. I love your lens. What does it actually mean to be AI ready? And if a company says that to you, what are you expecting to see in that portfolio?

Roy Illsley: It’s a very interesting question because when you talk to customers, if they’re AI ready, they have got a lot of things in place that all mean that when you discuss stuff with them, you get something that makes perfect sense to me in that privileged position to understand. And it begins with having the training and the skills across the organization. So the C- suite have got to understand what AI is. And all of your business users, all of your end users have also got to understand what AI is, what it can do and what it can’t do. As an interesting example, I was speaking at an event about a year or two ago now, and I had this person excitedly after my presentation come up and tell me that they’ve got a great use case for AI. This use case was, we can develop this for our cabin crew and our airline that will tell them what the weather’s going to be, where they’re flying to, so they can pack just the amount of clothes that they need. Well, I looked at them and said, ” Google will tell you that.”

Bradd Busick: That’s so good.

Roy Illsley: So once you’ve got the understanding, then you come into the practicalities of, well, am I ready for it? It was a conversation I had with the CEO of Farmers Insurance a few years ago. And what she was saying then was it took them seven years to sort their data out before they could even begin to start making sense of AI. So you’ve got to have the skills, you’ve then got to understand AI relies on data, quality, security. So once you’ve got those two things in place, the next thing that makes an AI ready is having the maturity of the governance, having somebody with responsibility. And I think you saw this in the US in a lot of the states about a year ago suddenly started employing people as their chief AI officer for their state because they understood what it meant to deal with AI, the rules, the regulations, the governance, and how it could be policed.

So they needed a person responsible for it in the organization that other people not necessarily reported into, but they could make the decisions that are in the best interests of the company and of AI. Because let’s be honest, if you talk to a developer, they’re probably super keen on using something like an OpenClaw and developing this and developing that, and they’ll be brilliant at it. And it might solve a problem, but it might actually introduce governance issues or risks. And that’s the key that you have to balance, and that’s where this role comes in. So I’d say those are the three key steps, and then you can start having meaningful discussions.

Bradd Busick: Yeah, there’s a rhythm and order and discipline, I think, to those three key steps. And to your point, it still feels new enough that we don’t fully understand the problems that might exist when we say yes to autonomous coding or a headless CRM. We literally don’t understand the butterfly effect for this. And so as I think about organizations, quote, unquote, ” being AI ready and what’s real and what’s hype,” I actually want to get into the economics behind this because I think everyone’s familiar with hidden expenses. When it comes to being burned on cloud costs or you have a wake- up bill from your large language model, it’s like, ” Holy cow, I had no idea that we were burning $30, 000 a month in tokens because someone just had a specific model at full throttle.” But this idea of hidden costs being tied up in something like employee resistance or adoption to me is much more insidious. I’d love to get your lens, just given everybody you get to talk to, what’s the most expensive surprise that you see organizations encounter once they actually start scaling AI?

Roy Illsley: Well, I suppose it takes two forms really. I was talking to, and I think it was Google, I might have the company wrong, and they were telling me about developing their AI and how one developer burnt through thousands of tokens in a night and they spotted it, stopped it, and then reassessed how they did it. But to your early point, developers using tokens, not understanding the tokens, businesses not understanding the token economy is one aspect that you regularly see. And in fact, when I was at GTC earlier this year in San Jose, I was talking to some companies that were going, ” Well, in our hiring in Silicon Valley now, the developers want to know what token budget we are giving them. Not just how much we’re paying them, but how many tokens are they going to play with?” So that aspect of it is one thing that people are beginning to get their head around.

But where I think the biggest thing with AI that people just really do not understand is A, where is it in the organization? Do you know who’s using what AI in your organization? Lots of people pay $ 20 a month per person per employee for Microsoft Copilot. How many people are actually using Copilot? Are you paying millions of dollars to Microsoft for something that is not used or used by 5%? We don’t monitor and measure it. And if we don’t monitor and measure it, we can’t then understand the costs and then we can’t work out what the value is. And strangely enough, I was in a conversation with IBM the other day about how you get and understand from a FinOps point of view, not just the cloud costs, but the token costs, and then translating that into business value and how that scales. Because you might start AI and find this brilliant use case which costs X, but as soon as you scale it, the scaling costs may actually outweigh the benefits that you’re going to get in. We haven’t done their maths. We don’t know how it works.

Bradd Busick: That’s really good. That’s such a great call out. And I am incredibly mindful now more than ever that not only do organizations not understand AI and what it costs, but they also don’t understand who should be leading it. I’ve seen organizations have a, quote, unquote, ” strategy committee that AI sits in,” which is absolutely the wrong spot because nobody from IT is on that strategy committee. But I think our listeners are really keen to this idea that you just mentioned, which is should we start thinking about in our recruiting strategy, what models do you get to use? Are we an OpenAI shop? Are we an anthropic shop? Are we a copilot? Are we Grok? And what is your token rate? I’m wondering, just given your lens, Roy, how do you actually think about quantifying adoption and value creation in an AI build- out? Are there some use cases that you can talk about that have set that benchmark where it’s measurable, the ROI was meaningful, and everybody could look across the room and say, ” Yep, we need more of that.”

Roy Illsley: I think one of the things that organizations have quickly come onto is this ability to understand and fail fast. If you try it, it shows that return in that market. It means the market, the technology is mature enough for what we’re doing there. In another case, it’s not. But I think what we are really just doing here, and we’re already talking about here, is we’re tinkering about with the edges, we’re talking about 20 to 50% productivity gains. The real benefit of AI, and particularly agentic AI, is the three, four, five, 600% productivity gains. But that is going to challenge any organization. I don’t know an organization that’s got a head around this yet, because when you think about what that means, let’s take a step back and let’s say we do a job as an individual. I’ve got a job, you’ve got a job. We’ve all got jobs.

Agentic AI is not going to replace our job. What agentic AI will do is a particular type of operation. So it may be I’m an optimization AI engine, or I’m a customer service sort of AI engine. It’s got a particular function and task, and it does that. Whereas in a human job, the human job includes a bit of optimization, a bit of this, a bit of… So it includes all of them. So the agentic AI is not a direct replacement for a human being. It’s picking bits of what each human does, put it together. And that means that the roles, responsibilities, and organizational structures and the skills you need in organizations in the future are going to completely change. And to get that level of benefit out of it, effectively, you’re going to have to redesign the organization and have a completely different business model. Nobody yet has done that because as humans, we are very slow to adopt to change. We can always think of a reason why we shouldn’t do something.

Bradd Busick: Yeah, that’s right.

Roy Illsley: Even though it looks good, we go. So I think-

Bradd Busick: There’s a reluctance.

Roy Illsley: … where we are in the evolution here.

Bradd Busick: Well, you’ve got a chance to see a ton of wins, but also some areas that we’ve talked about even today where people are struggling. We know that the hardest part of embedding AI into business workflow is that transformation or the surprises that you just don’t anticipate until you start it. Have there been any horror stories that would just make our listeners’ blood run cold that you’ve seen that was just like the, ” Hey, I’m going to use this as the case study of what not to do?”

Roy Illsley: Strange enough, I was speaking at the Park Place cab in Phoenix a couple of years ago, and I was talking to a couple of the customers there about it. And it’s this notion that your data that you’ve got in your organization is your differentiator. Because let’s think about this in reality. The AI platforms, the AI technology, there’s multiple and varied ones of them out there, but essentially they’re all doing pretty much the same thing. Slight variations on how they do it, how good they are and stuff like that, but they’re all doing the same thing. So if everybody’s got access to the same tools, how do you compete in the future?

And your competition is based on the quality of your data that you put into those models and the quality of your people to extract the value out of those to drive your business forward. So it comes back down to your data. We have heard horror stories of companies without the governance structure where people have uploaded to Gemini Pro or Grok or Claude corporate data, use the AI, been quite happy with the result, and then not realized that data that they uploaded has been retrained in the model and is now available to anybody. And I think it was Samsung where they had some data that did this a couple of years ago.

Bradd Busick: It was.

Roy Illsley: And it was sort of like, ” Woo.” Yeah, that’s the horror story. So make sure governance is in place. Make sure your people, this is where I come back to the original training. If people know that’s the data, I’ve got to treat it like that. I can only do it this way. I can’t put this information out there. This is how I want to use it. Then that helps you because if not, you could leave yourself open to compliance issues, all of the risks, and eventually you could just be, ” Oh, everything that my company’s developed is out there in the internet for everybody else to find.”

Bradd Busick: Yeah, it’s a perfect example of an organization bleeding out and not actually knowing that they are until their competitors have all of their data. When you think about the requirement to change as an organization, what does a company have to do differently to make the cost of their AI build out worth it?

Roy Illsley: Well, like everything, it is all of it. But it starts with a mindset that says we have to have this view that, yes, everybody’s empowered within the governance rules that we’ve set. Everyone knows what the cost of what they’re doing is. You’ve got to have people that are aware that this is not an academic exercise. You’re a business. Don’t get swept up in it. Understand what it’s costing you and what you can do. And if it’s too complex, if it’s not working, if it looks risky, then speak to somebody else because a collaboration of people will help.

And this is where organizations sometimes, particularly back in the early days of the internet, where everybody started using different browsers. You had the browser walls and your computer would have 10 different browsers on it, then everybody would have the favorite one. We’re in a similar sort of scenario now. What you’ve got to do as a company is almost legislate, here’s your budget, here’s the rules. You’ve been trained, go play with it, but know your limits. Share ideas. Because all too often as what happened with Excel, you get a thousand different versions of the Excel spreadsheet that calculates profit and loss in the company.

Bradd Busick: That’s right.

Roy Illsley: You do not want a thousand different agentic AI systems being developed. So it’s mundane. It’s control, it’s management, it’s knowing what you’ve got and knowing the cost of it. And those fundamentals don’t go away with any technology. If that’s all in place, then it will work. If it’s not, you’re just going to go through what the PC did because I was working in the industry then when we went, ” Ah, that’s all right. And business users won’t do anything with PCs.” Two years later, we’ve got to find out how many PCs we’ve got and what software’s on it and what they’re using it for and where the data is.

Bradd Busick: Absolutely.

Roy Illsley: And we never really did.

Bradd Busick: No, and we still haven’t. It’s actually gone further. I think in healthcare now you’ve got, ” What is this thing that a patient plugged into the wall?” Oh, they brought their own Xbox to the hospital and plugged it into a network port that happened to be live and security’s all over it. So I agree, we’re introducing some new risk. We’re also giving every user a pair of scissors to go run with and hope they don’t hurt themselves or hurt other people. But I hear you saying so much about success in this new world is really changing our processes versus deploying a model. Because at this point, all the models are pretty darn good.

Roy Illsley: I was discussing this with a colleague the other day, which is you are going to find the use of CPUs, GPUs, and quantum processes all being used together to solve AI at scale for big problems. But for most organizations, 99% of organizations, they’re never going to do that. They’re not going to have that level of need and complexity. So it comes back down to, as Dell said a couple of weeks ago, you can now do desk side AI and you can buy their little boxes. They’re nice little boxes that cost 20, 30,000 pounds for these boxes, water cooled and video chips for a developer, but they pay for themselves in six months because effectively what you do is the tokens that you would’ve spent on the cloud, you could do on this box.

Bradd Busick: Incredible.

Roy Illsley: And it will generate the same number of tokens. And once you bought the box in three months, it’s paid for itself because software to add on and various other bits, but that’s why it’s becoming so challenging for CIOs and businesses because everybody’s giving them a different answer. Do I use the cloud? Do I use on- premises? Do I use hybrid? Do I do it myself? Do I use partners? Can I use Chinese technology? Can I use American technology? It depends where you sit in the world, what industry you’re in, what regulations you’ve got. All of a sudden they’re going, ” I’m scared. What do I do?”

Bradd Busick: Absolutely. And they should be. It’s both an exciting time, I think, to be a CIO or to be an IT leader in this space. If you operate from this premise that today’s the worst it’s ever going to be, it’s a really awesome opportunity to go and innovate and create. And what is true today will absolutely not be true tomorrow. I think the savviest CIOs that I know and have the pleasure of working with understand this balance and there’s nothing definitive, but it’s the agility to be able to change if and when we need to get a bigger return to lower our costs or to accelerate something across the organization that wasn’t even possible 30 days previous. So it’s an exciting time to be a leader for sure.

Roy Illsley: Yeah. And the other thing I think we have to factor in here is that the rate of change here is accelerating. Whereas in the past, everybody has adopted a new technology as it’s arrived. With the current evolution of technology, enterprise customers may have to go, ” You know what? I only adopted that technology last week. I’m not changing for another three months or six months. I’ll skip the next technology. I’ll go onto the one after it.” That skipping technology sections, that’ll be a choice that strategically you make of when we change and how fast we get business value out of that new one. Because the technology vendors will be selling you the biggest, the best, the greatest, and telling you, ” This is all going to save you money.” But if all you do every day is keep change, you don’t actually do anything.

So there is this pragmatic approach that says, ” If change is too fast, I can’t accommodate it. I’ve got to accommodate what I can and get value out of it before I can do the next change.” So that is something CIOs have got to get their heads around in terms of what does that look like? How can I do that? Can I do it continuously? Which currently we can’t, but who’s to say with AI in the future that AI can’t autonomously adopt a technology, understand it, and tell us that it’s now ready to use, so it’s fully adopted and do that within hours rather than taking months. We don’t know. This is all speculation, but it’s a world that is going to require a different way of looking at it. And the brave, if they succeed, will be the leaders. Those that are slower will find that they are too far behind to even catch up.

Bradd Busick: They can’t catch up. I totally agree with you. And it’s a brand new paradigm. All right. Well, it’s been a pleasure chatting with you, Roy. I’m grateful for your time. Thank you so much for joining us on The Savvy CIO.

Roy Illsley: Thanks for having me and happy to help and chat anytime.

Bradd Busick: I loved my conversation with Roy today. So many areas to cover, but the key takeaways for me, I think Roy beautifully highlighted the importance of agility and the organizations that are willing to be agile and have an aptitude for change in support of driving down cost or realizing value. And he also was incredibly surgical about those organizations that aren’t going to have that rigor and the risk that they face, which is one, you’re going to fall behind. And two, the pace of the technology is actually running faster than your business today. So how do you actually measure value in a way that keeps you on the path to know that you’re winning, but also having a careful balance to know that once you scale to a certain level, going back to check on that ROI to say, ” Is it still returning what we thought it would return?”

That’s all for today. Thanks so much for listening. Please follow us so you don’t miss an episode. This has been the Savvy CIO brought to you by Park Place Technologies. If you want to learn more about Park Place, go to www. parkplacetechnologies.com. And now one last word from our guest, Roy. This show’s called The Savvy CIO. So what’s the savviest choice you’ve made in your career to date?

Roy Illsley: Oh, that’s a tricky one, but I would say it was when I left industry and became an analyst, being freed into a role where I don’t manage people. I don’t have a budget to manage. I just get to go out and talk to the brightest minds about the future technologies, understand how they operate and work, talk to the CIOs, understand their pressures, and bring all of that together for people. It really doesn’t feel like a job, obviously. Every time you’ll meet an analyst, you’ll go, ” When I had a real job,” because it feels we’re so privileged to get access to all of this, bring it out to people and share it so people can assimilate the information in a quick, condensed form.

Bradd Busick: I love that answer. And look, AI is never going to replace Roy, but if there’s a Roy LLM, I’m subscribing for $ 100 a month. So I’m absolutely in. I’m your host, Bradd Busick, and as always, IT shouldn’t just be at the table. IT is the table.

Roy Illsley has over thirty-eight years of IT experience, working for a variety of consultancy and end-user companies with experience in the defence, utilities, automotive, retail, and Fast-Moving Consumer Goods (FMCG) industries. He works in the Cloud and Data Centre practice and is recognised as Omdia’s expert on Cloud, Virtualisation, and Management. He also has experience of Data Centre technologies, Quantum Computing, IT Service Management, and covers IT Strategy and Policy. He has a BSc (Hons) in Electronic Engineering and Telecommunications, an international MBA, and is a Member of the Institute of Engineering Technology (MIET) holding the registered status of a Chartered Engineer (CEng).