The Savvy CIO: Episode 2 – “Liquid Cooling: When, Why, and Which?”

Zac Smith: If you’re looking at putting computers into market before 2030, you should probably be looking at understanding liquid cooling and where and how it fits within your portfolio. Can you support it in your data centers? What are our limitations or options there? Because I think you’re going to wake up in 2028 or 2029, it’s going to be here before you know it.

Bradd Busick: You’re listening to The Savvy CIO: Modernize Wisely by Park Place Technologies. I’m your host, Bradd Busick. This is a show where we talk about budget pressures, AI adoption, security, and the art of keeping the lights on without burning everything down, and the people who are tackling these problems every single day.

For the past 30 years, the number one rule in data center etiquette has been really simple, keep liquids away from the hardware, full stop. And for the past three decades, that rule has actually worked really well. Air cooling was scaled, racks stayed manageable, and no one had to put rubber boots on if something went wrong. But now, we have this little thing called AI. It feels like almost overnight, the power unit inside a single rack could cause a warehouse- wide meltdown. Air can’t cool things fast enough, and thermal management is almost impossible to control. So now, the industry is betraying the number one rule. We’re actually putting liquids right next to the chip itself. As is usually the case for new tech, there’s no one right way to do it. But whether you go with immersion cooling, direct- to- chip cooling, rear door cooling, or play it safe and stick with traditional air cooling, each option comes with its own trade- offs, headaches, and net- new questions. And to be frank, this isn’t the next five- year problem, it’s a now problem.

So whatever your business, whatever your industry, whatever your vertical, whatever your next project or deployment, you need to start thinking about this now, and you need to know what liquid cooling actually is, what it isn’t, and how to make the smartest decision before that decision gets made for you. Zac Smith is exactly the right man to tackle this conversation. He’s spent his career on the leading edge of what happens when software ambition runs headfirst into physical infrastructure. He’s pioneered pluggable liquid cooling as an open standard, and knows that it’s less about chasing trends and more about solving a real, hot, and expensive problem. He’s got a whole tech and musical resume behind him that led him to where he is now, the CEO and co- founder of Datum. Welcome to The Savvy CIO, Zac.

Zac Smith: Thanks for having me, Bradd.

Bradd Busick: It’s good to be with you. I appreciate you making time. So look, I mean, you’ve been on this twisting path that led you to becoming an expert on liquid cooling. Can you tell us the story that allowed you to get to this point in your career and how you arrived at this moment?

Zac Smith: Yeah. I think like most people, I didn’t go to school and get a degree in cloud computing in the data center in the 1990s. Frankly, I was like every other kid growing up in the ’80s, I loved video games. I wasn’t very good at playing them, but I did have some friends who were very good, so I always usually ended up being the hardware guy and modding the PC, figuring out if we could overclock it, things like that. Fast- forward a bunch of years and I ended up going to music school in New York City. And as a 17- year- old trying to pay my way through college, I ended up fixing PCs as my side gig. And so, that was my extent of computer training.

But shortly after I graduated, this was 2001, the dot- com boom was in full swing, and I landed myself out of college with a degree in classical double- bass performance and ended up starting my own business to do web hosting for musicians. I was going to get them online. And at the time, you kind of just went to your local data center, there was a lot of dial- up and DSL providers who had maybe a couple racks or things like that, and you figured out how to borrow a machine and get some 1U colo or a shared something or other. And before you know it, I was hosting a Linux server, and ended up partnering with a friend who knew a lot more about computers and Linux, and we built one of the first Linux- based cloud computing companies.

And along that path, you had to kind of get into the data center. I remember the very first one, I’m sitting not far from a place called Telehouse, 25 Broadway in Lower Manhattan, where we started racking some of the very first AMD Opteron chips. And if you were around in the early 2000s, that was the Porsche 911 of server chips because they introduced 64- bit, you could put more memory, and they kind of ran hot. And that was my first interaction with overheating a data center, because at the time, you bought data center space by the rack, and they didn’t have any concept of how much power or cooling. You just kind of got the rack and stuffed whatever you could in it, and boy, did we stuff. We were putting, I think it was 42 DL140 HP servers with Opterons in it, and we melted the place.

Bradd Busick: That’s incredible.

Zac Smith: And that was my first interaction with cooling.

Bradd Busick: That is incredible. I mean, one, what an awesome background, coming out right about the same time as you. Natalie, can I do a throwback to my super skills that were hardened during cheat codes to Mike Tyson’s punch out 007-373-5963, which takes you right to Mike Tyson.

Zac Smith: Obviously.

Bradd Busick: But, I mean, you are a classically- trained musician. I mean, what an awesome, beautiful mind that actually probably enables a lot of the work that you’re doing today in a space that so many people don’t even appreciate or care about until it breaks. And then, all of a sudden, it’s like, ” Zac, what do I do?” So, I mean, now that we have a little bit about your background, let’s set the table here for our listeners. Why is everybody talking about or should be talking about liquid cooling right now?

Zac Smith: What we have is a really scaled world of computing that frankly for most people has just worked, and that’s pretty amazing, but it has meant that it’s been abstracted away from a lot of people. We’ve had really good system integrators, we’ve had great data centers, and the rate of change on the Moore’s Law of doubling computing capacity every 18 months, and over the last five to seven years, we’ve really run out of the physics. We’re doing chips at three nanometers, fractions of a human hair kind of thing, and what that meant is we ended up having to put more power through the chips, and then more chips on the chip. So chiplet design and just putting more circuits. And at first, it was improving the physics to use more transistors. And then, it was putting more layers by stacking them. So we just kept extending, extending this kind of movement of computing.

And then, here comes GPUs, which are basically vector multi- lane processing, thousands of chips on a card, so it took that to a nth degree. What used to be a little thing you’d run for your video game is now chock- full of processors and memory. So as we had to get denser in memory and faster, we were putting more power through those memory chips. And then, it went to network cards. And so, every part of our physics has just gotten more power- hungry. And as these applications around AI have really taken advantage of these new accelerators, we’ve moved to what I guess people call Jensen’s law, which is what can we do with the amount of power we have? Because that is kind of the limiting input right now. So how much intelligence or how much value can you drive per watt?

And we’re doing that primarily through a lot more transistors and power. And in the data center world, where we’re building off of 30- year designs, how do you make a building last for 20 or 30 years, but we have computers changing at this one to two- year cycle in leaps and bounds? These are just a mismatch, and that’s why cooling in general, I guess we could call it heat extraction, has become a major pain point in data centers that really are not tooled for the current machines we’re trying to put in them.

Bradd Busick: So Zac, I mean, part of the skills that I think today’s savvy CIO has to have is being able to translate wildly complex concepts and tasks to board members who may not know what they’re talking about, but also be able to talk with engineers and partners. And so, I love some of the skills that have come out with the LLM models, one of them being ELI5. Explain it to me like I’m five. If we had to break this down for our users, what exactly is liquid cooling and what isn’t liquid cooling?

Zac Smith: I used to teach a class at the local public school about how the internet works, and the kids were kind of confused at first because they thought it was just Wi- Fi, the whole internet was basically wireless. They got it because planes don’t always have Wi- Fi, so they figured that one out. But if we’re to take cooling and data center problems, you’ve been on your laptop, had your MacBook, it’s pretty good at handling heat, but it does get hot sometimes. You’re watching all the movies and doing all the things and it gets warm.

In physics, you’ve got this problem where whenever you use something, you’re going to create some other action, and my dumb understanding is that when you use an electron, it’s going to create heat. And that’s the exact thing we’re dealing with in the data center, is how we have so much power coming into these computers, those are all going to create heat. And normally, we would have a process called air cooling, which we would blow a fan around and move cooler air, and try and push hotter air out. But there’s only so much, frankly, air is a really, really bad medium for heat exchange. And so, now, we had to look at, because we’re creating so much heat, how do we capture that heat? And the answer looks just like your radiator, like in your car.

And so, we basically have started to have to put radiators, heat exchangers at various points. There’s a bunch of different ways to do this. You could use a radiator on the roof of the building to fix that with your air conditioning. But you can start to move that closer and closer to your workload. And so, that’s kind of what I think about it, is just taking this heat exchanger and moving it to different parts of the IT stack.

Bradd Busick: So with that mental model in mind, are all types of liquid cooling created equal or are there trade- offs?

Zac Smith: Yeah. I think there’s so many trade- offs, and this is probably the challenge for the modern CIO or infrastructure consumer right now, is because you’ve got a bunch of choices to make and there’s not a perfect one. So first and foremost, I think the vast majority of CIOs are dealing with mixed environments. They don’t just have clean rooms where they get to go spend a billion dollars and build out the latest generation of everything, and then two years from now, go build it again. They’re probably dealing with a whole bunch of generations of things, and budget constraints, and existing facilities and leases that you might have to be thinking about or retrofitting, or co- location partners that you might be using that have constraints.

You mentioned earlier about the age- old problem of don’t bring water into the data center. I mean, there are a lot of co- location facilities that are still really not capable in thinking about, well, how am I going to bring this in? And even if they do or are open to liquid cooling, they might not support different types. And this also has regulatory and regional impacts, because some of the fluids, we say liquid, we think water, that is not always the case. So you might be familiar in your air conditioning, we used to use CFCs, and they were really efficient, but they were bad for the environment. Then they got outlawed in certain areas, and we have different kind of coolants that you can use, similar in your radiator.

That’s the same problem we’re dealing with in the data center, which is are these water- based? If so, how much water do they use? Are they closed loops, they can kind of keep using those same water, or are they not closed loop and they have to evaporate it constantly? Maybe you have a water shortage, actually, and that’s a very common thing in a data center. Or is it a different kind of chemical, like a glycol that you might use in an industrial cooling loop, or maybe some other sort of coolant? And so, I think that there’s a lot of choices, and so I think it’s really important to have a partner or to be able to evaluate your specific situation and what you’re trying to accomplish there and your environment that you have.

Bradd Busick: That’s a great summary of all the different choices that a CIO could, or in some cases, not even know that they’re making.

Zac Smith: All the problems.

Bradd Busick: Yeah.

Zac Smith: All the challenges.

Bradd Busick: Yeah. That’s wildly complex. I mean, as a CIO, how would you actually know what type of liquid cooling you need or what’s right for you? And what are the business trade- offs that you’re making when you try to figure out what path makes the most sense?

Zac Smith: I spent some time at a large data center operator, and we were multi- tenant, and so we were dealing with a bunch of different customers, thousands of customers, who all had different needs and designs. I think the first thing that you should look at is understanding your workload, what you’re trying to accomplish, because what might be a good scenario for your AI factory that could have specific requirements from your vendor could be pretty different from your general purpose IT or modernization workload that you’ve got. So I don’t think you can start solving this problem without thinking about what workload and the technology stack that you’re going to implement from a hardware standpoint.

And then, the second thing I would go to is just looking at your facilities. If you’re shopping for a new facility or building something yourself, you might have some different options and you can be a little bit more opinionated about the market. But if you’re an existing facility, you can dream up all the most interesting ideas, but you’ve still got to go talk to your data center partner. And that would be one of my first steps, is saying, ” Hey, what options are you supporting today?” I would ask if you’re supplying a building loop, if they’re letting you bring a CDU, a cooling unit into your rack or cage. These would be first questions I would ask, because that will help lead you down the options of, do I look at a rear door? Do I look at something that’s chip- adjacent with a CDU? Do I look at full enclosed or immersion? That’s going to end up there. And the secret one I would also just ask before you go down the immersion game is just floor loading.

Bradd Busick: Okay.

Zac Smith: I know it sounds like a boring question, but the number of people who would ask, ” Hey, can I do this?” And we’re like, ” No, actually, it’s in Japan and they cannot take that amount of floor loading per square meter.” So don’t get yourself into a situation where you’ve maybe designed and procured some potentially very expensive equipment, and your facility can’t take it.

Bradd Busick: That’s really good. The sound of conviction, by the way, that we’re all hearing is all the CIOs that just drove off the road because the last time they were in their data center was never. So I love the call- out, look, can we tap a vendor partner here to say, ” Have we thought about this?” And I know in the world of healthcare, it’s like going to your provider and saying, ” Hey, I heard about this treatment plan or this care pathway. What do you think?” And actually surfacing that with your partner who should be able to help guide you. So I love this call- out.

Let’s maybe now take it a step deeper. I think the lie that we’ve always told ourselves is liquid cooling was just for hyperscalers and Bitcoin miners. Were you saying, at this point, that everybody, regardless of industry, regardless of vertical, should be surfacing liquid cooling as a capability to run their shop, or is it unique to certain niches at this point?

Zac Smith: Yeah. It’s kind of interesting, because the dynamics of the industry, when you look at processors and systems made in the world, I would suspect 60% or 70% of the world’s computers end up at a hyperscaler. And so, whether you’re designing for the same scale as Amazon or Microsoft, the fact of the matter is the supply chain is drafting towards where the volume is.

Bradd Busick: Yeah.

Zac Smith: And that means that hyperscalers are moving towards liquid- cooled designs and high- density architectures, and that means you’re going to be moving that way too.

Bradd Busick: Yeah.

Zac Smith: You might not feel it in exactly the same way, but I don’t think you can buck the trend, or want to, because you want to draft in the sweet spot of the supply chain, for most cases. There are obviously exceptions. So I think that although there will always be air- cooled options, and I see that going for quite a long time, I suspect we’ll have optimized solutions that are built for air- cooled, I think that liquid is going to become part of most people, if not everybody’s environment, to some degree. And that will just be like how we all move to high core count servers because that’s frankly where the supply chain moved the market with the large buyers.

Bradd Busick: So there’s clearly a lot of different thoughts about liquid cooling. Here’s Chris Carrero, chief technology officer at Park Place Technologies, on some of the challenges data centers face in implementing liquid cooling.

Chris Carrero: One of the items is governance. A lot of that governance, it doesn’t exist yet in the liquid cooling. There’s really not much standardization. Different OEMs have their different products, different connectors. There’s different systems that don’t interoperate together. But that standardization definitely needs to come together.

There’s a lot of differences between traditional air cooling and liquid cooling, where air data centers have been around for 30 years trying to sustain the three 10kW cabinets. Now, you’re talking 50, 150kW, 200kW cabinets. And then, on the horizon, we’re looking at half- a- megawatt, one- megawatt racks with some of the things that Nvidia’s talking about. These data centers aren’t built for it. I would say the greenfield space, so new builds, new designs, they’re starting to implement the different provisions for liquid cooling, where the brownfield is lagging behind and not having different pieces to get these data centers built up is kind of the challenge in where we’re at right now.

Bradd Busick: And so, Zac, if this is going to become gospel for all of us in this space at some point, whether it’s now or three years from now, what does that runway look like for you where CIOs will actually start seeing this manifest for their transition? Is it three years? Is it 2030? Is it next month?

Zac Smith: Yeah. I suspect most CIOs are no longer on a five to seven- year hardware refresh cycle in their data center. They might be a little tighter than that-

Bradd Busick: Yeah.

Zac Smith: … with some of the technology changes. But three years is a long time. And so, I think that if you’re looking at putting computers into market before 2030, you should probably be looking at understanding liquid cooling, and where and how it fits within your portfolio. Can you support it in your data centers? How would we do this? What are our limitations or options there? Because I think you’re going to wake up in 2028 or 2029, it’s going to be here before you know it. So I wouldn’t ignore it. I’d get your ducks in a row. Maybe you start with a trial or some sort of a lab system to understand that with a partner.

And then, start to get your hands around it operationally, because there’s a bunch of operational changes that happen when you introduce liquid and these high- density… It’s not just liquid. You’ve got an extreme amount of power, maybe a system. For example, if you’re investing in AI infrastructure, you might be looking at the new Vera Rubins or your Grace Hopper. These servers are a couple of hundred thousand dollars a piece.

Bradd Busick: Yeah. Yeah.

Zac Smith: And you can’t just go in there and fix them. You might have a significant portion of your IT infrastructure out of commission because-

Bradd Busick: That’s right.

Zac Smith: … you had a liquid cooling loop problem and you didn’t monitor it, and then it fried the $300, 000 system and now you’re bummed out of warranty.

Bradd Busick: That’s exactly right.

Zac Smith: Even if you’re going to have a little bit of this, either AI or high- performance, you need to start thinking about getting some education and trial balloons going.

Bradd Busick: I love the educational lens here. I mean, there’s an entire generation of IT professionals that we both have worked with, who their entire career was built around air- cooled architecture, and we’re used to having fans.

Zac Smith: Yeah.

Bradd Busick: Or during a downtime, people are wearing headsets, calling in from the data center to try to get it up, but they had to wear the headset because it was so loud because of the fans.

Zac Smith: So loud.

Bradd Busick: If we have to help our colleagues that are listening here through this change curve, talk a little bit about some of those cautionary tales that you’ve heard about liquid cooling implementation. I mean, things look really awesome. I think all the CIOs that I know go to executive briefing centers and we’ll go to trade shows and I’ll talk with my partners, and it looks amazing on the vendor side, but you kind of talked a little bit about some of those operational headaches. Give me some of the talk track for what should you be asking about and what should you be thinking about if you want to start piloting that capability?

Zac Smith: Well, I think that we have, for a long time, benefited from what I’m going to call either a blissful unawareness or a bifurcation between our hardware partner and our data center partner, where we could kind of buy whatever we wanted and generally put it wherever we wanted.

Bradd Busick: Yep.

Zac Smith: Okay. Maybe it costs a little bit more because we’ve got some more power whips. But those days are over, and you can’t design and buy hardware and not be talking with your data center-

Bradd Busick: That’s right.

Zac Smith: … or facility partner along with it. And I think it’s not just can it work, do we have the right hookups, availability, regulatory approval, whatever, you have a couple of real operational concerns that I think you kind of need to surface. Number one is just observability. Are you having the system tools to observe and monitor this additional amount of mechanical infrastructure? That might be a CDU, that might be a pump, that might be a throughput or a temperature at each server. So the metrics, and I’ve heard this from some of our AI partners who I’ve worked with, those can be extreme, where you’re pulling telemetry-

Bradd Busick: Yep.

Zac Smith: … off of liquid cooling infrastructure more than you’re pulling it off of your server infrastructure, to some degree-

Bradd Busick: Yeah.

Zac Smith: … because you’re operating basically a mechanical plant, or somebody is somewhere.

Bradd Busick: Yeah.

Zac Smith: And so, I think that’s one area, is just looking at your toolset in terms of software and IT management, and is it going to be sufficient or do you need to invest further in the operational tools around that?

And then, the second one would be, I like to call it the 2: 00 AM- in- the- morning problem. And the 2: 00 AM- in- the- morning problem is your super cool server, it’s running production workload, everything’s happy path, and I don’t know, something breaks. Let’s just say it’s a hard drive. You’ve got to go replace the hard drive. Normally, put in a little support ticket, your data center partner, your tech team there is going to go into the thing, swap out the drive, way to go.

Bradd Busick: Right.

Zac Smith: But what if you have a whole bunch of liquid pipes and you can’t pull it out of the rack because you don’t know how to open up the liquid pipe? What happens if I unplug this?

Bradd Busick: Right.

Zac Smith: It used to just be the data would go down, but now I’m going to have some weird coolant and it’s going to kill me or it’s going to-

Bradd Busick: Yeah.

Zac Smith: … ruin the data center? I’m not going to touch that. And so, that’s the other thing that I think really needs to be talked through is, hey, day two operations, do we know how we’re going to work this? And do our partners there, who might just be a vendor that we call, might be a remote hands team, or it might be our own team, are they going to be educated? Because they might be like, ” I can’t touch this. I don’t know which is red or blue.”

Bradd Busick: 100%. Yeah.

Zac Smith: “Do I pull that one?”

Bradd Busick: Yeah.

Zac Smith: That’s the 2: 00 AM- in- the- morning problem.

Bradd Busick: Well, it’s not lost on me that the topic for today is talking about keeping things cool, and I think all the listeners would agree, we’ve had one of the coolest cats in the game with us today. Zac, thank you so much for joining us on The Savvy CIO. We appreciate your time.

Zac Smith: It’s my pleasure.

Bradd Busick: As we think about this wildly diverse space, I think it was very clear that physics were not the problem anymore, and we are quickly outpacing what our data centers can handle. And so, the modern CIO today needs to start thinking about this idea that their AI workloads are blowing way past what normal cooling can actually consume, which then introduces really interesting conversations with your vendor. How do you have the right questions to ask your vendor, and what are your new operational requirements that your vendor may not be thinking about yet? The new modern AI infrastructure requires power and cooling at a different scale and at a different grade, having a thought partner that can help walk you through some of the questions that until you listened to this episode, you may not know that you even needed to ask.

That’s all for today. Thanks so much for listening. Please follow us so you don’t have to miss an episode. This has been The Savvy CIO, brought to you by Park Place Technologies. If you want to learn more about Park Place, go to www. parkplacetechnologies. com. And now, one last word from our guest. So this show is called The Savvy CIO. Zac, what is the savviest choice you’ve made in your career to date?

Zac Smith: Probably the savviest choice I ever made was asking dumb questions to absolutely everybody I ran into. I think it’s one of the superpowers of our industry. There’s lots of helpful people, everybody is excited to share, and that has served me so many times, being the dumb guy in the room and just asking one more question.

Bradd Busick: That’s such a great call- out. You don’t always have to be the person with the answer. You actually just have to be the person that has the guts to say that ” I don’t know.” Thanks so much for that. I’m your host, Bradd Busick, and as always, IT shouldn’t just be at the table. IT is the table.

Veteran cloud builder with 25 years experience. After getting my start in the early days of Linux hosting (Voxel), I built a bare metal automation startup (Packet) and then led digital infrastructure services at Equinix. Now building Datum to help power the next 1,000 clouds.