AI’s Volatile Power Demand Is Damaging Its Own Data Centers

Skip to Content News Archives Economy Energy Oil & Gas Renewables Electric Vehicles Mining Commodities Agriculture Real Estate Mortgages Mortgage Rates Finance Banking Insurance Fintech Cryptocurrency Work Wealth Smart Money Wealth Management Investor Personal Finance Family Finance Retirement Taxes High Net Worth FP Comment Executive Women Puzzmo Newsletters Financial Times Business Essentials More Innovation Information Technology FP500 Podcasts Small Business Lives Told Tails Told Shopping Financial Post Store Obituaries Place a Notice Advertising Advertising With Us Advertising Solutions Postmedia Ad Manager Sponsorship Requests Classifieds Place a Classifieds ad Working Profile Settings My Subscriptions Saved Articles My Offers Newsletters Customer Service FAQ News Economy Energy Mining Real Estate Finance Work Wealth Investor FP Comment Executive Women Puzzmo Newsletters Financial Times Business Essentials This advertisement has not loaded yet, but your article continues below.HomePMN BusinessAI's Volatile Power Demand Is Damaging Its Own Data CentersEquipment failures suggest unforeseen costs and reliability issues at the multibillion dollar facilities.Author of the article: You can save this article by registering for free here. Or sign-in if you have an account.glzmsnkbgjouvvhu5ejmc9m]_media_dl_1.png BloombergNEF, Bloomberg Intellig(Bloomberg) — Artificial intelligence’s tremendous hunger for electricity is already well known. What’s less familiar is how the rapid fluctuations in data centers’ appetites can break essential equipment at the facilities.THIS CONTENT IS RESERVED FOR SUBSCRIBERS ONLYSubscribe now to read the latest news in your city and across Canada.Exclusive articles from Barbara Shecter, Joe O'Connor, Gabriel Friedman, and others.Daily content from Financial Times, the world's leading global business publication.Unlimited online access to read articles from Financial Post, National Post and 15 news sites across Canada with one account.National Post ePaper, an electronic replica of the print edition to view on any device, share and comment on.Daily puzzles, including the New York Times Crossword.SUBSCRIBE TO UNLOCK MORE ARTICLESSubscribe now to read the latest news in your city and across Canada.Exclusive articles from Barbara Shecter, Joe O'Connor, Gabriel Friedman and others.Daily content from Financial Times, the world's leading global business publication.Unlimited online access to read articles from Financial Post, National Post and 15 news sites across Canada with one account.National Post ePaper, an electronic replica of the print edition to view on any device, share and comment on.Daily puzzles, including the New York Times Crossword.REGISTER / SIGN IN TO UNLOCK MORE ARTICLESCreate an account or sign in to continue with your reading experience.Access articles from across Canada with one account.Share your thoughts and join the conversation in the comments.Enjoy additional articles per month.Get email updates from your favourite authors.THIS ARTICLE IS FREE TO READ REGISTER TO UNLOCK.Create an account or sign in to continue with your reading experience.Access articles from across Canada with one accountShare your thoughts and join the conversation in the commentsEnjoy additional articles per monthGet email updates from your favourite authorsSign In or Create an AccountBatteries, generators, cooling units and other critical systems are put under such strain at AI computing facilities that they are malfunctioning or prematurely reaching the end of their lives.As the AI boom accelerates, these technical problems suggest added costs and unforeseen reliability problems, with even a few minutes of lost uptime hitting data-center developers’ revenue. They come at a time when investors and lenders are already jittery about hyperscalers’ hundreds of billions of dollars of spending, amid growing concerns that these facilities could be depreciating much faster than estimated. Get the latest headlines, breaking news and columns.By signing up you consent to receive the above newsletter from Postmedia Network Inc.A welcome email is on its way. If you don't see it, please check your junk folder.The next issue of Top Stories will soon be in your inbox.We encountered an issue signing you up. Please try againThe problems are also a potential source of wider instability in power grids that are already straining to keep the lights on.“AI does create very unusual power demand,” said Amber Villegas-Williamson, principal consultant at the Uptime Institute in the UK, which advises electricity suppliers and data centers on standards and reliability. “It’s like over-revving your car wears out the engine faster than keeping a constant speed.”Data centers have been around for decades, gulping down electricity while they ensure that everything from your favorite streaming show to your online grocery order functions smoothly. But facilities designed for AI computing are different because their demand is so large and swings much more dramatically. Power increments equivalent to the consumption of factories, towns or even cities can appear and disappear within seconds, creating repeated shocks that connected equipment struggles to absorb. A gigawatt data center is equivalent to a city the size of Boston, half of which can flicker on and off every few seconds, said Shannon Miller, founder and president of Mainspring Energy Inc., which works on micro-grid projects for industrial and data-center customers. Some AI campuses planned in Texas and the US Midwest are more than five times bigger, consuming nearly as much power on average as New York City.AI data centers put particular strain on their power supply when they are training new models — a process that mobilizes all of the graphics processing units in unison. Like the digital equivalent of bees swarming or a school of fish changing direction, hundreds of thousands of GPUs can power up and down on a millisecond basis.AI at times sees power usage spike as much as 50% above its design capacity, “so a 1 gigawatt facility may use 1.5 gigawatts for a split second,” said Drew Baglino, a former Tesla Inc. executive who started Heron Power Electronics Co. The company is developing equipment to manage power fluctuations for Nvidia Corp.’s even more energy intensive next-generation of servers, due in 2027. This advertisement has not loaded yet.This advertisement has not loaded yet, but your article continues below.Most equipment isn’t designed for such big swings in power consumption. Jon Parrella, chief executive officer of energy-storage developer Terraflow Energy, likens it to driving a Ferrari and shifting straight from sixth gear to first. “You can’t swing that fast,” he said. This story is based on interviews with more than three dozen power experts in the US and Europe, including generators and other power suppliers, data-center developers, grid operators, utilities, investors, standards developers, insurers and regulators, almost all of whom said the physical stresses on the facilities were evident. Cranks on small natural gas combustion engines used to generate power at data centers have broken off, several of those people said. At xAI’s Colossus computing facility in Memphis, Tennessee, gas-fired turbines had developed cracks, one of the people said. Batteries were installed within the system to help smooth out power swings and reduce the strain on spinning turbines, the person said.SpaceX, the parent company of xAI, didn’t respond to requests for comment.Turbines have also cracked at much smaller data centers in the UK, said Andrew Cunningham, CEO of GeoPura Ltd., which is providing hydrogen for use in fuel cells that smooth out power flows at some sites.Cracks or wear on devices can cause electrical arc flashes — when a current jumps between conductors — potentially damaging AI chips, said Jennifer Scanlon, CEO of UL Solutions Inc., which tests and certifies new technology.There is a suite of equipment such as batteries, capacitors, transformers and flywheels that can help stabilize power flows. However, in the bid to build AI computing capacity quickly not enough of these technologies are being used at new data centers, several of the people said. Batteries that have been installed for this purpose have sometimes needed to replaced within months or even weeks due to the high strain, according to the Uptime Institute and other people working with operators.This problem is seen at data centers around the world, from the Middle East and Africa, to Europe and the US, said Villegas-Williamson of the Uptime Institute. These issues are already causing delays or curtailing operations – and therefore revenue – at some AI computing facilities. To ensure that a planned 2.67-gigawatt AI campus in West Texas can achieve the 99.999% reliability required by Microsoft Corp., extra time was baked into the schedule for engineering, said Chris James, CEO of Joulent Inc., which is developing the facility with energy giant Chevron Corp. This means power delivery will begin in 2028 instead of 2027, he said. If essential equipment breaks down prematurely, “the financial consequence is not primarily replacing a pump or a breaker or some power component — it’s the the value of that expensive compute capacity not generating revenue because it’s offline,” said Jason Hoffman, chief strategy officer at data-center builder and operator Switch.The cost of downtime in terms of lost revenue varies widely, with estimates ranging from thousands to hundreds of thousands of dollars per minute, depending on the type of facility and its workload.Data centers are built on the assumption that, once they are online, they will operate around the clock 365 days a year, said a person involved in the financing of such facilities. In reality, some are seeing uptime closer to 80%, and unless resolved this could hit investors in certain projects in the next 12 to 24 months, the person said. Any issues with reliability add to wider concerns about the returns generated from hundreds of billions of dollars of planned AI investments. The rate of depreciation of another crucial piece of equipment at data centers, the GPU racks themselves, has raised questions about whether the industry can be as profitable as it promises. These problems with reliability also have the potential to destabilize the wider power grid. This sprawling web of high voltage lines, transformers and power plants requires constant calibration, something that has become more challenging with each passing year due to aging equipment, rising demand and extreme weather.The expansion of intermittent wind and solar generation, which often result in big swings in supply from one hour to the next, are already a destabilizing force. AI data centers can put this volatility on steroids. “These loads are extremely dynamic or fluctuating, which causes grid instability and can lead to, if not corrected, potential blackouts or power outages,” said Sreemant Roy, a power-quality expert and global offer manager at Schneider Electric in Nashville, Tennessee. Of particular concern is a data center’s ability to cause sub-synchronous oscillations in the power flow, which can damage equipment connected to other parts of the network, he said. “That has made utility companies globally very worried,” Roy said. Within the last two years, the North American Electric Reliability Corp. — the top US regulatory body establishing standards aimed to keep the lights on — has repeatedly warned and issued alerts that data centers are one of the greatest risks to grid stability.NERC evaluated more than 33 gigawatts of operational data centers in the US an found about three quarters of their load models “are insufficient to represent data-center dynamic behavior,” according to a September report. Earlier this year, the agency issued a rare level-three alert requiring big data centers to address these immediate risks and submit their responses by Aug. 3.From top to bottom, the AI industry is aware of these issues and actively working on solutions. Nvidia started working more closely with power experts when it developed Blackwell GPUs, which were first released in 2024 and have become pervasive in data centers, said Dion Harris, senior director of hyperscale infrastructure solutions at the chipmaker. “We’re building the chips and processors” but also using them in the company’s own data centers, Harris said. Nvidia is working to make its deployments a lot smoother “both on the data-center build out, design and engineering phase, as well as on the power delivery.” Data-center users have deployed techniques to smooth out the power fluctuations of AI workloads by running side computations — essentially dummy math that isn’t part of the training process — to keep GPUs operating steadily. However, this approach has been criticized for wasting electricity at a time when power demand is surging. Last year, the National Laboratory of the Rockies near Denver, Colorado set up a test-bed on behalf of the US Department of Energy to figure out how to integrate AI safely onto the grid, said Martha Symko-Davies, the NLR’s program manager for the DOE’s office of electricity. The site has GPUs and power generation on site that developers can use to figure out if their setup can handle the variability of AI. One power supplier said they will use the facility to test batteries, software and other equipment intended to smooth out oscillations between the data center and grid that can cause damage on either side.“We have the opportunity right now to get it right,” said Symko-Davies.—With assistance from Dina Bass.Notice for the Postmedia NetworkThis website uses cookies to personalize your content (including ads), and allows us to analyze our traffic. Read more about cookies here. By continuing to use our site, you agree to our Terms of Use and Privacy Policy.