Private AI Needs Infrastructure Built to Evolve, Not Just GPUs

63% of companies surveyed are already using an open AI model, according to Linux Foundation research — and that single number explains why enterprise AI infrastructure strategy is being rewritten right now. When open-weight models became good enough for everyday work, the value of an enterprise AI application shifted to the company's own data rather than the model behind it. Once companies see it that way, running the model close to their data and under their own control stops being exotic and starts being the obvious move.

That's the argument laid out in a new thought-leadership analysis making the rounds among infrastructure planners. Its core claim: private AI — running, tuning and governing models inside your own environment — is winning ground for two reasons. Regulated businesses like banks and pharma firms were never going to put proprietary research or customer records in someone else's tenant. And for everyone else, the push comes from the inference bill at scale, plus a reluctance to depend on a single provider for something now core to how the business runs.

The piece is an industry argument rather than a company announcement, but the stakes it describes are concrete. Enterprises that size their private AI infrastructure wrong will own capacity that doesn't fit the workload they actually end up with — and retrofitting existing facilities is expensive or simply impossible, because some were never built for this density. The takeaway is blunt: flexibility has to be designed in up front, because the technology is changing faster than anyone can pour concrete.

What does 'build the base, rent the spike' mean?

The analysis pushes a capacity model that splits AI demand in two. You don't need the biggest model for every job — summarizing documents, classifying tickets and answering questions against your own knowledge base are all tasks an open-weight model can run privately, at a cost you can forecast. Frontier models get saved for the work that genuinely needs them.

That split dictates infrastructure shape:

• Build private capacity for the steady demand you can see coming every week.
• Rent outside capacity for spikes and heavy jobs that exceed what you own.
• Make both behave like one system — same data access, same controls, no pipeline rebuild when a workload moves between them.

This is how companies have always run compute, the piece argues. AI doesn't change the logic; it just raises the density of the base and widens how far the peak can swing. The hard part is sizing the base accurately while keeping the private and rented environments unified.

Where does the infrastructure actually go?

Buying the GPUs turns out to be the easy decision. Then you have to put them somewhere, power them, cool them, and connect them to the rest of the estate. High-density AI hardware doesn't drop into a standard enterprise data center without significant changes to power and cooling systems, and it doesn't belong in every site a company runs.

The numbers back this up. In Uptime Institute's 2025 AI Infrastructure Survey, 37% of respondents identified power availability as one of the main factors influencing where they run AI inference workloads. Power, not compute, is increasingly the deciding variable.

Most companies land on a mix of colocation and cloud, and data usually decides it. Enterprise data is already spread across several places, it's expensive to move, and in regulated industries some of it legally cannot leave a country or a particular environment. Consolidating everything into one building next to the GPUs "sounds clean and almost never happens," the analysis notes. The real question is how compute reaches the data it needs securely, without piling on cost and complexity you'll regret in three years.

Why is this 'the third exchange'?

The piece frames AI as the third great infrastructure exchange in computing history. The first was the internet, and connectivity solved it. The second was cloud, and fabric solved it — software-defined interconnection that let businesses reach different providers from a single location. AI is the third, and fabric alone doesn't solve it, because AI also needs a great deal of power in the same place the fabric is.

The companies that get private AI right, the argument goes, will be the ones that put compute where power and connectivity meet, instead of running those as two separate procurement exercises. That's a planning shift as much as a technology one.

How does agentic AI change the connectivity problem?

Agentic AI makes all of this more urgent. A conventional application takes a prompt, calls a model, hands back an answer. An agent is in motion the whole time: it calls several models, queries multiple systems, pulls data from wherever it lives, and hands work off to other agents. Every hop crosses a boundary, adds latency, and raises the question of who is allowed to access what.

Now put the model in one location, the data in another, and a rule that some data must stay inside a particular country. Companies need real control over those paths — which systems can reach which data, what happens when a workload moves, how a new model gets wired in without disturbing everything already running.

The answer, per the analysis, is programmable infrastructure: connections between models, data and environments should be something you configure and change, not an engineering project each time. Standards like the Model Context Protocol (MCP) point the way, giving an agent a common way to talk to the tools and systems around it. An agent needing a connection to a new data source could request it the same way it would call any other service, with policy and controls still in place. New models ship every few months, and if every change means a rebuild, "you've lost the flexibility you went private to get."

What are the economics of private AI?

The piece puts forward a plain unit of measurement: power to tokens. Companies doing private AI well understand the relationship between the power they put in and the tokens they get out — and with that understanding, they can size the environment with real confidence. The second piece is headroom. New capabilities arrive every quarter, and good planners leave margin above today's exact requirement rather than building to its edge.

The full bill is bigger than the hardware:

• Housing, cooling, powering and connecting GPUs can match the hardware spend.
• Operating a high-density compute environment is a skill set "not something you hire in a quarter."
• Idle hardware is the penalty for overbuilding — hence base-and-spike again.

What's the timing trap?

Underneath everything sits a mismatch: a data center takes about two years to build, and in two years the models, hardware and applications enterprises want to run will look considerably different. Whatever you commit to now will run workloads nobody can fully describe yet.

Companies are still figuring out which workloads belong privately and when they need outside capacity — a reasonable place to be, the analysis says. The thing to avoid is making infrastructure commitments that take that decision away from you before you've had the chance to make it.

The closing judgment frames the long game: whether private AI pays off will depend less on which model you choose and more on whether your infrastructure can keep up as your requirements get clearer. Build for today's workload, keep the connections programmable, and put the compute where fabric meets power.