Why AI’s Next Infrastructure Bottleneck Is Between The GPUs

For the last few years, building more powerful AI infrastructure has often sounded fairly straightforward. Get access to more GPUs. Move to newer accelerators when they become available. Give them enough power and cooling to do their jobs. Then, at least in theory, you have more compute capacity to throw at larger models and heavier workloads.

The problem starts when those GPUs have to work together. A cluster containing thousands of very fast processors isn't useful just because each processor is fast on its own. They need to exchange information constantly, and sometimes they need to wait until other processors have finished communicating before they can continue working.

That changes what “more compute” actually means. Adding another accelerator can increase the amount of processing hardware installed in a data centre without increasing usable AI capacity by the same amount. Eventually, performance depends just as much on how efficiently those processors communicate as it does on how quickly they calculate.

We're already seeing infrastructure spending follow that shift. Dell'Oro Group reported in September 2026 that switch sales for AI back-end networks overtook traditional front-end data centre networks for the first time during the second quarter. It took the AI back-end networking market around three years to become larger than the long-established front-end market.

The race for faster accelerators isn't going away. But the infrastructure connecting those accelerators is becoming a much bigger part of whether all that expensive compute can actually be used.

Faster GPUs Don’t Automatically Create Faster AI Systems

A GPU working on its own can calculate extremely quickly. A large AI model creates a different problem because the work is divided across many GPUs. Those processors repeatedly need to exchange results, share information and synchronise before different parts of the workload can continue.

Think of it less like adding faster workers and more like adding people to the same project. If they can all work independently, adding more people helps. If every few minutes they have to stop, exchange work and wait for everybody else to catch up, communication starts determining how quickly the project moves.

Distributed AI has its own version of this. Operations such as AllReduce allow processors involved in training to combine information and distribute the result back across the cluster. The calculation happening inside each accelerator is only one part of the process. The information also needs to move between them quickly enough to keep the workload progressing.

Meta described this problem particularly clearly when it introduced its MetaRoCE networking technology in August 2026. Its AI infrastructure includes clusters with hundreds of thousands of GPUs across multiple data centres and regions. During collective operations involving thousands of accelerators, Meta says the slowest transfer can set the pace for the entire job.

This is why GPU utilisation is becoming such an important infrastructure measure. A processor can technically be available without spending all its time doing useful compute. Every communication delay creates another opportunity for expensive hardware to sit idle while it waits for data from somewhere else.

More network bandwidth won't solve every cause of low utilisation. Workload design, scheduling, software and memory all play a role. But once communication sits directly in the path of the workload, AI networking stops being something that simply connects the compute. It becomes part of how well the compute performs.

AI Networking Has Become Its Own Infrastructure Layer

Talking about “the network” can make this sound like one problem with one answer. In reality, different communication challenges appear depending on how far the data needs to travel and how many accelerators the system is trying to connect.

This is where the distinction between scale-up networking and scale-out networking becomes useful. Both are concerned with connecting AI compute, but they're solving different parts of the problem. And as clusters become larger, infrastructure teams increasingly need to think about both.

Scale-up connects the accelerators

Scale-up networking connects accelerators closely enough that they can behave more like parts of one large computing system. Nvidia's NVLink is one established approach, while the Ultra Accelerator Link Consortium is developing UALink as an open alternative for high-bandwidth, low-latency accelerator communication.

The UALink 200G 1.0 specification supports scale-up connections for as many as 1,024 accelerators inside an AI computing pod. The consortium is already extending the standard with capabilities such as in-network compute, where some communication-related processing can happen inside the interconnect rather than always returning to an accelerator.

That scale tells us something about where AI architecture is heading. The useful unit of compute is getting bigger. Instead of thinking only in terms of what one GPU or server can do, infrastructure design increasingly has to consider how hundreds or thousands of accelerators behave when they're effectively being asked to function as one system.

Scale-out connects the clusters

Scale-out networking takes the problem beyond those tightly connected accelerator groups. It links servers, racks and clusters, which means the AI network fabric has to maintain high throughput and low latency across a much larger environment.

InfiniBand has been widely used for this kind of high-performance computing, but Ethernet is expanding rapidly. Dell'Oro found that Ethernet represented around two-thirds of data centre switch sales in AI clusters during the first quarter of 2026. InfiniBand sales also more than tripled during the quarter, so this isn't a simple case of one replacing the other.

Ethernet itself is changing to support the workloads being put on it. The Ultra Ethernet Consortium, for example, is building an open communications architecture specifically for AI and high-performance computing. Its current specification, version 1.0.3, was released in July 2026.

MetaRoCE shows the same trend happening at the transport layer. Remote Direct Memory Access (RDMA) lets machines exchange data with less CPU involvement, which helps reduce communication overhead. Meta redesigned the transport around it because networking approaches built for smaller environments become harder to manage as accelerator counts and distances increase.

The important change isn't any single protocol. AI workloads are creating communication requirements significant enough to drive dedicated hardware, open standards and new transport technologies. The network around the GPUs is developing into an infrastructure layer in its own right.

The Electrical-To-Optical Boundary Is Moving

Once enormous amounts of information need to travel quickly between processors, distance becomes surprisingly important. Copper is extremely good at moving data over short connections. It's relatively efficient, reliable and inexpensive. The difficulty is maintaining those advantages as bandwidth rises and the connection has to travel further.

Microsoft Research describes this as a developing “networking wall”. Its research found that the copper links typically used for high-bandwidth GPU communication are generally restricted to distances below two metres. Optical fibre can reach much further, but traditional optical links bring their own trade-offs around power, cost and reliability.

So optical networking isn't replacing every copper connection inside AI data centres. The boundary is moving instead. Copper can continue doing the job where distance and bandwidth allow it, while optics move closer to the processors and switches as electrical connections become harder to scale efficiently.

That shift is already attracting serious commercial investment. Nvidia announced separate $2 billion investments in Coherent and Lumentum in March 2026, putting $4 billion into partnerships designed to expand optical technology, manufacturing capacity and research for future AI infrastructure.

Two developments in September make the direction even clearer. Qualcomm and Amazon are working together on optical connectivity reaching up to 1.6T for AWS data centre networks. Meanwhile, STMicroelectronics expects around 80 per cent of its projected AI data centre revenue of more than $2 billion in 2027 to come from chips used in fibre-optic data links.

Co-packaged optics takes the idea another step by moving optical components closer to the switching or processing silicon. A peer-reviewed paper published in Nature Electronics in August describes electrical interconnect limits around bandwidth, latency and energy efficiency as a growing constraint, while also highlighting unresolved challenges including thermal management, manufacturing and standardisation.

So there isn't a neat point where AI data centres simply “switch to fibre”. Infrastructure architects are balancing distance, bandwidth, power consumption, reliability, cost and serviceability. As cluster performance rises, though, the distances at which electrical links remain the best answer may continue getting shorter.

Network Architecture Is Becoming Part Of Compute Capacity

This brings us to a slightly uncomfortable problem with how AI capacity is often discussed. Saying an organisation has 10,000 GPUs tells us how much accelerator hardware it has installed. It doesn't tell us how effectively those 10,000 GPUs can work together.

There is a useful distinction here between installed GPU capacity and usable AI capacity. Installed capacity counts processors. Usable capacity reflects how much productive compute the wider system can deliver once communication, topology, software and other infrastructure constraints are taken into account.

Recent research shows how significant that communication problem could eventually become. A Nature Communications paper published in August tested an early-stage photonic approach to collective GPU communication. Its predictive modelling found more than 50 per cent lower energy per AllReduce operation and over 100 times lower latency at frontier scale than the switched-fabric comparison used by the researchers.

Those numbers shouldn't be read as something enterprises can buy today. The work includes an early proof of concept and predictive modelling, not production benchmarking. What's more important is the problem researchers are trying to solve: conventional ways of coordinating increasingly large GPU clusters can themselves become a limit on scaling.

Once that happens, network architecture affects the economics of compute as well as performance. Improving communication doesn't add another GPU to the inventory. It can help the organisation get more productive work from the GPUs already sitting inside it.

AI Infrastructure May Need To Scale Across Data Centres

The communication problem becomes harder again when all the compute can't fit comfortably in one place. Power availability, facility capacity and other physical constraints can make building ever-larger clusters inside a single data centre difficult, which creates interest in distributing AI workloads across multiple sites.

This is often described as scale-across networking. The distances involved are much greater than scale-up or scale-out environments, so latency and wide-area bandwidth become part of the compute problem. Connections that are perfectly reasonable between racks can behave very differently when the next group of accelerators is kilometres away.

Microsoft researchers explored this problem with Atlas and BubbleTea, two experimental systems for language-model training across multiple data centres. Together, they achieved up to 17 times faster training than the comparison architectures used by the researchers and GPU utilisation as high as 94 per cent.

These aren't benchmarks enterprises should plug directly into a business case. They show what researchers are already trying to optimise as AI workloads spread geographically: bandwidth allocation, idle GPU time and the relationship between network conditions and useful processing.

The optical market is preparing for the same possibility. LightCounting's July 2026 data centre optics research now forecasts coherent transceivers used for networks connecting cloud data centres, including scale-across deployments, while tracking technologies reaching 1.6T DWDM and 3.2T Ethernet.

That doesn't mean every enterprise AI cluster is about to span several data centres. But it does extend the infrastructure question. Once compute resources become geographically distributed, the network between facilities starts influencing what those resources can realistically do together.

GPU Procurement Can’t Stop At The GPU Anymore

This changes the questions infrastructure teams need to ask when evaluating AI data centre networking. Accelerator count and benchmark performance are still important, but neither tells you whether the surrounding architecture can support the workloads the organisation actually intends to run.

There also isn't one ideal network design waiting to be copied. Training, inference and mixed workloads communicate differently. A small deployment has different constraints from a huge cluster, while an architecture designed to remain inside one facility doesn't need the same connectivity as one expected to expand across several.

A more useful infrastructure assessment starts with the system rather than the individual component. Before treating additional accelerators as additional capacity, teams can look at:

• Workload communication requirements: How much data needs to move between accelerators, and how often do processors need to synchronise?
• Scale requirements: What needs to communicate within a pod, across racks and clusters, and potentially between data centres?
• Bandwidth and latency: Can the network keep pace as accelerator performance and cluster size increase?
• Topology and utilisation: Does the architecture help GPUs stay productive, or are communication patterns creating avoidable idle time?
• Power and cooling: What does the networking layer add to the facility's power and thermal requirements?
• Reliability and serviceability: What happens when links, switches or optical components fail, and how easily can they be replaced?
• Interoperability and upgrades: Can the infrastructure evolve without locking every future accelerator or network decision into today's architecture?

Those questions don't turn network planning into GPU procurement. They do the opposite. They recognise that AI infrastructure planning increasingly has to happen across compute and networking together, because neither side can be evaluated properly without understanding what the other is expected to do.

This also changes where bottlenecks should be investigated. If accelerator utilisation is lower than expected, buying more processors may simply give the existing communication problem more hardware to affect. Sometimes the more valuable capacity investment could be somewhere between the GPUs rather than inside them.

Final Thoughts: AI Compute Is Becoming A Networked System

Buying AI compute used to look mostly like a processor decision. As clusters have grown, the picture has become more complicated. Faster accelerators create more communication. More communication creates new demands around interconnects, switching, topology and distance. Eventually, those demands start influencing how much of the installed compute can actually be used.

The industry response is already visible. New scale-up standards are appearing. Ethernet is being redesigned for AI. Optical connectivity is attracting billions in investment. Researchers are exploring entirely different ways of coordinating accelerators, while infrastructure teams are beginning to consider what happens when the available compute has to extend beyond one facility.

None of this makes GPU performance less important. It changes what GPU performance means once the processor becomes one part of a much larger machine. AI compute architecture is increasingly the architecture of the whole connected system, not simply the specifications of the accelerator sitting at its centre.

As GPUs continue getting faster, the infrastructure between them is unlikely to become less important. The organisations able to turn future accelerator investments into useful capacity will be the ones looking beyond how much compute they can install and asking how efficiently all of it can work together.

That wider relationship between compute, networking, power and operations is becoming one of the defining infrastructure questions around enterprise AI. It's also one EM360Tech will continue exploring as the architecture behind AI becomes every bit as important as the models running on top of it.