Why does the loss compound? Conduction loss in copper scales as I², not linearly. Triple the current, ninefold the loss.

Let’s pencil it out. Vcore (the compute logic supply voltage) is locked at 0.6 to 0.8 V by transistor physics; we’ll use 0.7 V. Given that P = VI, we can estimate the current draw for a few Nvidia GPUs:

Plug the currents into I². Hopper to Blackwell is ~3x the loss (1,714² / 1,000² = 2.94). Hopper to Vera Rubin is ~11x.

And conduction loss is only half the story. Transient voltage droop, which gets harder as workload transients steepen, adds a second loss term on top.

The only way out is to shorten the high-current portion of the path.

Power delivery is splitting into two domains. The rack-to-board step (48 V to 12 V) stays where it is, because at those higher voltages, the current is still low enough (tens to a few hundred amps) for standard motherboard copper to handle without overheating. The board-to-die step (12 V to ~0.7 V) is where current explodes past 3,000A for a Vera Rubin, and that’s the step that has to migrate from the motherboard onto the package substrate, and eventually under the die itself.

Intel said as much at ISSCC 2026 in February:

Last week, Analog Devices (ADI) agreed to pay $1.5 billion in cash to acquire Empower Semiconductor. Empower ships kilowatt-class power delivery as SiP modules that mount on the package substrate next to the compute die. Short lateral paths, in-package magnetics, but the active silicon still sits beside the SoC rather than embedded inside the substrate. A real step in the right direction, and ADI just paid $1.5 B for it.

Twenty years of IVR attempts have left one architectural step still open beyond Empower: a merchant, on-package, in-substrate, monolithic-magnetics IVR chiplet. A startup called PowerLattice is aiming at exactly that slot.

In this post, we will look at:

• How AI accelerators lose power before they compute. The two loss mechanisms worked from first principles.

• Intel’s published ISSCC 2026 loss budget for a 5 kW SoC.

• Why ADI just paid $1.5 billion, and the impact of Nvidia bumping Vera Rubin from 1.8 kW to 2.3 kW.

For paid subscribers:

• What Intel and Empower build today

• How PowerLattice’s architecture goes further

• Why transient response matters a lot

• Which power-IC incumbents lose their AI socket if the architecture lands, which are partially protected, and which benefit either way

• What about Nvidia?

How AI Accelerators Lose Power Before They Compute

To keep current manageable, data centers step voltage down in a cascade so the highest-current section is the shortest:

Yikes! That last step, 12 V → 0.7 V at the Motherboard Voltage Regulator (MBVR), has crazy current flowing through it! In current designs, the MBVR sits outside the compute die, either on the PCB just outside the package, or on the package substrate flanking the SoC. Either way, thousands of amps travel horizontally through copper traces to reach the compute die. That lateral path is the Power Delivery Network (PDN), and every millimeter of it loses power two ways:

1) I²R conduction

P = I² × R. Focus on current squared.

Say you have a 200 microohm section of the PDN carrying 1,000 A. That dissipates (1,000)² × 0.0002 = 200 W.

Next generation you roughly triple the current. Same path, same resistance: (3,000)² × 0.0002 = 1,800 W. Nine times the loss!

2) transient voltage droop

AI workloads jump from idle to full power in tens of nanoseconds. The MBVR sits centimeters of high-current lateral copper away from the die. Inductance in that lateral path means voltage at the die sags before the MBVR can respond. If the sag dips below the logic minimum, errors are introduced.

To avoid this, designers add a guard band. So the logic might need 0.75 V, but the MBVR supplies 0.95 V (200 mV of margin):

Because dynamic power scales as V², that means we need say ~1.6x dynamic power to avoid droop problems. (0.95² / 0.75² ~ 1.6x)

The ideal fix is move the regulator from centimeters of lateral substrate copper to micrometers of vertical pillar directly under the die.

Putting the IVR (Integrated Voltage Regulator) directly under the load collapses both losses at once. I²R drops because the high-current path shortens by orders of magnitude. Droop drops because proximity allows much smaller, faster inductors and capacitors, so the regulator responds much quicker

Intel’s ISSCC 2026 Chart Puts Numbers on the Problem

At ISSCC 2026 in February, Intel’s Kaladhar Radhakrishnan presented “Integrated Voltage Regulator Solutions to Enable 5 kW GPUs.” Check out this slide that shows the waste that happens to a conventional MBVR architecture as GPU power scales from today toward Intel’s end-of-decade forecast of 5 kW per chip:

At 1 kW per GPU (Blackwell B200 territory), today’s MBVR architecture is fine. The system draws 1.25 kW from the wall to deliver 826 W of useful compute, for 66% efficiency. I²R loss is 89 W. Droop waste is 174 W.

But at 5 kW per GPU, the architecture struggles. The system now pulls 8.3 kW to deliver just 3.5 kW of useful compute. Efficiency has fallen to 42%. I²R loss has grown from 89 W to 2.2 kW! That’s a 25x jump from only 5x more current, because the loss scales as I².

Droop waste has grown from 174 W to 1.5 kW too.

Roughly half the system input is now burned as heat, not as useful compute.

Intel’s has ideas for alternatives, for example an in-package landside IVR. Landside meaning mounted on the bottom of the package, opposite the compute die, where the BGA balls connect to the PCB.

Intel calls one version a C2VR (Continuous Capacitive Voltage Regulator).

Applied to the same 5 kW SoC, the losses are so much smaller:

I²R loss on the input path is reduced from 2.2 kW to 476 W. System input falls from 8.3 kW to 6.8 kW. Useful compute rises from 3.5 kW to 4.1 kW.

Same compute job done with 1.5 kW less wall-socket power!

This is a $1.5B problem

Just last week, ADI announced an all-cash $1.5 billion acquisition of Empower Semiconductor, a maker of silicon capacitors and the Crescendo kilowatt-class vertical power delivery platform. It’s built as System-in-Package (SiP) modules (multiple silicon dies bundled into one package) with integrated magnetics that mount on the package and scale to 2,600 A+ peak current:

My initial take is “Nice! A $1.5B price tag! IVR is an important problem space!”

Of course, Nvidia faces the same problematic physics. Vera Rubin was originally specified at 1.8 kW per chip. In late 2025, SemiAnalysis reported

Supply chain rumors have indicated that there are 2 different “SKUs” with different power and performance profiles: a Max-P variant at 2,300W and a Max-Q variant at 1,800W. However, these are not distinct hardware SKUs but the 2 default power profiles that Nvidia is offering users based on their workload needs. Max-Q is what Nvidia believes offers the best performance per Watt. Max-P offers the greatest absolute performance though this would come with an efficiency penalty. Running the Max-P setting results in a 20% increase in rack power draw but the performance gain fall well short of this 20% power consumption increase.

As we discussed, there are tradeoffs with the 2300W TDP. More watts at fixed Vcore means more amps. More amps mean more I²R and more droop. Which means they really need on-package IVR!

PowerLattice

In November 2025, a startup called PowerLattice emerged from stealth with a $25 million Series A jointly led by Playground Global (where Pat Gelsinger is now a General Partner) and Celesta Capital. The three founders came out of the Qualcomm/NUVIA group.

Peng has a long history of power delivery design, including a 12-year stint at Intel with many patents awarded. Per Pat Gelsinger, it’s a dream team:

“There are very few teams and people that can do it,” said Pat Gelsinger, general partner at Playground Global. “We have assembled what I’d argue is the dream team of power delivery.”

PowerLattice’s innovation is the Rainier micro-IVR, a monolithic, vertical-design silicon die combining proprietary on-die magnetic inductors, advanced control circuits, and a programmable software layer. The chiplet brings voltage regulation from inches away on the motherboard to within hundreds of micrometers of the compute die, eliminating most of the lateral substrate copper that bleeds power to I²R and droop.

The architecture scales by ganging multiple chiplets in parallel, each at a 5 A/mm² current density, with a low-hundreds-of-micrometers z-height. Z-height is thin enough for land-side mounting, substrate embedding, or interposer embedding.

PowerLattice claims >50% reduction in effective compute power, an order-of-magnitude lower power noise, lower cooling, longer processor lifetime, and 2× or more performance per watt where AI compute is data-center-power-constrained.

First chiplets are being produced at TSMC; customer testing planned for H1 2026. Which should be roughly now.

Can PowerLattice compete? What does this mean for ADI+Empower? And other public power-delivery semi companies?

For paid subscribers: Intel and Empower architectures, how the PowerLattice architecture goes beyond, and what it means for publicly traded power-delivery incumbents.

Read more

Deepu’s team builds the platform that essentially every robotics company on the planet uses across the three computers that physical AI requires: training in the data center (GB300, Vera Rubin), simulation (RTX Pro 6000, Omniverse), and the runtime at the edge (Jetson Thor and Orin). Roughly 2.5 million developers and more than 10,000 companies are building on Jetson today, but the industry is still shipping only a million or two robots a year against an opportunity Deepu pegs at tens of billions.

Subscribe now

In this interview, we walk through physical AI from first principles. A few things that surprised me:

• Agentic AI isn’t just a digital (agent) story. The industrial business of the future runs on fleets of robots of different embodiments, people, and digital AIs — all needing orchestration. You don’t validate that orchestration on a live manufacturing line, which is why Nvidia built Mega: a blueprint for simulating an entire factory’s worth of agents in a digital twin before you touch the real one

• The industry has marched from VLMs to VLAs to world models in the past few years. World models matter because when a robot moves an atom, the rest of the world reacts, and you need to model that reaction, not just the action. “Necessary but not sufficient” is the new consensus

• Edge robots won’t run in isolation. Deepu expects hybrid edge-cloud as the default, where the robot does as much as possible locally but can “phone a friend” to the cloud for long reasoning. You never have enough compute at the edge

• Every robotics application has a “10-second mark” — the qualifying time before you’re even in the competition. Self-driving cars have hit theirs in the last six to twelve months thanks to two unlocks: end-to-end models replacing stitched-together specialist models, and reasoning that lets a system handle scenarios it never saw in training

• Most of the spend at robotics startups today is not on edge deployment, it’s on training and simulation. Until accuracy is solved, there’s no point scaling deployment, so the action sits in the first two computers

• Simulation finally works for robotics because the sim-to-real gap has closed enough to be useful. Nvidia open-sourced Newton — a physics engine built with Disney Research and Google DeepMind specifically for robotics — to push that further

We also cover Nvidia’s new data-center-vs-edge reporting structure and why Deepu thinks the next manipulation and locomotion tasks will hit their 10-second mark in the next year or two.

This interview is lightly edited for clarity.

Why Robotics Is Suddenly Possible

Hello listeners, today we have a special guest, Deepu Talla, VP and GM of Robotics and Edge AI at Nvidia. Welcome, Deepu.

DT: Austin, hi, good morning.

Good morning, thanks for coming. I’m super excited to talk to you today about physical AI and robotics and all things edge AI. A lot has changed in the past two years or so. Listeners both understand that physical AI is here, physical AI is coming. Jensen has said the ChatGPT moment for physical AI is here and coming. On the other hand, most people aren’t experiencing it in their day-to-day life. And so there’s differences from the data center portfolio to the edge portfolio. I look forward to unpacking it with you just to help everyone have a better intuition, understanding, pulse of what’s actually happening out there.

DT: Yeah, absolutely, sure thing.

So I teed up a list of questions, so we’ll jump right into these. My first question for you, set the stage for us. A lot has changed obviously with the rise of LLMs, which everyone knows, but more specifically at the edge with VLAs, vision language action models. Could you talk to us about why is robotics suddenly possible? What are the technical changes that have happened that make it so that all of a sudden there’s all these humanoid startup companies and self-driving taxis are suddenly really good? What has changed?

DT: If you think about the opportunity itself, we’ve known it for, gee, what, 50 years, growing up watching Star Wars and Star Trek and the need for robotics, physical AI has always existed. Whether it’s for doing dangerous jobs or labor shortage and so on. But the technology has not been good enough. There’s fundamentally two things that you need for bringing physical AI and robotics into the real world.

The first one, of course, is the model or the algorithm or the technology needs to be accurate enough. Intelligent enough. If it’s not able to do the job well, accurately, then what’s the point? In the physical world, because there’s no human to back it up, the accuracy requirements are extremely high compared to the digital world. For example, if you’re using ChatGPT or Claude to summarize an email for you or compose an email for you, it does a pretty good job. It’s getting better and better, but you will go off in the last one or two percent. You’re going to tweak it and make it right and ship it. But in the physical world, that can happen. If you want a robot to do some manipulation tasks and finish some things, humans are not going to be backing it up. So the accuracy requirements are 99 point — how many nines after the 99 point? Depending on the application, a self-driving car is probably somewhere between eight and 10 nines of accuracy that you need. A surgical robot, surely we would want to be much more than that. If it’s a consumer robot in the home, maybe four or five nines might be okay. So that’s the first thing. That’s why it’s been super hard. Technology has not been good enough to make it accurate, number one.

Number two, let’s say you created a super accurate robot. Now that needs to get integrated into the physical world. In many times what happens is that robot is working with other robots or humans or other processes that are happening in the real world. For example, in a manufacturing setting, there’s many other things that are happening. It needs to integrate very well into that existing workflow or system. It’s kind of like you hire an engineer or an employee into your company. They’re brilliant. That’s why you hired them. But they also need to be equally good at integrating with the rest of your employees so that they can be more productive.

And that is actually a pretty big problem we haven’t been able to solve, because typically what happens is when you integrate a robot or autonomous operation into your existing workflow, you have ERP systems, you have warehouse management systems, you have security systems, you have PLCs, programming logic controllers for these robots, many of which could be 10, 20 years old on different software. And it’s super hard for humans to build that glue logic, if you will, to bring all those things together. Now, luckily, in the last three months with the rise of agentic AI and coding basically becoming easier and easier for these agents to solve, it brings us great hope that once you solve the accuracy, the integration piece is also going to be solved reasonably well.

So what I hear you saying is, for LLMs when humans were using them to start, the human’s in the loop. Maybe you don’t have to have quite as high of a need for accuracy. Even with some of the integration pains, the human could copy from here and paste over there. But it sounds like a lot of physical deployments, maybe you don’t have a human in the loop, so you do need better accuracy or more intelligence, but then also yes, there’s an integration challenge of how do you actually make this useful in the workplace or in an industrial setting or something, as opposed to just the toy chatbot stuff that we did early with LLMs.

DT: Yes, right. The analogy that I use with my team in general is imagine you are a 100 meter racer. Your goal ultimately would be to win the Olympic gold. But before you win the Olympic gold, you need to qualify for the Olympics and you need to hit a certain time. In the case of men, it’s roughly 10 seconds. It’s incredibly hard to hit 10 seconds. But unless you hit 10 seconds, doesn’t matter. You’re not going to qualify for the Olympics and you can’t get there. And of course, in order to win the Olympics, you probably have to hit 9.7, 9.6, eventually. But 10 seconds is the golden mark.

So for each application, I believe there is a 10-second equivalent. Until you hit that, you’re not in the game. You keep trying. And if you look at all the physical AI and robotics applications, almost in every case, we haven’t hit the 10-second mark. I believe we have recently hit the 10-second mark in autonomous vehicles. You’ve got to keep asking yourself, how is it that suddenly in the last six months to a year, there are so many Waymos out there, suddenly Tesla self-driving has hit that 10-second mark, if you will. Now that doesn’t mean the 10-second mark is good enough. It’s just that it puts you in the game. Now it’s all about scaling to hit that 9.7 and really go off. So you’ve got to ask yourself what changed? We’ve been trying this for 10, 15 years, but suddenly something happened to hit that 10-second mark.

I think there’s two things that really happened for self-driving cars. Number one, end-to-end models. Until very recently, until a couple of years ago, since 2015 to 2023-ish, it was all about building specialist models for whether it’s lane detection, whether it’s path planning, whether it’s for sign detection, whether it’s for all those kinds of models. You would have these 20 different so-called specialist models, and you’d put them together, and they would kind of work. They would be brittle because you would never be able to solve the long tail problem. They’re not quite the 10-second mark. They’re probably the 10.5-second mark, which is great, but not good enough. So end-to-end models is one.

And then secondly, what we’ve seen in the digital AI world in the last one year is reasoning has become extremely important, kind of like humans today. How is it possible that some 16-year-old who gets a license, you can practice for 10 hours and you are on the road, and you’re driving similar to somebody who’s been like 30 years experience, with millions of miles experience potentially on the road? Because we do reasoning. We haven’t encountered all the scenarios in our training dataset, but we are able to have some intelligence and then we reason about it and then we act appropriately. So because of that, end-to-end models and reasoning, self-driving cars hit the 10-second mark.

Now, then you expand it into what are the robotics applications similar to that that we can get there. You go to the ultimate application, the extreme right goalpost — humanoid robotics, let’s call it general purpose with fine-grained dexterous manipulation with so many degrees of freedom. You can navigate anywhere. You can manipulate any object from rigid bodies, which is easier, to soft bodies and fluids which requires extreme physics simulation, and you need to do all that analysis. Those are the increasingly hard problems that we need to solve and technology is evolving to get there.

But the left goalpost, if you think, is the self-driving car. We’ve gotten good enough. You’ve got to ask yourself, what is the next one that feels like we are reasonably good enough technology between end-to-end models, between reasoning and simulation, of course, because you have to test it in simulation. It’s too dangerous and too expensive and too slow to test it in the real world. What are the applications? You can start to feel like you can start to see that a lot of off-road delivery robots, you can see things like autonomous mobile robots and industrial environments. You’re starting to see this kind of getting deployed.

And then the next wave you can think of like video analytics as an application, which is cameras and outside-in. We think of them as robots. Typically when you say robot, most people think of a robot like a human, a humanoid or an AMR — sensors and actuation are on the robot, and we do perception inside out. Because we have sensors on us, we look inside out and then we process it and then actuate. But there’s also an outside-in robot, kind of like a traffic controller. Kind of like, GPS in your car somehow tells you even though you don’t know what the route is 500 meters away or 500 yards away, what the traffic looks like. That’s coming because of outside-in from other agents that are being spatio-temporally analyzed and you’re combining all of that information. If you did that, you can solve all of the safety applications. You can do situational awareness using cameras and other sensors in a building or a factory or a city and so on. So that’s how we are seeing it.

And it’s amazing right now the pace at which these models are evolving — what started with language three and a half years ago, moving to vision, vision language models, to multimodal, add reasoning on top of it, went to vision language action models, and now we’re seeing world models. Because an action model takes an action and does some manipulation, but when you take an action and do some manipulation in the real world, the real world is moving. Atoms are getting moved, something is getting changed. And then the world reacts appropriately because of the change, and you need to be able to model that. That’s where the world models are coming in. I’m sorry, long answer, but I’m just super excited as you can tell how fast this technology is evolving.

From Specialist Models to World Models

No, this is really, really helpful. I heard a couple key unlocks that have happened recently. You talked about simulation and world models, which we can touch on, but you also talked about end-to-end and reasoning. Let’s dive into the simulation and world models. Tell the listeners a little bit more about that, because I don’t think a lot of people have spent a ton of time thinking about these. What is happening in that space over the past two years where that’s also enabled a key unlock, for example, for the sort of easier or the first thing that we’re all experiencing, which is the self-driving cars. What’s different from 2020 or 2023 and today in the simulation world model space?

DT: Until ChatGPT happened in November 2022, the technology we had mostly was convolutional neural networks and transformers, of course, but we were building so-called specialist models for a specific task. They would kind of work, but the world, especially in physical AI robotics, because the accuracy requirement is so high and also it’s very hard to maintain the world in a structured manner. If you maintain the world in a structured manner, you exactly define each component is arriving at a certain time and you program that robot and put a model, you kind of solve it. But that’s really solving 1% of the real opportunity. That’s where technology was.

When ChatGPT came about, it fundamentally transformed from so-called a specialist doing a very narrow task to a reasonably good enough general purpose model. In the case of ChatGPT, it was trained on everything that we had on the internet using language and so on. It could do many jobs because it was a good generalist. It’s kind of like, you could potentially train a 10-year-old human to do some very narrow task and they’d be special and they could do it. However, they’re not very good generalists because they cannot do much more.

In humanity, we define a good generalist as somebody like getting an undergrad degree, for example. So a 21-year-old. That’s a reasonably good enough generalist because they have knowledge on multiple things. And then what happens is if you want to solve really important problems after that, you take that reasonably good enough general purpose brain and then you derive a specialist from them. It’s kind of like you hire an employee at 21 years old, very good generalist, but for the next 30, 50 years, they’re going to train in a specialty using the general purpose capability, not losing the general purpose capability, but becoming increasingly specialized in something. That’s when you can solve really difficult problems.

So until 2023, the technology was, before that, so-called specialist models, you would gang multiple of them together. You could solve some problems, but 1% of the opportunity and very brittle if you take it to anything else. Once ChatGPT moment happened, applied to language, what the physical AI robotics folks realized is, okay, hold on a second. We can use that same technology and use multimodal, because in the case of ChatGPT it started with language, but in the case of physical world, vision is one of our most important sensors. And of course there’s sound and then there’s touch and then all the other things, but vision is one of the biggest sensors that we use. So can we add video camera, of course you can add radar, you can add lidar, you can add ultrasonic, you can add speech, in addition to text as language as the modality? That’s what researchers started doing for robotics.

What came out of it in 2024 was so-called vision language models. And then they said, okay, you can analyze, understand the scene using computer vision. But what’s a robot if you don’t take action? Ultimately, you can analyze everything that you want, but you have to take some action. So they said, okay, fine, we have vision language model that’s understanding the scenario, but we need to take action. That’s when VLAs came about — vision language action model. You combine language, combine vision, you analyze it and then you take action. And that unlocked quite a few number of use cases in the last 12 months, especially use cases as it relates to relatively structured world and doing some sort of manipulation with rigid bodies. For a rigid body — if you look at this thing for example, it’s a rigid body and I can hold it this way, I can hold it this way. It’s not that complicated because I can apply a force or torque that’s between 1X and 10X and it kind of works okay. But when it’s a soft body, you can do it of course, because if it’s deformable, it squishes or it breaks and so on.

So today with vision language models, action and VLMs, we are able to solve those types of use cases which are relatively structured, but rigid body. So that’s where we are today. But then the realization is, well, that’s good, but still not good enough. Because even solving rigid bodies, you probably solve from 1% of the problems to 2 to 5% of the problems, let’s say. We need to get to 100. We want to solve general purpose robotic problems so that we can really expand.

So this is where world models come in. Because when I picked this bottle and when I moved it somewhere here, physics changed. The atoms got moved. The robot did it, but the rest of something changed in the world too. So you need to model that. What’s happening in the world because of this actuation needs to be modeled. If I place this bottle on a table here right beside me, but if I placed it at the edge of the table, it can fall and something happened as a result of that. So all of that scenario also needs to be modeled. This is why the industry believes — if you look at all the researchers right now, they started with language, went to VLM, went to VLA and now they’re all saying, necessary but not sufficient. We need to add a world model.

So now you hear these things called world foundation models. You see things like world action models, that’s one of the latest terms that you hear about. Which is essentially combining, in addition to the VLM, VLA, you add simulating the environment around you. In order to truly solve this problem, you need to install what you’re doing in the robot, but you also need to understand what’s happening in the world because it’s always going to be interactive. That’s where we are today.

The Three Computer Architecture

That was a great little history lesson that got us to today. So my question is, given that we’ve moved from specialist models to these generalist models that are now multi-modal, they can take action, ultimately to world models that can represent not only the action you’re taking in the world, but what’s happening in the world around you. What does that mean for edge computing? Does that mean we need way more memory, way more compute, more memory bandwidth, impacting your portfolio and your roadmap?

DT: If you think about robotics or for that matter edge AI, there’s fundamentally four steps. If you walk backwards from the last to the first, the last step, of course, is the deployment. That’s the runtime. In the case of a robot, the robot is operating at the edge, at the point of action, like a car or a humanoid robot. There are sensors and actuation. So you need a computer there. And because for latency reasons, for cost reasons, for connectivity availability reasons, safety and all of that decision-making, especially for physical AI robots, you want to do as much as possible at the point of action where there is sensing and where there’s actuation. So that’s the edge computer. And there’s a lot of work that we’ve been working on making that happen.

But before you do the deployment, the third step is you’ve got to test it. Until you’re sure that it’s good. And then the best place to test it is in simulation because it’s faster, safer, cheaper. That’s the third step.

Before you test it, you’ve got to train it. And this training is no different than training large language models. It’s typically done in a data center. That’s the second step, is training.

And the first step before you train is you need to have data. And data, unlike ChatGPT, large language models where there’s a corpus of everything that humanity has created in the last 50, 100, 200 years is reasonably well represented, it’s not well represented in the robotics world, especially when you’re talking about — you can see YouTube videos of dances, but you don’t see YouTube videos of extreme fine-grain, precise manufacturing tasks and so on. And even if you see that, there’s no physics modeled in that. You kind of see how it’s being done, but you don’t know what’s the force, what’s the torque, what’s the angle, what’s the best way? The trajectory planning? You don’t see any of that. So that’s the problem.

So we’re working on all of these steps right now because we at Nvidia, we don’t build robots. We’re building a technology platform that helps everybody building robots. We provide the core infrastructure and provide the acceleration libraries and workflows for data generation, training number two, testing and policy evaluation and simulation number three, and the last step is the edge computing deployment.

So your question of what does it mean for the edge computer? In order for robotics to really take off and scale, first you have to solve the accuracy problem and the integration problem. And today much of the action is in solving the accuracy problem. Until you solve the accuracy problem, there’s no scale out happening in the edge and deployments. That’s where we are today. Now let’s imagine we’ve solved the accuracy problem and the integration problem is going to be solved increasingly with agentic AI. Now you essentially come to the edge computer and you need to make it scale out from whatever, hundreds of thousands of robots today, maybe a million robots, to ultimately, if the vision comes true that there should be multiple robots per human like a C-3PO and R2-D2, kids will grow up with a robot and the robot keeps changing over time and the memories will stay forever. Billions of robots, if not tens of billions of robots, possibility in the future.

So if you look at it from that perspective, we are in less than 1% of that. Because we are barely shipping a million robots, the industry is shipping barely a million or two today, but the opportunity is in tens of billions. So you are 10,000 times away to get there. So what needs to happen? Of course, there’ll be many different embodiments as time goes by. But especially for the robots, like humanoid robots, that need to be reasonably general purpose and have good enough general purpose intelligence to do multitask and then also have a component of the brain that’s going to be super specialized in doing certain tasks better than anybody else, because you don’t need every robot to be good at everything at super everything. It is going to be super expensive and the mechatronics may not even allow it because you have to make some trade-offs. If you don’t have the right mechatronics on you, it’s unlikely for you to be able to do all sorts of jobs.

So you need faster edge compute, of course. Memory, especially in today’s world, you see with all the supply chain issues, memory capacity is a problem. So in terms of using the right amount of memory, optimizing, using the right numerical precision for getting the right accuracy, but at the same time, because edge computers are more constrained in terms of area, in terms of cost, in terms of power, you need to be much more efficient. And we’ve been on this journey for over a decade. In fact, the first Jetson was 2014. So 11 years plus into this journey.

And interestingly, Austin, the thing that I realized in all of this journey is, remember the four steps that I mentioned from data to train to test and simulation or deployment? When we started, the only technology we had was the deployment technology. In fact, that is the destination. The ultimate destination is to have the physical robot. And what I realized is that the slowest way to get to the destination is to work on that problem. Because there are not that many robots. But we didn’t have the technology for training at that time. We didn’t have the technology for simulation at that time. So these got built now. And we are seeing, as we work with literally every robotics company on the planet, of course, everybody has to build a physical robot to test. But there’s 1,000 times more activity happening in training and testing and simulation today.

Okay, this was really good. I asked really about the edge computer, but you zoomed me out, which was good, and said, hey, don’t forget it’s not just about the deployment, but it’s about collecting the data so that you can train a model, so that you can simulate it, so that you ultimately have the confidence to go deploy it. And to your point, you guys have been doing a lot of work in the simulation world, because obviously it’s cheaper and ultimately makes a more robust deployment if you can simulate all this stuff instead of having robots go out in the real world and either have to wait around for conditions or just get broken. So there’s a lot of learning in simulation. Can you maybe walk listeners, remind them, so this ties into the three computer business model — computers for training the model, for simulating, and then ultimately you’re deploying at the edge. Can you walk us through each of those and remind us what kind of platforms are people using? I think a lot of people are familiar of course with training but what about simulation and then could you walk us through maybe the portfolio at the edge?

DT: Absolutely. So you’re right. Robotics, we think, needs three computers, as you mentioned, because you need the third computer, which is the brain inside the robot — that’s the runtime. And we’ve been working on it the longest, believe it or not. So we have this portfolio of products called the Nvidia Jetson. It’s incredibly popular — over close to, I think, two and a half million developers on the platform. Our current generation is Thor and Orin. More than 10,000 companies have been building robots either shipping or in the process of developing and about to ship robots. Incredibly robust ecosystem with so many partners. And they go into all sorts of form factors, all sorts of embodiments from humanoid robots, to agriculture robots, to medical robots, to delivery robots, to drones, to video analytics appliances, to telepresence type of devices, you name it. The breadth of end equipments and industries that companies and developers have been leveraging has been amazing. So that’s the third computer.

And then we keep on improving the performance. If you think about when we first launched the first Jetson, let me see, it was 192 gigaflops of processing. Today the latest generation is two petaflops. So that is 4,000 times performance increase in roughly 10 years. And then along the way came AI and we support all the latest, greatest models. The beautiful thing about Nvidia, we’re fortunate because we share the same architecture, what you’re running in the data center, what ChatGPT or Claude or Gemini or anyone, Qwen or you name it, any model or Nemotron from us, everything runs on our GPU because it’s fully programmable. That same GPU is in our Jetson portfolio as well. So as a result, we can run data center type of models at the edge. It’s just a question at that point of do you have, what’s the number of tokens per second? How fast is it? What are the different trade-offs that you need to do? So that’s our third computer.

The first computer is where we do the training. You mentioned it, most people are familiar. It’s exactly the same computer that are in all the different clouds and different enterprises and all the different neoclouds, sovereign clouds and so on. Same GB300 is our current latest shipping product. There’s a lot of Hoppers and Grace Blackwells and then Vera Rubin about to ship in next quarter, coming up fairly soon. So that’s first computer. You train the data and training happens in that computer.

And then there’s the computer in the middle, which you asked, which is where you need to test it. And you want to test it in simulation. And people wonder about when I say you must test in simulation, people would be like, no kidding. Nothing new. In fact, we’ve been building chips for 30-plus years now and every chip before we send it out to tape out to manufacturing, we have 100% simulated, emulated it and we know it’s going to work. Without simulation, it’s impossible for us, because if you don’t simulate and test it and if the answer comes out wrong out of the manufacturing fab, you’re one year away. And can you imagine if you’re one year late on any of the products that we’re doing? We are making products every one year now, going into hundreds of billions of dollars of infrastructure and eventually trillions of dollars of infrastructure. So we know that it works. That’s why we do simulation, emulation for chips.

Then you ask the question, why are you telling me in robotics that simulation is so important? It’s a no-kidding. It turns out that the simulation in robotics, the technology was not as good — the sim-to-real gap is sufficiently large until recently that you can simulate all you want, but it’s not exactly representative of what happens in the real world. So you’re almost throwing it away. That has been the problem because remember in the case of robotics, the simulation, the physics and all of that is extremely complicated.

So now the technology has become reasonably good enough thanks to AI and thanks to our investments in the Nvidia Omniverse, which we’ve been working on for well over 15 years for all sorts of simulation, started with games first and then went into all sorts of general physics and chemistry and all the high-performance computing modeling. Because of that, we’ve been able to build this platform that now the sim-to-real gap is manageable for many tasks and increasingly that gap is getting closed with thanks to reinforcement learning and new physics engines. Recently we announced this open source physics engine called Newton — work with Nvidia and Disney Research and Google DeepMind, and it’s completely open. So this is truly the first physics engine being built for solving robotics problems.

A lot of work had to be done to create this second computer in the middle, and it’s Omniverse. And so our best computer today is RTX Pro 6000, and there’s different flavors of it, different RTX Pro versions of it, but that’s our flagship. And it’s available also through different clouds. It’s available in workstations and computers from all the different OEMs. So that’s the three different computers for training number one, simulation number two, and then the runtime.

And the last thing I would add is once you deploy a robot, your journey doesn’t end. That’s actually the first, because these robots are going to be in the field for 5, 10, 15, 20 years in some cases, and you would expect them to get smarter over time. Just like you hired an employee and they’re good to go on day one, but they’re going to be learning new skills and important things and new problems need to be solved in the next 20, 30 years. Which means this loop of data generation and training and testing and deploying, this is a forever loop. This flywheel is forever. Deployment is just the first step.

Where the Spend Goes Today

Okay, wow, this is so cool and so interesting. So RTX Pro for simulation, that’s interesting. Do customers — you mentioned they’re in various clouds. It probably depends on the customer and on the domain, but are customers ultimately buying a lot of this hardware or are they just renting it as the simulation needs are on demand? And I assume that because you said there’s this loop of you’re always trying to make things better, are robotics companies just kind of constantly training, simulating, deploying, iterating?

DT: Absolutely. Much of the spend, if you look at all the latest, greatest robotics labs or startups, who have raised hundreds of millions of dollars, a billion dollar valuation because it’s the toughest problem. Much of their spend today goes into training and simulation, because until you get a reasonably good enough, accurate model, why bother deploying at the edge and scaling out? You do want to deploy at the edge to make sure you’re testing right. But the scale of deployment at the edge initially is going to be limited until you get accurate. So much of the action today is happening in training and simulation. And it depends on — so the compute is absolutely available in all the clouds and neoclouds. So that would be renting, on a demand basis. And some of these companies are also able to build their own local on-premise cluster for both training and simulation. So it’s going to be a hybrid model depending on how much compute is required.

Sure, that makes sense. So maybe a timely business question. On the earnings call last night, Nvidia introduced new business units, kind of rolling things up differently. So there’s the data center and then there’s the edge. And data center was hyperscaler and non-hyperscaler and then there’s the edge. But when you’re talking about the three-computer business model, it sort of spans both of those. And you talk about, like early, like a startup, maybe they’re raising a ton of money and right now they’re investing a lot in training and simulation and then small in deployment, but eventually that will ramp. How do you sort of track that across the different ways that it rolls up?

DT: So the announcement is not about new business units, it’s a new way to report so that investors and analysts and everybody can understand our business better. Today, much of the action is happening in the data center and especially for digital AI, for enterprise AI. And then increasingly we’re starting to see, even though we started investing in this well over a decade ago for physical AI, in the next three to five years, we are expecting major unlock in technology for the whole industry. And as a result, the amount of compute that’s going to be consumed, whether in the cloud or an enterprise cloud or on-prem edge, physical AI robotics will span across all of these computers.

So my job and my team’s job at Nvidia is to essentially create the workflows, create the technology that all these companies that are building robotics — they could not be building a full robot, they might be just building a brain, they might be just doing simulation, or they might be doing sensing actuation — is to provide the technology that they can use to essentially build their product and our solution. And the technology that we deliver is going to span across a cloud — could be AWS or GCP or Azure or OCI — or it could be in a neocloud like a Nebius or a CoreWeave, or it could be through some local on-prem workstation or even clusters that are built, or it could be a Jetson Omniverse cluster. It doesn’t matter. So the way we think about it is, it doesn’t matter where the computer is. It’s all about creating the right workflows and that leads to the right computer usage appropriately.

Agentic AI and Fleet Simulation

Sure, totally, it makes sense. Going back, you talked about ultimately for robotics and physical AI to actually be used in the real world and useful — first, there’s the getting in the game, which I love the 100 meter dash analogy by the way. I love track and field athletics. So first, you’ve just got to get to the 10-second mark just to get in the game. And you talked about a big part of that being essentially how performance, how good, how believable the model is. And then beyond that, you talked about integration and you mentioned agentic AI. There was reasoning. These models just have to be good, have to give good responses. But then you talked about reasoning, taking it to the next level, but then you also mentioned agents and agentic AI, and I’m super curious to unpack — what does agents at the edge look like? What do you even mean there?

DT: That’s a good question. So most of the things we talked about today is about making a robot extremely useful. But ultimately, if you think about many of the applications in industry, in enterprises, it’s not going to be just about a robot. It’s going to be about fleets of robots. Just like if you have a company, it’s not about an employee, but it’s going to be about all of them working together to create something. Imagine a factory. A factory in the future will have robots of different embodiments, different levels of intelligence. There’s likely going to be people too. And there’s going to be digital AIs. And you have a game plan of a manufacturing plan, which includes doing it in a safe manner, improving throughput, and you have to manage all the inventory of supplies coming in and supplies going out, and all of that is going to have to be orchestrated.

So how are you going to combine each and every one of the robots with different capabilities and somehow integrate them all together and evaluate scenarios and what is the best way to do it? This is where agentic AI comes in, because it will integrate each of these different digital AIs and physical AIs to have Uber policies. And now the next question is, how do you validate that the policy is the best one? You don’t want to stop your manufacturing line to test these policies. And it turns out you actually want to do all of this in simulation too. This is where a digital twin of an environment of a factory matters.

And one of the things that we’ve been working on, there’s this technology called Nvidia Mega, which is a blueprint for doing fleet simulation of a complete factory level or a city level or a building level, if you will, doesn’t matter what that abstraction is, and simulating all the different physical agents, digital agents, AIs and orchestrating all of that for testing. This is why I keep talking about agents is going to be super important, because ultimately it’s not going to be about a robot. A robot needs to be good, but it’s about how you integrate all of them together to create a bigger job, bigger task.

Fascinating. So that’s super interesting. I think a lot of people are starting to think about agents. I’ve got OpenClaw running and I’ve got the agent that checks my email and the agent that does this and that and it summarizes things and whatnot. And you’re saying, yeah, those are digital agents, but we’re going to also have physical embodied agents. And so this orchestration plane in an enterprise or in an industrial setting is going to need to not only be able to interact with an orchestrator, digital agents but also physical agents and orchestrate across them. And then you made the super interesting point, which is okay, well you ought to be simulating and testing a digital twin of that too. Because how do you know that — yeah, because of course there’s going to be some sort of supply chain manager, logistics person, or factory manager that’s going to want to play around with this and it’s going to be a lot easier to test all of that in simulation again rather than just willy-nilly try something in the factory and have the line shut down or whatever. Super, super, super interesting. Yeah, you guys are definitely living in the future.

DT: That’s right. And the joke I have is, I’ve been living in the future for more than a decade, but for the first time I feel like the future is coming to the present.

The Road Ahead: 2029–2030 and the Edge Roadmap

Yeah, absolutely. So look ahead three years, 2029, 2030, let’s say. How close are we to that sort of world where there is a business and it’s orchestrating across humans and digital agents and physical agents?

DT: I think it’s going to happen. It’s going to happen in stages. So the first step is a robot. What sort of jobs or tasks is it able to do at the right level, sufficient level of accuracy, throughput, cost and so on. That has to happen for those kinds of robots to scale up. And then increasingly robots will become good at solving more and more of these tasks. So what we’ll see is it’s going to be like a continuous journey where, just like autonomous vehicles have hit the 10-second mark, we’ll see that the manipulation, rigid bodies and locomotion type of tasks the next year or two will hit the 10-second mark. So you’ll see more of those being deployed in factories and warehouses. And once they’re deployed, agentic AI will absolutely be required to orchestrate all of that. So that’s likely going to be deployed by 2029, 2030.

Now, as time goes by and as we solve general purpose intelligence and solve more dexterous manipulation, fine-grain with soft bodies and deformables and all of that, then you can imagine, you’re going to unlock more and more use cases. And that really is going to be technology — you got to the 10-second mark. When you hit the 10-second mark, you would know. And then the next two, three years after hitting the 10-second mark is trying to win that Olympic gold.

Yeah, fascinating. So last question. As you’re talking about this future where you have different embodiments and different use cases being solved, that are eventually, they’re kind of like point solutions until they hit this 10-second mark and then it’s like, good, throw them in the mix, start orchestrating across them and more and more sort of unlock that level of sufficiency to be deployed and orchestrated across. Ultimately, what does that mean for the edge computing portfolio? Are there going to be tons of different SKUs because people need different memory and different compute for those different work cases? Or do you ultimately see that actually a lot of this is still solved by maybe a tighter compute portfolio just maybe deployed at different power levels? How are you guys thinking about the future of the roadmap?

DT: So we already have different SKUs. We have in the Thor family, we have two SKUs. In the case of Orin, we have like six different SKUs, same software, but depending on the performance required, depending on the cost, power, and functional safety and industrial grade, because the number of applications that you think about is so broad. So we have a pretty broad portfolio.

And then ultimately, we will address portfolio where extreme high compute and all of this is important, but there will also be applications where maybe you don’t need all the performance that we’re providing because a lot of these would be hybrid edge-cloud. Because there’ll be a lot of intelligence at the edge, but there’s no reason why you wouldn’t want to phone a friend and call into the cloud to get some answer for something, especially for long reasoning, long thinking type of things, which you never have enough compute to do locally at the edge.

So our portfolio is, we have a pretty vibrant portfolio from, same software but leveraging the same compute architecture, but scaling in performance, price points and power. And then we’ll continue to do that.

Nice, that’s cool. I hadn’t thought — it’s good to be reminded that these robots don’t need to exist in isolation, but they can always call the local cloud computer or whatever and kind of phone home or phone a friend. That’s interesting. Alright, well, that’s it for today. A lot to chew on. Thank you so much. I learned a lot. I think listeners are gonna love this. Appreciate you spending time with us, Deepu. Thanks.

DT: Yeah, my pleasure, Austin. Take care.

Chipstrat is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

There are physical limits at higher bandwidths too, and that changes what technologies matter over time. EMLs, SiPh, VCSELs, AECs, ACCs, AOCs, microLEDs, etc.

On top of that, capacity constraints are in play (e.g. InP wafers), so decisions made 18–24 months earlier determine who can actually ship when the doubling lands.

Timing is everything.

But there’s a lock to unpack. Where to start? Let’s wrap our heads around the cadence the industry is chasing by reading recent earnings call from the public semis, optics, networking, and contract manufacturers in the stack.

Here’s what they laid out, at the highest level:

• 400G: mature.

• 800G: volume cycle in 2026 and 2027.

• 1.6T: production scale in 2027.

• 3.2T: launching in 2028, volume ramps 2029–2030.

In chart form:

One caveat on reading the chart. We have to infer the mix shift ourselves. No CEO says “our 400G sales are declining” on the call.

Subscribe now

So that chart captures the cadence at the highest level, as decoded from ~15 companies’ earnings calls. But there’s a ton of nuance to unpack. First, there are differences between roadmaps for scale up/out/across. Plus, the further out stuff is fuzzy, so we should compare what the CEOs optimistically said against actual OFC (R&D) announcements.

For paid subscribers who want to go deep, we’ll hit on

• Year by year: Scale-out, Scale-up, Scale-across

• The receipts. ~50 direct CEO quotes from ~15 companies grouped by generation and year

• Timing nuances, especially 3.2T

• OFC 2026 reality check

We’ll also check back in quarterly to see how the communicated roadmaps change.

Read more

Natalie is a co-founder of Gimlet, alongside CEO Zain Asgar (a Stanford CS professor). Beltir spent years at Intel before joining Gimlet five months ago, after Gimlet had been one of her portfolio companies. The company was founded in 2023, has raised $92M (Series A this March), reports more than $10M in annualized revenue, and runs a two-track business — deploying its orchestration software inside customers’ data centers, and operating its own neocloud with mixed silicon.

In this interview, we walk through how Gimlet thinks about both the architecture and the business. Important insights:

• Most neoclouds are backed by one silicon vendor and gave significant equity in return. Hardware amortization is ~70% of their annual costs, leaving very little room to optimize bottom line. That equity entanglement means they can’t diversify their silicon, which is why the only software innovation they can ship is disaggregation on top of a single vendor’s stack — never across vendors

• Gimlet’s two-track business model is the answer to that constraint: deploy software inside customer data centers (frontier labs, hyperscalers, sovereigns) and operate their own neocloud with mixed silicon for AI-native customers. Supply-chain diversity optimizes the bottom line, differentiated token performance commands a price premium on the top line, and one track funds the CapEx of the other

• Hyperscalers and frontier labs already run multi-vendor silicon (NVIDIA, AMD, in-house ASICs), but the orchestration layer is getting more complex faster than internal teams can keep up. They’d rather spend engineering attention on next-gen training and product differentiation, so some outsource orchestration to Gimlet — and some go further, having Gimlet take on the CapEx and data-center burden so they can experiment with hardware combinations without staffing a forever-team

• AI-native customers aren’t just price-sensitive — they have product latency budgets (e.g. one-second response windows, voice agents) where faster tokens unlock entirely new user experiences, not just cheaper ones

• Sovereign clouds are a prime customer segment — Europe, the Middle East, India, Asia, and Korea have government funding and some have emerging local silicon vendors, but lack the in-house software talent to write optimized kernels across N chips. Gimlet’s pitch is “make an API call, not a porting project”

• On the architecture side, Gimlet’s stack traces a PyTorch workload as a graph, splits it at optimal points, then lowers each segment to the target vendor’s framework (TensorRT on NVIDIA, equivalents elsewhere). They don’t try to build a universal programming language across chips

• On GPT-OSS 120B with 8K input and 1K output, running the speculative decoder on a d-Matrix Corsair card while NVIDIA B200s handled the verifier delivered roughly a 4× shift in the throughput-vs-interactivity Pareto frontier compared to GPU-only speculative decode

They’re also hiring across the stack: scheduler, compiler, kernel optimization, and distributed-systems engineering, in person in San Francisco.

This interview is lightly edited for clarity.

Meet Gimlet

Hello, everyone. Today we have special guests from Gimlet Labs. We have Natalie and Beltir. And we’re going to talk all things heterogeneous silicon and rethinking the data center. So let’s start, Natalie, with you. Our audience probably doesn’t know you guys or Gimlet. So tell us more about you.

Natalie: My name is Natalie. I’m a co-founder of Gimlet. Gimlet, we can go more into it, but what we are is we’re an inference cloud built for agents. And one of the key aspects about our technology is that we’ve built this inference cloud across heterogeneous hardware. And we can get more into that, but we think that that is going to be the future of inference.

Nice, exciting. And Beltir, who are you and how did you get to Gimlet?

Beltir: I’m Beltir. Nice to meet you. I joined Gimlet roughly five months ago. Before joining Gimlet, I was at Intel, and Gimlet was one of my four portfolio companies that I’ve been working very closely with. I’m amazingly excited about what we’re building at Gimlet, so I ditched my corporate job and jumped onto the startup again and trying to build a very exciting business. It’s been an amazing five months so far.

The Case for Heterogeneous Infrastructure

Okay, I’m just going to try taking us through some of your slides that I clipped from blogs and found online to just get your reactions and have you talk through for our audience, in real time, what you’re trying to do, what problem you’re trying to solve, and why it’s important. I’ve written a lot about this shift from a GPU, one-size-fits-all GPU, to multi-vendor, multi-silicon environments. So I was excited when I saw that you guys are thinking about this too. Natalie, tell me — make the case for heterogeneous infrastructure.

Natalie: Just starting with the context that we’re all probably familiar with, it feels like every day you hear an announcement that one of the large frontier labs has made some kind of compute deal for capacity with some chip vendor, whether it’s Trainium, AMD, TPUs, or NVIDIA. And another piece of news you hear, it feels like almost every day, is that there’s a new accelerator company that just launched. They have amazing performance on inference. They’re designed for inference specifically.

What we’re basically seeing in the broad context is that everyone’s extremely capacity-constrained at this point, trying to scale out their inference. They’re trying to improve the performance of their inference. They need as much compute as possible. And then they also need specialized compute potentially to make it even faster. So how does that all fit together? Sometimes people ask, is this new chip going to be the GPU killer or something like that?

So the way that we see it at Gimlet is a little bit different. We think that all of these options are really great for different purposes. And that’s important because agentic inference is not a uniform workload. Different parts of it have different compute needs and different bottlenecks. So when you think about a really, really large-scale workload that you need to be very fast and efficient because we’re pouring trillions of dollars of CapEx into it, then you want to start thinking, how can I optimize the attention of this model? How can I optimize my speculative decoder or my tool calls? Each of those components actually benefits from a different type of hardware because it has different trade-offs. So what we see is that the industry is moving toward a heterogeneous stack for inference in order to meet the performance needs.

Yes. So I found you guys had this slide here, and this feels like it’s exactly what you’re saying, which is breaking down the workloads that are running at scale. It used to be there was a time when it was kind of like, let’s accelerate everything and we’re not sure what the dominant workloads are. So a GPU can run high-performance compute, scientific compute, or AI. But now obviously all of the inference that’s happening is really about LLM inference for these few frontier labs at scale, and so it feels like you can start to take that inference workload and ask, what is the right silicon for this workload? Maybe to the point of the table — what are the different parts of that workload, what are their system requirements, and how might those actually fit onto different hardware?

Natalie: I think one thing about GPUs is they’re incredibly versatile. So we definitely think they’re going to be an important part of the inference stack. When you look at this table, we have it broken down by different very high-level phases of inference, showing the resource needs for each of them and how they vary, and how you actually can’t have one chip that is optimal for all of these. It’s just literally not possible. But each of these is a critical stage.

So how do you solve that problem? Our core belief is that you solve that problem by disaggregating the workload and running each segment on the chip that’s best suited for it. The other thing I want to point out about this table is that even this is very, very coarse-grained. You can subdivide each of these components into more segments, each of which have distinct bottlenecks from each other. So it’s one of those problems that even at this level, people will benefit from disaggregating, but we’re thinking even more so than that — even within LLM pre-fill, how can we disaggregate that further?

Disaggregating the Workload and Orchestrating It

Say more here. You’re talking about, how can we split the workload up ever finer and finer? And then apparently in real time, being able to distribute that across the correct hardware.

Natalie: Right, and you also don’t want to indefinitely subdivide, because there is cost between sending the data from one chip to another. But it’s about expressing the workload, finding the optimal points to split it up, and then scheduling and scaling it across the available hardware.

Okay, so do you do that in advance of running it then? You look at the workload and figure out where those points are to break it up?

Natalie: Yeah, that’s a great question. Some of the other slides will go into it a little bit more, but the way to think about it is that we take the workload, we trace it. So if you give us PyTorch, we’ll trace that. It could be something else too. And then we’ll actually turn that into a graph representation. Our orchestrator and scheduler basically figures out how to segment it into its component parts for further compilation. So we trace it, we walk that graph and we understand what’s there, we break it up and optimize how we do those splits. And then for each of those segments, we’ll lower it to the target hardware.

One thing that I also like to point out here is we really work closely with our hardware partners because we’re trying to use the frameworks that they have available at the low level, not trying to create a stack that is a programming language for every single chip. So once we have those segments, we’ll actually compile them and lower them down to, for example, TensorRT on NVIDIA, or other similar frameworks on other hardware.

Fascinating. So I feel like what I’m hearing from you is that this is obviously more than just a hardware play, but obviously you’re doing a lot in the software stack to really orchestrate, is what I’m hearing.

Natalie: That’s right. We think of ourselves primarily as focusing on the software layer. We have to tap into hardware as well because we’re connecting these different platforms together. We’re connecting chips that have never been connected together before because no one has taken chips from vendor A and vendor B and plugged them together and orchestrated a single workload across them. So we end up having to play in that layer a bit too. But what we’re really emphasizing at Gimlet is the software layer for orchestrating across this hardware.

We think that, to the slide that you just pulled up, this is a problem that is going to compound, not ease, over time, because everyone is still coping from the massive scale-up of simple LLM inference. But what everyone’s moving to, and we see this with coding agents, is multi-step agents that are doing searches, running things on your machine, maybe calling out to other agents. These are even more heterogeneous than the LLM chat models, which were much more heterogeneous than people really even account for on their own. Once we start moving to background async agents that are all communicating with each other, they’re multimodal, there’s different model types — the whole problem of the inefficiency of a homogeneous stack is just going to get completely untenable.

Yes, that makes a lot of sense. We’re moving to where it’s not just the human interacting with the LLM, but you’ve got agents, the agents are doing different things, calling different models. So there’s lots of opportunity to optimize. Thinking like this sort of agent end-to-end workflow, what does that look like from an orchestration perspective? You have an illustration here, but in my head, it just feels very complicated when I’m thinking about agents, tool calling, and all that stuff. Talk to me more about this orchestration layer.

Natalie: We touched on it a bit before, but I think we think about optimizing and orchestrating across an entire agent, not just an individual model. We represent these things as graphs in our system. At the end of the day, we don’t really care what type of model it is, what things it’s doing, as long as we can represent it in our compiler’s framework and then figure out what its bottlenecks are and then schedule it on hardware.

So whether it’s one model, two models, models with functions — it’s all kind of the same in the way that we’ve designed our system. The important part is that we can trace that entire thing and then split it up and then, like this diagram shows, route it to the appropriate accelerator.

CPUs, Tool Calls, and the High-Speed Fabric

In this diagram you showed GPU, specialized accelerator (so maybe like an SRAM-heavy one), and CPU. CPUs are all the talk lately. Tell me how you’re thinking about CPUs. What type of workloads are you putting on the CPUs?

Natalie: That’s a great question. It’s been really awesome to see the excitement about CPUs recently because they are a really important workhorse of these agentic workloads. A pure LLM or a pure model only has so much capability unless you can actually connect it to the outside world and the ability to do general-purpose tasks. So the most obvious application of CPUs is things like tool calls, but you can also use them for things like smaller models or data processing and other types of things that benefit from the CPU’s trade-offs. But tool calls for me are the most exciting thing. When you actually run that tool call in the same place that you’re running the LLM, it really improves the end-to-end latency of the overall agent.

What do you mean by in the same place as the LLM?

Natalie: For example, when I’m using a coding agent today, the LLM is running on someone’s server. And then it’s coming back to me, saying to my machine, please look up the contents of this file, or please do a web search. And then that is executed from my laptop. This introduces a very network-bound aspect of the workload because it has to constantly jump back and forth between my laptop and where the model is running. So what I’m saying is that for cases where you can actually run those tools on the server side, you end up with much, much better performance.

Okay, so would you say then, especially in your architecture, the CPU rack should be in the same data hall on the same network, or is it just as long as it’s off of your laptop and running in the cloud, maybe there’s lower latency?

Natalie: It depends on the needs of the workload, but what we would generally say is that the way we approach it at Gimlet is we want to connect all of this hardware together through high-speed fabric. So that’s why we’re not just saying this data center is for hardware A and this data center is for hardware B — we’re actually physically connecting these racks together. In general, I think that it’s better the closer it is.

Beltir: The reason we want that proximity is actually latency, because there’s a big demand for really fast tokens and higher user interactivity. This today usually comes at the expense of a throughput hit. And in a world where everybody is power-constrained, capacity-constrained, people have to make really hard choices. Whether am I going to have a throughput hit but for high, low-latency tokens, or am I just going to optimize for throughput? By putting these different types of hardware in the same data center, interconnecting them, we’re trying to give customers a solution that actually expands that barrier where they can make these choices without as much of a trade-off on either end.

Okay, interesting. So at the end of the day, if we want as fast tokens as possible, you’re saying we should disaggregate the workload and put it on the right silicon for that shape of the workload. And we need a high-speed fabric, and ideally you would have all of the hardware that you’re scheduling across sitting on the same fabric to reduce latency.

Natalie: That’s right. For some types of disaggregation, this matters more than others. So for something like pre-fill/decode disaggregation, you might be okay with a hop, because that’s only happening one single time between the ingestion of the context and the outputting of the first token, then hop to emitting every subsequent token. But for more fine-grain disaggregation, it becomes more important.

Three Customer Segments: Sovereigns, Frontier Labs, AI Natives

Okay, so at a high level, we’ve talked through some of what you’re trying to do, which — reflecting back for listeners — you’re saying, hey, what if we built an inference cloud for agents where actually inside the data center, there’s lots of different kinds of hardware, and we’ll write a software stack that’s like an orchestration stack that looks at the workload, figures out where’s the right place to break it into little subtasks, and then we will give it to the correct hardware, whether that’s CPUs or SRAM accelerators or HBM accelerators, and we’ll have it all on a high-speed fabric so they can all communicate really well. So I guess that leads to the question — who’s this for? Who are the customers and why is your cloud going to be compelling for them? Beltir, I’ll hand it off — educate us on the customers.

Beltir: I put our customers in a couple of big buckets. The first bucket is frontier labs, in my mind, who are making all these contracts with many different silicon vendors. Again, everybody is power-constrained, capacity-constrained today, and as we talked about, they’re all trying to solve the problem of, how can I provide the fastest tokens — which is better user experience — without compromising my throughput, or getting as much throughput as I can from my existing investment. This is a never-ending problem as the baseline keeps moving and the capacity constraints become more and more of a bottleneck for everyone. That’s one bucket.

The second bucket of customers we get a lot of interest from is sovereign cloud vendors who are interested in supply-chain diversity, who are putting together multiple of these contracts in place, but lack the capability to be able to serve them at scale. Bringing up a new hardware vendor is a lot of work. Porting one same workload from NVIDIA to AMD to MatX, it’s a lot of work. What we’re talking about is not saying that we will take your workload and we will port it. What we’re saying is you shouldn’t be worrying about these different hardwares and porting your workload to each and every one of them separately. You should just make an API call, or you should have an intelligent software stack — if you’re deploying our software stack in your data centers — that actually takes your workload and figures this mixing-and-matching algorithm itself, rather than your engineers trying to write kernels for each and every one of these hardwares. This also creates a big bottleneck for them to get these new emerging architectures. Because who’s going to write those kernels for those? It’s pretty hard.

The third set of customers are what I call the up-and-coming AI natives who are buying tokens at scale. ElevenLabs, Notion, Glean, Harvey-type companies. Companies who are building the next-generation diffusion models, very latency-sensitive. They’re amazingly constrained by what the current infrastructure is offering them, which is, we have a good enough product but it doesn’t give the latency or the fast tokens that you need to be able to innovate the next-tier user experience. The first bucket for us is a combination of both — us deploying our software in their existing data centers. The second and third tier of customers is mostly around customers who buy tokens at bulk from our new cloud infrastructure.

Okay, this is super interesting. So I want to unpack this and go into each of them. Let’s start with sovereigns. So sovereigns, what I heard you saying is — you’re sovereign, you’re standing up your own data centers, you’re going to buy from different vendors over time so that you can have that supply-chain diversity. And then you’ve gotten yourself into a situation where you already have different hardware, but now you’re stuck with: man, that has increased the amount of software engineering we have to do, because now we have to decide maybe manually which workload goes where, and we have to write optimized kernels to run on the different hardware. So it’s like a software burden on maybe a customer who doesn’t have a huge software team. So you guys can come in and say, hey, we’ll take a look at your hardware and we will help you orchestrate across that hardware. Is that ultimately?

Beltir: That’s ultimately what we’re trying to go for.

Okay, that makes a lot of sense. You’re kind of like the cracked software engineering team that they need.

Beltir: Cracked software is the software platform that they need. Today, most of these infrastructures are set up as a bare-metal-as-a-service infrastructure, which has its own challenges from a software engineering perspective. What we’re offering them is not a set of software engineers — we’re doing this work for our own neocloud offering anyway. We are building this orchestration stack in deep partnership with those hardware vendors for our own business. What we’re offering them is a ready-made platform that we can deploy in their existing data centers for them to very quickly get to market with the existing investments they’re making, but not only time-to-market, also better throughput, better capacity, and better user experience from that as well.

Because all the sovereign clouds also, they just don’t want to build this for the sake of building it. They also want to be at the frontier of the innovation as well. If you look at Europe, there is a lot of government funding that’s going in this area for them to be part of the innovation ecosystem. Same in the Middle East, same in India, same in Asia. How can you give them an offering that actually helps them get there faster, differentiate them — is another part of the equation. And the other one is they are very keen on supply-chain diversity. Having NVIDIA and AMD doesn’t solve the problem. There is a lot of hardware innovation that’s happening outside of the US as well. That also has the same issues. If you look at Korea, there are really interesting chip companies that are coming out of the Korean ecosystem that we’re talking together with right now. They’re also thinking through how can these emerging hardware architectures be consumed without the software burden. Because being able to do this kernel engineering, the software model porting — it’s a lot of work.

Fascinating. I didn’t think about the point that we tend to focus on American chip companies, but there are actually other chip companies elsewhere. So not only for sovereigns can you solve the “hey, you don’t have to worry about software, our platform solves that for you” piece — and then I liked your point, which by the way, we will make sure that it’s highly optimized, so it’s not just that you got it to run, but we’re going to optimize it for you. On top of it, yes, you can as a platform take on the burden of getting comfortable and making sure that you work with all sorts of vendors from different countries, because that makes sense for you as a platform and then that’s something that you can offer to all of your customers.

Natalie: If you want the best performance, you really have to partner closely with the chip company. That applies to pretty much everyone. If you’re running a production-scale workload, you need to get a very close relationship with the hardware maker that you’re running it on. Doing so for N hardware platforms — and also keep in mind, it’s hard enough to get performance on one — moving it to another is another step up. Taking it and breaking it up and running it on even more, that’s something that we think is optimal from an efficiency standpoint, and it’s why we’re building Gimlet, but it would be very difficult for everyone in the space to replicate that.

Yeah, totally. Not to mention merchant silicon vendors only have so much bandwidth. I’m sure they can only help so many people that come to them. So I can see how it could be win-win for them if they can just work with you, and then you can make it work with everyone.

Natalie: One more point — the chip companies, GPUs are amazingly versatile. You have other hardware that’s really, really great at many parts of inference. By putting it alongside other types of hardware, it can really shine in the tasks that it’s best suited for.

Beltir: And it also de-risks from a customer experience perspective. You all are very comfortable with running on NVIDIA, running on the AMD ecosystem, but you will have a hard time porting your model on another vendor’s cloud-only option. Many hardware silicon vendors try to stand up their own clouds because customers were hesitant to use their cloud infrastructure. But what they’re also seeing is, even if they set up that cloud infrastructure, at-scale customers — for them, it’s also a lot of engineering effort to move their workloads to one vendor’s cloud only. So those clouds are not scaling. What we’re offering is a mix-and-match environment for the customers who are looking to benefit from these emerging architectures, and for the emerging silicon vendors, a way to go to market at scale without taking on the burden of building their own cloud infrastructure, because that’s not their core business.

Okay, so now let’s go to the hyperscalers or those serving the frontier labs. Hyperscalers, we know they have multi-vendor silicon. Meta’s always talked about a lot lately — they run NVIDIA, they run AMD, they have their own MTIA chips. Now, unlike the sovereigns, a hyperscaler has plenty of software engineers, even though this is a laborious task to optimize their kernels for all the different hardware. So tell me, why is it better that they would partner with you rather than just try — maybe they’ve already built this sort of orchestration themselves — or where is what they’re doing suboptimal compared to what you’re doing?

Beltir: I think there is no one-size-fits-all answer for a hyperscaler or frontier lab. There are different stages in this journey, because three years ago we weren’t talking about this level of disaggregation of inference workloads. We didn’t know what inference was going to look like. P/D disaggregation was a very early PhD-thesis-type of implementation. Today it’s becoming more commonplace and we’re talking about way more complicated disaggregation methods. There are different phases and different stages in their journey of figuring out how they’re going to serve inference at scale.

Some of them are trying to build this in-house with competing priorities. Some of them, the ones we’re working very closely with, are telling us that this is not their current strength or current focus. They’re in a place to meet the next-generation training wars, and in next-generation, differentiating their product. Rather than trying to bring up a different infrastructure, writing kernels, getting them up and running, they would like to outsource all of this so that they can experiment — because they know their workloads — so that they can experiment which of these combinations will give them the best alternative.

The other part of this is, as they’re investing more and more CapEx for their data centers, their margins are getting thinner and thinner. So what we’re also seeing is they’re trying to outsource some of these investments to companies like us, saying, okay, you take the data center burden, you take the CapEx burden, you bring it up and running for me. See how this will work for me in this particular hardware combination — because they already have those deals with the hardware vendors. So we see a couple of different reasons depending on where they are in their journey.

That really resonates with me, especially when you talk about what are their core competencies and what is their ultimate business model, and how can they spend as much time training a better model or whatever. It actually reminds me — when I was in grad school and when I was an undergrad and I did research, both times I benefited from people who came before me. A PhD student would spend like four years building a system and then they would only have two years left to quickly run some experiments on it. And then I would walk in and I just run experiments the whole time. And I’m like, man, I’m glad I didn’t have to spend four years building this. It’s kind of the same thing. You’re trying to say, let us build that infrastructure so that you can experiment on top of it. Let us handle optimizing and really focusing in this, and you guys just worry about your experiments.

Beltir: Exactly.

Natalie: No one really wants to go to all of those chip companies, optimize them for all. It’s a lot of work.

Totally, and you’re signing up to do that forever. You just built a team that is committed to doing that forever. I do like the idea of just outsourcing it, so you don’t let a company exist solely to solve that problem.

Beltir: Exactly. Think about every new hardware coming up — but not only that, the maintenance of an infrastructure like this is also a big ongoing commitment for them. Every new rack release you have to update. All of these create a lot of issues. And I think everybody is in a race to differentiate themselves rather than trying to figure out some of this plumbing in-house.

Yes, totally. Now, lastly, let’s talk about the AI native. So an AI native today, they don’t own their own infrastructure. They’re just trying to buy tokens as a service from APIs directly, or Amazon Bedrock or Google Vertex or something. And if I heard you right, what you were saying was today they can only get tokens. You can pay a lot for a fast token or pay less for a slow token, but maybe they don’t have enough fine-grain control. Or is it ultimately just like, by buying a token from you, it will be faster and lower cost? What’s the pitch?

Beltir: It’s a combination of both right now. I think there are two different types of customers. One of them are big enough so that the token cost is hurting their profit margins as they’re growing. So they’re more cost-sensitive and they’re looking for options to reduce that cost for them as they grow. The second one are emerging innovators that are building diffusion models, video-based solutions, voice-based solutions, where latency is a big, big bottleneck for them to bring a competitive product to a market. They have options, like I mentioned, on emerging new clouds’ own cloud solutions, but it comes at a very different trade-off for them to be able to do that. They have to spend their limited resources porting their models to those cloud solutions as well. So it’s a combination of two different customers that are approaching us right now.

Natalie: There’s another thing here — actually there’s two points I want to make. The first is that you get tokens from someone. At the end of the day, the limiting resource might be like power capacity. If we can deliver a shift in the Pareto frontier for the available power by leveraging heterogeneous hardware, we can translate that for our customers to lower latency, to higher throughput — it can be a variety of benefits, because you’ve actually shifted what’s possible by doing this. And what the folks in this bucket tell us is that, taking latency as an example, it’s not just that it’s better to get tokens faster. It’s actually that different product experiences have different latency budgets. The user can’t wait for a response more than one second. By making it three times faster, five times faster, what those folks tell us is it actually lets them enable new experiences that wouldn’t have been possible when using the providers that run homogeneous stacks.

So I only have a second to respond here so I can only do a couple of things — but if I could do a bunch of things in that second, then yeah, I can unlock a new user experience that is differentiating.

Natalie: This is especially important for things like voice agents.

The d-Matrix Partnership and the Pareto Frontier Shift

Totally. Okay, so you mentioned Pareto frontier. Let’s give one example before we end so people can understand what we’re talking about. Tell us about d-Matrix, your partnership with them, and then I’ve got the Pareto frontier slide after this.

Beltir: One of the things that we’ve been talking about is mixing and matching different architectures, but especially with GPUs and SRAM-based architectures. Without going into the technical details — this is how you can actually pair a throughput machine like an NVIDIA B200 or GB200 with an SRAM-based architecture, which are amazing decode machines and can push the latency frontier way more, multiples of what an NVIDIA or a GPU-based architecture can do.

We had this hypothesis that mixing and matching together can actually shift the Pareto curve faster. We partnered with d-Matrix. d-Matrix is only one of our partners that we can name publicly right now. We are partnering with multiple of these SRAM-based architectures. The d-Matrix team has been amazing from a time-to-market, speed, and partnership perspective in optimizing a software stack and a hardware for this. What we’ve done with them is basically putting together in our own data center a d-Matrix Corsair card in the same rack with NVIDIA B200s, directly connected to each other, to be able to test how much we can push the frontier curve. I’ll let Natalie talk about what it means and what we’ve done.

Natalie: Let me first orient the chart. I think your listeners are probably familiar with the classic chart that Jensen often shows, but just in case, let’s recap it. So on the y-axis, what we have is throughput per kilowatt in terms of tokens per second per kilowatt. This is basically saying, if I have a 50-megawatt data center, how many tokens per second can I push through that data center? Then on the x-axis, what we have is interactivity. So if I’m a user getting tokens being processed in that data center, how quickly can I get those tokens as my personal experience?

You would think those two things at first order would be very related, but they’re actually at odds. That’s because the longer you give me to serve a token, the more efficient I can be with how I generate that token. But if you say, no, I need this token really fast right away for Natalie’s use case, then you have to pull out all the stops to get that token to that user as soon as possible. So we show these things as a frontier where you can optimize for one or the other or somewhere in the middle, but you’re never going to get something that’s fully in the upper-right quadrant because they’re fundamentally at odds.

So let’s now look at what we did with the d-Matrix side of things. We show three different Pareto frontiers for three different configurations for the same workload. This workload is running GPT-OSS 120B, 8K input sequence length, 1K output sequence length. What we’re showing is the frontiers for that workload.

We have three configurations here. The green one is a traditional pre-fill/decode disag on GPUs. So we can see that that offers a certain tokens per second at a given interactivity level. Usually the way people think about it is, my requirement is that my users need at least X tokens per second. And then from there, I try to push the throughput as high as possible. So you would set a latency budget and then try to maximize throughput given that latency budget.

A common technique that people adopt to speed up their workloads is they introduce speculative decoders. What speculative decoders do is they say, wow, running decode is really slow and inefficient, because I have to run the full model for every single token. But sometimes I could maybe use a smaller model, or something like Eagle which works a little bit differently, to guess at the next token. And maybe if I could guess multiple tokens in a row, then what I could do is take my large model and verify if they’re correct. Because it’s a lot more efficient to verify five tokens in a row and say, are these correct, than it is to actually generate them one by one with that large model.

So what we have in the blue line is a GPU-only speculative-decode flow. What we can see is, compared to the pure pre-fill/decode disag, it offers a shift in the Pareto frontier that’s quite significant. That’s why folks are adopting speculative decoders — because it really, really helps deliver better experiences, have more capacity, et cetera.

But what we did is we decided to take this a step further and say, okay, what if we take that same speculative-decode setup, but instead of running all of those parts on a GPU, we’re going to take the spec-decode part and run it on d-Matrix Corsair? That’s because d-Matrix Corsair offers a lot of on-chip SRAM, and it’s really, really fast when you can store the model weights in memory. So by running that smaller 1.6B spec-decode model on the Corsair — even on top of the blue line, which is already quite optimized — we see a dramatic, dramatic performance benefit. At a reasonable point on the interactivity side or on the throughput side, you can get like a 4× benefit.

Awesome, interesting. Zooming back out for listeners — we talked about taking parts of the workload and scheduling it to the right hardware. So d-Matrix’s chip is one of those SRAM-heavy ones. And so if there’s part of this workload, which is like, can you do this really quick guessing and see if you get it right in advance so that you have to do less work, what if that just ran on an SRAM-heavy chip that could do this guessing really fast? And that allows — now I stole the chart — where it’s like pushing it horizontally, for the same throughput per kilowatt it would unlock a much higher interactivity. I know you had other charts that said, of course, depending on what people are trying to do, if they’ve got a latency budget, they could also stay at a fixed interactivity if they wanted and get a higher throughput. So serve more customers more efficiently.

Natalie: Right. You can choose: do you want your customers to get their tokens four times faster, or do you want to serve two or three times as many customers at the same latency?

Well said. I just clicked one other slide which you showed that you can get even more of an unlock if you use a verify stage of 20 tokens instead of five. But maybe on this slide, a point here is for someone like a sovereign — it shows that you guys are thinking a lot about how to tune infrastructure, how to run little experiments, how to take the latest and greatest like speculative decoding and then take the latest and greatest chips like a d-Matrix one and figure out what are all the right knobs so that your customers can just come to you and say, make it faster, and you say, we got you.

Natalie: We all have limited capacity. We need to serve a lot of tokens. Inference is supposed to become the dominant workload over training this year. So what we are doing here, this is one example of the type of disaggregation that we can do and the type of hardware we can deploy across, but it’s not limited to this. This is to illustrate what you can get when you adopt a heterogeneous stack.

Beltir: And this only starts from customer-back, because every customer has unique requirements. They have different workloads. They run MoEs, some of them sparse experts. That changes what type of disaggregation methods you need to apply. That also changes which hardware combination would be the best for that particular workload. That’s also something that we can help with the customers as we learn more about their workloads. Because giving them an unlimited end option is also not the solution. There needs to be a limited solution space also for it to be cost-advantageous. So what is that right optimal combination for that particular workload?

We start from the other way around, saying, okay, this is the customer, this is the workload, this is their constraint — either latency, or power, or throughput. This is the characteristics of the workload and their customer base. So based on that, we run simulations and tell them, here is what we think would be the best architecture mix of hardware for you, and based on your needs, this is how you can push the frontier and what limits you can get. And then based on that, we start designing what is the smallest data center stamp that is required, because it has to be a repeatable implementation to be able to scale. You need this to be in the same data centers, you need to network them. What is that network topology? And then build a path to scale that implementation. So this is an end-to-end partnership with the customer.

Why Gimlet Differentiates in the neocloud Landscape

Okay, that’s super interesting. I love that you start with the customer needs first and build to their needs to get the most optimal experience for them. Also, in the back of my mind, one of the things I’ve been thinking is — if you guys are essentially like a neocloud, there are tons of neoclouds, how do you differentiate, who captures that in the long run? Some are just bare metal. So it’s like, okay, how can you differentiate there? But what I heard you saying is, no, no, no, we are a full-service, almost consulting partner where we’re helping you design your data center footprint, and we’re helping you optimize it, and we provide the software platform that will help do this for you. So it’s very differentiable compared to others in the neocloud space.

Beltir: Correct. And also if you think about us versus everyone else in the neocloud space, most of them are today backed by one silicon vendor. And in return, they gave significant equity. So it’s hard for them to diversify their silicon ecosystem. So it’s hard for them to do the mixing and matching that we do. And if you look at what’s going on in this ecosystem, inference prices are coming down, so everybody is getting more pressure on their top line. They have very limited opportunity to diversify supply chain. They have no negotiation. Hardware amortization is usually 70% of their annual costs. So they have very little room to optimize their bottom line. This is why the software innovation you see them announcing all these disaggregation methods — because they’re trying to create a sustainable business model.

What’s slightly — I will say grossly — different is we have two different dimensions that we work with the customers. A is the end-to-end software service and the scaling motion with large-scale customers. But for customers who are up and coming, who cannot yet commit, we also build our own neocloud with fundamentally different economics. Because from a bottom-line perspective, we actually have a supply-chain diversity that optimizes our bottom line. From a top-line perspective, since we can offer very differentiated token performance, we can also command a price premium rather than trying to race to the bottom from a pricing perspective. So if you ask us, Beltir, are you differentiated? I think we are very, very much differentiated. And having these two different dimensions in our business model gives us the liquidity and the financial stability, because one can fund the CapEx investment of the other.

Fascinating. I really like it. You make the very interesting point, which is the incentives that other people have or are bound by that would prevent them from going in this direction. You can buy whatever silicon makes sense, and then of course you have the chops, the software chops in-house, to disaggregate whatever workloads across that hardware as you see fit. Control your destiny.

Beltir: The other thing — the neocloud, I see it in two dimensions. You have CoreWeave-type people whose core strength is buying GPUs, data centers, and offering that as a bare-metal-as-a-service, but they lack the full-stack experience today. They’re trying to acquire companies to figure that out, but it’s really a long journey, very hard journey to mix and match acquisitions to create a unified stack.

The other ones are like Together and Fireworks, who are only software and trying to acquire the capacity from usually one silicon vendor’s infrastructure providers. So we don’t want to be either of them. We want to offer the end-to-end experience with two different business models that are very complementary to each other.

Series A, Hiring, and What’s Next

Nice, I love it. Last slide. Earlier this slide when we talked about business models, it also had the headline that I think in March you announced you raised the Series A. And then I saw you obviously are hiring people — if people click on view open roles, there are several roles. So tell us a little bit more about what you’re looking for and what’s up next for the rest of this year.

Natalie: I’d love to talk about that. We are very focused on hiring right now. We’re set to — I forget how many X we’re going to, is it triple, quadruple? It’s something crazy like that by the end of the year, because we are scaling rapidly to meet the demand that we’re seeing. So if you want to join a company that is in crazy-scale mode, this is a good time to join, because you’ll still be part of the old guard because we’re in that rapid growth phase.

Who are we hiring? In terms of number of roles, engineering is the biggest one. We are looking for people who know how to do high-performance AI systems across the stack, whether that’s by working on our scheduler, working on our compiler layers, working on how do we monitor these incredibly complex distributed systems, how do we write optimized kernels, how can we leverage AI to automate some of the optimizations that we’re doing ourselves. And then also general builders — folks that are kind of Swiss Army knives that love to go up and down in the stack and contribute to different parts. Definitely reach out to us if you’re interested. I will note we are an in-person office based in San Francisco.

Beltir: Just final words from my end. This is a crazy fast-growing rocket ship right now, because in many startups there’s always a concern, do I have product-market fit. We proved there is product-market fit. We are very well funded. We’re on the fast pace to get accelerated capacity. Most people are struggling with supply-chain problems — given our value proposition, that’s the least of our problems. Right now, our biggest problem is getting the right people to execute and deliver the customer commitments we have. So we are hiring across the tech stack from low-level kernel engineering to higher levels of software engineering. We’re building an end-to-end cloud stack, not a bare-metal-as-a-service. So across the tech stack, if people are interested, roles are open. We are looking for creative and innovative engineers who are looking to jump on a crazy growing ship.

Nice. Good pitch.

Natalie: I’ve been at startups most of my career, and I’ve been blown away by the scale of the opportunity here, and I pinch myself almost every day. We really look forward to welcoming our new colleagues.

Yeah, awesome, I love it. Having product-market fit, understanding your business model and having it figured out, and then of course just the macro environment that we’re in where there’s so much demand and so little supply, and being able to come in and figure out a unique way to make the most out of the constraints. Pretty exciting. So hey, I learned a ton. Thank you so much, Natalie and Beltir. This was really engaging and I know the listeners will walk away having learned something. So thank you.

Natalie: Thanks so much, Austin. It’s been a great conversation.

Well... allegedly. There are rumors of a warpage problem on the 4-die package, and chatter that TSMC is leaning on panel-level packaging (CoPoS) to deal with it, maybe even a fallback to a 2+2 config. Hold that thought.

So how do you connect four pieces of silicon together such that they behave electrically like a single chip? That’s the question of advanced packaging. And as AI accelerators keep getting bigger, the packaging itself is becoming the dominant cost variable in the bill of materials.

Today we’ll cover:

• A primer on 2.5D advanced packaging, and the reticle limit that started the whole story

• TSMC’s CoWoS family (CoWoS-S, CoWoS-R, CoWoS-L)

• Intel’s EMIB

• EMIB vs CoWoS-L

What is the reticle limit?

The way you make a chip more powerful, historically, has been to make it bigger. More transistors, more compute, more parallelism per die.

The ceiling on “bigger” is the reticle limit: the largest area a lithography stepper can pattern in a single exposure. About 26 mm × 33 mm, or roughly 858 mm².

NVIDIA’s H100 was already pushing the reticle. Blackwell broke through by stitching two reticle-sized compute dies together into a single GPU:

Once you cross that line (i.e. when one die isn’t enough) you need a way to physically connect multiple dies so they behave electrically like a single chip:

That’s advanced packaging. And as accelerator sizes grow, the cost of the packaging itself becomes a dominant economic variable.

What is 2.5D packaging, and why is it called that?

2D is one or more dies sitting directly on the organic substrate. No interposer, no bridge. Routing runs through the substrate itself.

That covers classic monolithic packages and chiplet designs where the dies talk through substrate traces. The constraint is density. Substrate pitch is coarse, so you get moderate die-to-die bandwidth, not the tight compute-to-HBM coupling AI accelerators need.

2.5D adds a passive silicon routing layer between the dies and the substrate. That can be a full silicon interposer, a silicon bridge embedded in the substrate like Intel EMIB, or silicon bridges inside an RDL interposer like TSMC CoWoS-L. It carries fine-pitch routing and sometimes TSVs, but no working transistors. It moves signals, it does not compute.

That is what makes tight compute-to-HBM coupling possible, and it is the dominant architecture in modern AI accelerators.

3D goes vertical. Silicon stacked on silicon — AMD 3D V-Cache, Intel Foveros, TSMC SoIC.

To get nitpicky: 2.5D with CoWoS-S is also technically “silicon on silicon”, but the interposer underneath is passive. Think of 3D as active on active and 2.5D CoWoS-S as active on passive.

HBM stacks are 3D internally, though they usually sit in a 2.5D package.

TSMC’s CoWoS family: three variants

Chip-on-Wafer-on-Substrate (CoWoS) is TSMC’s umbrella for 2.5D packaging. There are three commercially relevant variants. They differ mainly in how much silicon is used for interconnect.

CoWoS-S: full silicon interposer

The original. Entered production with Xilinx’s Virtex-7 2000T FPGA around 2011, where four FPGA slices were stitched together on a passive silicon interposer:

How it works:

• Multiple active dies sit on top of a large passive silicon interposer

• The interposer sits on the organic package substrate below

• The interposer carries fine-pitch metal routing for dense lateral interconnect, plus Through-Silicon Vias (TSVs) that route signals and power vertically down to the substrate

Important nuance: the interposer is not a “logic chip” in the compute sense. It’s processed on a mature silicon node optimized for routing density and TSV formation, not transistor performance. No logic, no transistors doing work on it.

Think of it as a tiny circuit board made out of silicon. Same job as a PCB, just at lithography pitch instead of PCB pitch, with several fine-pitch metal layers plus a forest of vertical vias.

Electrically, this gives you tens of thousands of short, fine-pitch interconnects between neighboring dies, at far lower latency and power than routing the same signals through an organic substrate.

Here’s what that original Xilinx example looked like:

HBM made this silicon interposer the default approach for flagship AI parts. The HBM interface is too wide and too dense for conventional packaging. Once GPUs adopted HBM (notably AMD’s Fiji / Radeon R9 Fury X), large silicon interposers became standard across every high-end AI accelerator that uses HBM:

There’s an economic problem though. The silicon wafer is being consumed for routing, not compute. That’s expensive. As HBM stack counts grow and compute reticle counts grow, the per-package silicon bill grows right alongside them.

CoWoS-R: organic RDL interposer

TSMC’s response to the silicon interposer cost is to build the routing in Redistribution Layers (RDL) of organic material instead of silicon:

This is cheaper, but isn’t a silver bullet. Organic processes have wider lithographic tolerances than silicon. Trace pitch widens, layer count climbs, and the assembly can’t match the bandwidth density that an HBM-to-GPU interface demands.

Thus, CoWoS-R is useful for cost-sensitive products that don’t need the densest die-to-die interconnect. It cannot carry flagship AI accelerator workloads on its own.

Trade-offs!

CoWoS-L: local silicon bridges in an interposer

TSMC’s current frontier, which places small silicon bridges only where you need high-density routing (compute-to-compute, compute-to-HBM). Use cheaper organic RDL material everywhere else on the interposer.

The bridges sit inside the interposer. The interposer is then attached to the package substrate as one large piece.

This is the TSMC architecture used for Blackwell-class accelerators.

The move is elegant balancing of trade-offs, with silicon where you need bandwidth, organic where you don’t. Beautiful in principle. The catch is that you’ve still got a separate interposer to build (an RDL carrier with those silicon bridges embedded in it), and then you have to dice it and attach the whole thing onto the package substrate. Two pieces, two attach steps.

Intel’s EMIB

Embedded Multi-die Interconnect Bridge (EMIB) shares CoWoS-L’s central insight (silicon only where you need it) but resolves it very differently.

EMIB skips the interposer entirely. The silicon bridges are embedded directly into the organic substrate of the package:

Here’s a cleaner version from Intel’s foundry blog:

Note that EMIB has only two layers. Dies and substrate.

EMIB-T and EMIB-M

Worth noting quick: Intel has next-iteration variants. EMIB-T adds Through-Silicon Vias through the embedded bridges themselves, which lets power and high-speed signals flow vertically through the bridge, not just laterally. HBM-heavy designs increasingly need this. EMIB-M integrates MIM (Metal-Insulator-Metal) capacitors into the bridge for on-package power decoupling.

Both are direct descendants of the same “embedded in the substrate” architecture. Worth a watch on Intel’s recent EMIB-T/M explainer:

EMIB vs CoWoS-L, side by side

So EMIB and CoWoS-L are both bridges right? Which is better?

• EMIB: silicon bridges embedded directly in the organic substrate. One piece, one attach step.

• CoWoS-L: silicon bridges embedded in an RDL interposer; that interposer then attached to the package substrate. Two pieces, two attach steps.

  

That one difference leads to these value props:

1. Cost

EMIB doesn’t have a separate interposer to amortize at all. The silicon bridges are small dice embedded in a package substrate that already exists.

Let’s be precise about what’s being compared here. Every flip-chip package, EMIB or CoWoS, sits on a panel-made organic substrate. CoWoS-L additionally builds and attaches a separate interposer (an RDL carrier with small silicon bridges embedded in it) between the dies and that substrate. That extra interposer, plus the extra process steps and the extra attach, is the cost difference. And in current-gen CoWoS-L that interposer is built in round-wafer format, so the panel-vs-wafer waste from the next section applies to it too. EMIB just doesn’t have any of it. The bridges are cheap because they’re tiny, and you get thousands per wafer.

Process steps eliminated:

• Interposer build

• Interposer dicing

• The interposer-to-substrate attach

That’s three places where cost and yield could go wrong, but won’t for EMIB, because they don’t happen.

2. Panel utilization

This is a big one, even if it sounds boring, and it matters more the further along the roadmap we go.

Silicon interposers are cut from 300 mm round wafers. Packages are rectangles. Rectangles on round wafers leave significant edge waste, and the waste fraction grows with interposer size. The bigger the interposer, the more wafer area you throw away at the edges.

Substrates use rectangular panels (a common size is roughly 510 mm × 515 mm). Rectangles tiled into a rectangle. The math is much friendlier.

Intel cites approximately 60% wafer utilization for interposer-class CoWoS versus approximately 90% panel utilization for EMIB:

That’s the cost headline. On a flagship part with a multi-reticle package, you’re looking at a substantial cost-of-goods delta before counting anything else.

And remember the Rubin Ultra rumor from up top? The fix that keeps coming up is CoPoS (Chip-on-Panel-on-Substrate), which is TSMC moving its advanced packaging off round wafers and onto rectangular panels. CoPoS isn’t EMIB. TSMC keeps its carrier-and-RDL approach; it just runs it on a panel. But the package got too big for a round wafer, and the fix is panels. Intel’s package substrate was a rectangle from the start.

3. Scalability past the reticle limit

This one is just geometry.

A reticle is roughly 26 mm × 33 mm = 858 mm².

• A 5-reticle complex (think Blackwell-scale, roughly Rubin-ish): ~4,290 mm², or about 43 cm².

• A 14-reticle ceiling: ~12,000 mm².

A 300 mm wafer has π × (150 mm)² ≈ 70,686 mm² of total area, and the usable rectangular yield is meaningfully smaller once you account for edge waste and dicing kerf. At 14-reticle interposer sizes, you’re approaching one interposer per wafer.

One. Interposer. Per. Wafer.

At that point the interposer absorbs the entire cost of the wafer. Packaging cost stops scaling and starts cliff-diving in the wrong direction.

EMIB stretches in X and Y across say a 515 mm × 510 mm panel (≈ 263,000 mm² of usable area). The “one interposer per wafer” problem doesn’t arise.

So the cost curves diverge. The larger the package, the wider EMIB’s margin gets.

And packages keep getting larger every generation.

4. Yield through smaller bonded pieces

Bonding a single 5-reticle silicon interposer onto a substrate is a tough operation. You’re moving a ≈ 43 cm² silicon piece through reflow, and silicon and substrate have different coefficients of thermal expansion. Warpage at that size is a yield-limiting problem. Remember that alleged Rubin Ultra issue above?

EMIB attaches dies individually to the substrate. Each attach is small, locally thermally controlled, and decoupled from the others.

Small-piece bonding is inherently higher-yield than big-piece bonding. The yield advantage compounds with package size for the same geometric reason the cost advantage does.

That’s the case for EMIB on the merits. For paid subscribers: three forward scenarios with my thoughts, the “TSMC will just do CoPoS” pushback, the Amkor partnership that adds a second source for EMIB (and why it’s a 2028 story, not a 2026 one), and what it all means for how you read Intel Foundry’s hand. If those tickle your fancy, keep reading.

Read more

CEO Jim Anderson: A significant portion of the sequential growth we expect in the current quarter is driven by 1.6T adoption. As a reminder, earlier this year at OFC, we were the only company to demonstrate 3 different types of 1.6T transceivers based on 3 different types of laser sources; silicon photonics, EML and VCSEL.

My “pay attention” radar went off when I read this. Why all three? Are they hedging bets here? Which bet is better? What are competitors betting on? Feels like a tech inflection opportunity that could shake up industry and market dynamics. Will one or more of these fail?

Answering those questions first requires foundational knowledge in each technology and current market dynamics.

Let’s start with VCSELs for datacenter communications. We’ll start simple.

• What is a VCSEL?

• Why has it dominated the short reach data center market for decades

• Why is the 200G-per-lane jump giving everyone heartburn?

What Is a VCSEL?

VCSEL stands for Vertical-Cavity Surface-Emitting Laser. The “vertical” part is the giveaway. A VCSEL is a tiny semiconductor laser that fires light straight up out of the top of the chip:

This contrasts with edge emitters, which emit light from the edge of the laser:

Honeywell began VCSEL research in 1993 in Minneapolis, moved it down to Richardson TX in 1995, and launched the first commercial product in 1996, initially targeted for datacenter communications. The history is a fun chain of acquisitions: Finisar acquired Honeywell’s VCSEL Group for $75M in 2004 (what a steal), II-VI acquired Finisar in 2018, and then II-VI acquired Coherent Inc. in 2022 and renamed the resulting entity Coherent Corp. So when Coherent talks about VCSELs today, it is effectively the heir to the entire 30-year US VCSEL development arc!

Check out this great infographic for more history. And note the volume of VCSELs shipped even a decade ago.

Commercial adoption was largely propelled by the Gigabit Ethernet (IEEE 802.3z) and Fibre Channel standards. VCSELs were a significant improvement over edge-emitting lasers in reliability, speed, cost, power efficiency, and manufacturability at similar wavelengths. For the first decade of existence, VCSELs were mostly used for short-reach datacom.

In 2017, high-power 2D VCSEL arrays were incorporated into the iPhone for Face ID, and consumer applications have since become the primary drivers of VCSEL production volume. Today VCSELs are everywhere:

• Smartphone face ID and 3D sensing. When your phone unlocks by reading the geometry of your face, that’s a VCSEL array projecting structured infrared light at you.

• Automotive LiDAR. VCSEL-based LiDAR is one of the architectures that self-driving systems use to build 3D maps of the world.

• Optical mice and touchless sensors. The dot of light under your computer mouse is (very often) a VCSEL.

• Short-range fiber communication and Active Optical Cables

ams OSRAM has a nice walkthrough of VCSEL technology for various applications, and their BELAGO 1.2 dot projector is a representative product.

As you know, below ~10 meters, copper cables still dominate but struggle as speeds increase:

Beyond a certain distance, single-mode fiber takes over to avoid modal dispersion. That distance is data-rate-dependent and has been compressing rapidly, historically around 300 meters at 10G-era links, around 100 meters at 25G on OM4, and dropping to roughly 30-50 meters at 200G PAM4.

That middle slice (multimode fiber, between where copper gives up and where single-mode takes over) is VCSEL territory. Note that the slice itself is shrinking as data rates climb.

Another important note: What matters for the analysis later is that VCSELs have a very mature supply chain. Billions of VCSELs ship per year (mostly into consumer devices). The reasons VCSELs got pulled into all those applications is because VCSELs are compact, energy-efficient, array-friendly, wavelength-stable, and uniquely testable on-wafer before any packaging happens.

Why We Care About VCSELs Right Now

VCSELs are at an inflection point in 2026. AI data centers consume short-reach optical interconnect at a scale nothing in computing history has matched, and the industry is racing from 800G to 1.6T with 3.2T on the horizon. To get there, incumbents are trying to double each lane from 100G to 200G, and 200G is where the current path struggles. And how would it even double again to 400G/lane?

This post (the first in a series) will explain why VCSELs have dominated short-reach data center optics for two decades and what, specifically, is breaking at 200G.

We will cover:

• The value props that built VCSEL’s moat at short reach plus wafer economics

• What came before VCSELs, and why edge emitters still own long-haul

• The named market players holding up today’s 800G/1.6T short-reach supply chain

• Why the 200G-per-lane transition is breaking the moat

Read more

On Monday, Microsoft and OpenAI amended their partnership. OpenAI is no longer exclusive to Azure; the license is now non-exclusive; OpenAI can serve all its products on any cloud. Like AWS.

On Tuesday, AWS launched Bedrock Managed Agents (powered by OpenAI) along with the new Connect family of agentic enterprise apps.

The same day, Ben Thompson published a joint Stratechery interview with Sam Altman and Matt Garman explaining the partnership.

By the way, AWS CEO Matt Garman is on X now! Less than 3K followers at time of writing. That will surely change:

Lastly, on Wednesday AWS posted very strong Q1 2026 results.

What to make of it all?

I’ve long been a bit bearish on Amazon. AWS is a phenomenal business, but at the corporate level it pulls Amazon’s margin up because retail is a tough low-margin business. Google is the inverse: GCP pulls Alphabet’s margin down because Search and ads print money. I’d rather be Google.

And my bearishness persisted. I hoped Amazon would see the AI shift as an opportunity to pour resources into AWS and expand its dominance and profit dollars, but instead, Azure and GCP ran ahead in GPU capacity. Plus Azure had an exclusive relationship with OpenAI, and Google’s own Gemini models seemed to keep up.

Man. If alpha was in the model, and GPUs powered frontier models, AWS didn’t have enough of the asset that mattered. Although Anthropic was showing some early promise…

But the agentic era is quickly changing things. No truer words, right?

The agentic platform, as I’ll explain, is sticky and value-accretive. AWS has an opportunity here. Moreover, Amazon’s Graviton and Trainium are well-positioned to serve it cost-effectively. And most importantly, Amazon’s own e-commerce operations are the best possible first customer for agentic platform primitives. But a platform needs an ecosystem, and SaaS is dead right? Nope. AWS shows a survival playbook for other operationally-grounded SaaS companies. And Connect family revenue is gravy on top, too.

Right now, AWS’ future looks bright; if they deliver on the vision, we’re in a period with dual mispricing opportunities for investors:

Read more

Will this hold? Officially in the AI supercycle? Let’s dig in and think through the implications.

From the earnings call:

Wendell Huang: In US dollar terms, revenue increased 6.4% sequentially to $35.9 billion, slightly ahead of our first quarter guidance. Gross margin increased 3.9 percentage points sequentially to 66.2%, primarily due to cost improvement efforts, a high capacity utilization rate, and a more favorable foreign exchange rate.

Q1 is usually the quarter when margins decline as smartphone sales slow after the holidays, and thus fabs run a little cooler and gross margin gives back 100 to 300 basis points. We can even see that was true this quarter for smartphones, down 11% QoQ:

Read more

In this interview, we walk through Meta’s ads infrastructure from first principles. A few things that surprised me:

• Recommender workloads have a different compute-to-memory ratio than a standard LLM GPU, and this difference gave rise to MTIA custom silicon

• Retrieval isn’t a generic workload either. Meta’s scale makes it memory-bound, which is why Andromeda got its own custom NVIDIA Grace Hopper SKU that Meta co-designed

• Meta’s adaptive ranking model is an LLM-scale recommender (~1 trillion parameters) served at sub-second latency. It’s distilled from GEM, Meta’s Generative Ads Recommendation foundation model, and scales compute per user based on interaction history length

• Consolidating N ad ranking models into one (Lattice) improved performance, not just cost. A single model trained across varied objectives outperformed the specialized ones

• LLM-written kernels (Meta’s KernelEvolve) flip the economics of heterogeneous fleets. Demand for software engineering is going up as the price comes down, and Meta now wants ~100x more optimized kernels per chip

We also cover how Meta’s GenAI and recommender systems teams cross-pollinate inside Meta, and what Meta’s infrastructure looks like two years out.

This interview is lightly edited for clarity.

How Meta’s Ad System Works

Hello everyone. Today we have a special guest, Matt Steiner, VP of Monetization Infrastructure, Ranking, and AI Foundations at Meta. Welcome, Matt.

MS: Thanks, great to be here with you, Austin. Thanks for having me.

What I wanted to get out of this conversation is to better understand Meta’s core advertising business and then how that drives infrastructure decisions. I’m going to assume listeners know nothing and we’ll walk through from first principles. At the highest level, how do ads work? What are the backend models that power Meta’s ad stack?

MS: Maybe let’s start with a quick overview of how the ad system works. On a very high level, an advertiser shows up and they say, “I have some creatives with some copy and I want to show them to some people.” Sometimes they pick explicitly who they want to show them to. Sometimes they say to our ad system, “show them to whoever is most likely to convert for the objective that I specify” — whether the objective is the person visits my website, the person adds something to a shopping cart on my website, or the person actually clicks buy on my website. Those are all different objectives. Advertisers can optimize for different things.

Once the ads are created, it is our job to record who these ads should be shown to. So we produce a big database and it says, “here are all the people that the advertiser would have wanted their ad to be shown to,” and we record in each person’s little mini database, “this is an ad that could be shown to Matt the next time Matt logs in.” Of course, that list of ads that could be shown to Matt the next time Matt logs in is very, very long.

So when Matt logs in and our front end asks for an ad, whether that’s on your mobile device on Instagram or Facebook, on the web — each front end queries our backend system and says, “give me the best ads to show Matt next.” The request goes through our systems and arrives at our indexing system, and our indexing system fetches all the ads that could be shown to Matt. That is where a piece of technology that we’ve talked about recently called Meta Andromeda comes into play.

A long time ago, we had a much shorter list of ads that could be shown to Matt. Today that list is extremely long, and to be able to process all of the ads that exist in that list we need to use a fairly powerful system. We worked with our hardware partners at NVIDIA and designed a custom hardware SKU with some GPUs in it, and we co-designed a machine learning model that runs specifically on that hardware SKU for the purposes of best assessing which ads are the top N ads to rank for Matt.

In the ads serving process, the two large steps are basically: find ads that could be shown to Matt, and then rank them to produce the top ads to be shown to Matt.

Andromeda operates in the first stage, which we call retrieval, and it uses a powerful machine learning model that has embedded some of my interests and past interactions to personalize which ads should be retrieved for me. Because not every product that is advertised to me is going to be a product that is interesting to me. So we’re basically sub-selecting the products and creatives that might be interesting to me in order to return to the ranking system to rank those.

The next step is ranking, where we apply these large and powerful machine learning models to figure out what is the right order of these ads in terms of highest conversion probability times expected value for advertisers. The ad system has a number of ranking models and they rank different ads based on the objective functions for the advertiser, and we have been on a long journey to consolidate those into a single ranking model using a technology we call Lattice.

The advantage of combining ads ranking models into a single larger model is of course cost savings. You don’t have to keep N copies of user interests in each machine learning model. You can keep one copy of a person’s interests in that machine learning model, which saves memory. You can compute the subnets for a machine learning model once instead of repeatedly computing the same subnets across a bunch of different models. You just do one computation. It’s more computationally efficient to have a single model. And then the other advantage is performance — a machine learning model trained on more data with more varied objectives performs better than a smaller machine learning model trained on all the data for that objective, partly because of the compute advantages, partly because of the memory pressure advantages, partly because each piece of data has some additional signal associated with it that the machine learning model can use to improve its own performance.

So: Lattice, consolidation. And then further along in the consolidation journey, we have built GEM, our Generative Ads Recommendation Model, which is our foundation model that we’ve tried to train on all of the data that’s available for Meta’s ad system to use to improve the probability of accurately predicting what somebody’s going to be interested in, what they’re going to convert when we show them an ad for achieving an advertiser’s objective. This large foundation model was then used to distill into smaller models that we could serve for specific purposes, encoding as much information as we can from the larger foundation model.

Now, like with any system, some people use it less and some people use it more. There are people that are very interactive with brands and content and ads. They’re commenting on the ads, they’re liking the ads, they’re interacting with the brand, they’re buying things from the brand. Those power users actually have much longer interaction histories with a brand or with all the brands together. It turns out that in our original architecture design, we did not have enough compute available to process all of those interactions given our extremely limited latency budget. For example, when a person shows up in a Meta property, we want to make sure that their feed loads and their ad loads in that feed in a certain fixed latency budget — let’s call it roughly one second. We want to have sub-second latency for all of our average retrieval requests. That means we can only process so many interactions when evaluating or inferring that machine learning model.

Recently, we’ve built a new ranking model called the adaptive ranking model that substantially varies the amount of compute used to evaluate the model based on how long a sequence from a user is of their interaction history with a brand or all the brands that are advertising on Meta systems. That way we can use a dramatic amount more compute for users with longer interaction histories and meaningfully increase the accuracy of our predictions about what they’re going to interact with next. That drives better results for advertising partners and much better experiences for the people that are seeing those ads. It’s all through the magic of right-sizing the compute and memory associated with each one of those requests, and right-sizing the model based on the amount of data that’s available to evaluate for a particular person.

Okay, this is so fascinating, there’s so much here. At the highest level, you broke it down to retrieval and ranking — retrieval was Andromeda, ranking was Lattice. With Lattice, you talked about having lots of models but trying to simplify that down into one model for many reasons. And meanwhile, the whole backdrop here is — what kind of scale are we talking about again? Three-plus billion daily active users?

MS: That’s exactly right. More than three billion daily active users across Meta’s properties worldwide. A lot of people seeing a lot of organic content in their feed, a lot of paid content in their feed, and interacting with both.

Take me back to GEM and remind me — we have retrieval and ranking, and where does GEM fit in?

MS: GEM is our foundation model. It’s the model that we train with all of the data that we can use for training to produce the largest, most sophisticated, most prediction-accurate model possible. At the same time, the model is so large it’s not servable effectively. So the model has to go through a distillation stage where a lot of the core learnings of the model are distilled into smaller models that are servable.

The next step after that was to try and make the largest possible servable model on the most powerful inference hardware we have available, to produce the most accurate predictions specifically for those users who are power users. They have long interaction histories with brands and content and interests that we can really do a lot better for — deliver them much better experiences and deliver advertisers much better predictions and consequently return on advertiser spend.

Long User Histories and Adaptive Ranking

Nice, and that’s where adaptive ranking fits in. This is really interesting, because I think people are starting to get used to the idea of a foundation model that’s so big you can’t serve it, and then the consequences and trade-offs of having smaller models that are servable. For listeners thinking of generative AI, they might be thinking of smaller models that respond faster but aren’t as “intelligent.” Broadly when people are thinking about generative AI, they’re thinking about optimizing for intelligence or for interactivity — how quickly does it respond. You talked about latency, but you also talked about being willing to spend more compute at inference time to get a better outcome for the advertiser and a better experience for the user. Can you talk more about the outcomes? Why does adaptive ranking and spending more compute because you have that longer history yield a better outcome?

MS: Maybe one way to think about this is: imagine that you’re married and you have an anniversary, and every year you buy something for your spouse — something that they like that’s in their interest set that’s not necessarily in your interest set. If you can look at a long interaction history for a particular person, and you see, “every September they buy this particular class of item,” you don’t have to even know that it’s their anniversary, but you can see in that long interaction history, every September they buy something in this category. Then you can use that information to make a better prediction for what they’re likely to purchase in September.

That’s one example, but maybe you have a history of purchasing specific things in specific months corresponding to your children’s birthdays or a holiday or an anniversary. You can see how looking at longer sequences of interactions can deliver much-improved predictions about what a person is likely to want and then what a person is likely to purchase based on those longer interaction sequences.

But you can only process those longer interaction sequences if first, you’ve stored longer interaction sequences, and second, you have the computational power available at serve time to be able to process that whole interaction sequence when a person logs in. Not everybody has long interaction sequences. Not everybody interacts every month with an advertiser, but some people do, and where the data is available to deliver dramatically improved experiences for those people, you of course want to give them the best possible experience you can. That is a function of whether you have the compute available to be able to process all that information within that latency budget through parallelization, etc., that GPUs and large-scale GPUs in the inference stack now allow us to provide for people. Better providing which products and services people are interested in delivers better results for our advertising partners as well, because we’re just matchmaking. We are matching the person who wants to purchase a thing with an advertiser who has the thing to purchase.

Yes, that makes a ton of sense. For me, you’re saying: if I only look temporally at the last month of what you’ve been doing, I could give you some ads. But you’ve been on Facebook since back when you had to get invited — so if I could look all the way back, maybe there’s interesting trends. But of course the trade-off — I’m thinking about an analogy to generative AI, which everyone can relate to. It’s kind of like context. I want a big model, I want to give it a ton of context, but that’s expensive and takes time. And with user-centric social apps, you’re thinking a lot about latency. So you’ve got that constraint of what is the most context I can give it, the biggest model I can give it, but still do it in sub-one-second. That’s a perfect segue to ask you more. You talked about co-designing with NVIDIA, you talked about GPUs. Take me back — did this stuff run on CPUs at one point? How has that evolved?

From CPUs to Custom ASICs

MS: Back in the day, retrieval of course ran on CPUs. And back in the day, even ranking ran on CPUs. There was always a push to deliver more compute for both retrieval and ranking, because the more compute available, the larger, more complex machine learning model we can evaluate, the larger the user history long-sequence context windows can be passed into those models, delivering better predictions.

We’ve been on a long march through smaller CPUs, medium-sized CPUs, larger CPUs, custom ASICs, GPUs, more sophisticated and powerful GPUs, more sophisticated and powerful custom ASICs. This is all in service of delivering better results for our customers at a reasonable cost to our business so that the ROI works out on both ends for both our advertising partners and Meta.

Okay, that’s amazing. What I heard you saying was: it’s been a long history for Meta of asking “how can we get more compute to serve better ads?”, which is a win-win — you’re in a marketplace with users and businesses and you’re sitting in the middle. This idea of using compute to do predictions better has been the story of Meta’s business for quite some time.

MS: At least the last ten years, we’ve been investing really deeply in performance-optimizing the hardware, the networks, the data center designs, the silicon chips themselves, the machine learning models, the software infrastructure, the tooling associated with them. It’s a very large, complex optimization CP-SAT problem that we have to satisfy to deliver the best results for our customers and for the people that use our products and services. It’s a really fascinating technology problem in addition to a business problem.

Co-Designing with NVIDIA

Yes, indeed. It’s an intersection of both. What did the practical process of hardware-software co-design look like when you were developing the retrieval engine, like with the NVIDIA Grace Hopper?

MS: We sit down with our partners and we say, “this is the amount of compute that we want to target for this particular use case. This is the latency budget. What are the configurable blocks you have in your portfolio that you could considerably make into a SKU, whether it’s a chip level or a hardware level, that would work for this particular use case?”

Our hardware partners have various configurations of machines and chips and boards available that they are willing to build in certain configurations. We looked at that and we said, “given the retrieval problem itself, it’s going to require a huge amount of memory. It’s maybe a little bit more memory bound than it is compute bound. So we need a lot of memory. We need a lot of specifically high-bandwidth memory, so there’s enough memory channels to keep those GPUs saturated when they’re doing that computation.” We wind up with a SKU design that is optimized for the retrieval space where it has the right amount of memory, the right amount of high-bandwidth channels between the memory and the compute, and the right amount of compute that is effectively balancing that for that particular use case.

That design is maybe different than the hardware SKUs that you would use in ranking broadly or in serving a web page. But we had some great partners to work with on the hardware side. And of course, we have truly brilliant AI researchers on the modeling side, and software engineers for distributed systems that are optimizing the software infrastructure layer, and networking engineers who are optimizing how these machines talk to each other so that we can minimize end-to-end latency while maximizing the parallelism and compute we have available to deliver the best results for people and businesses.

So you sit down with your partner and say, “hey, we are a large customer. We have particular workloads that we run at scale and we know the shape of those workloads really well. This one with retrieval has these characteristics — memory bound, needs high memory capacity and bandwidth.” Does that lead you to look at those certain workloads that you have and ask, “what is the right shape of compute? What is the right SKU for retrieval versus ranking versus GEM training versus adaptive ranking?”

MS: That’s exactly right. We are always trying to work both sides of this problem. One problem is: how do we influence evolution of the hardware to better meet the needs of the software stack, and where we anticipate the software and AI stack is evolving over the next couple of years? Because — you’re probably familiar — hardware has relatively long lead times compared to software. On the other side of the problem, we are trying to influence the software stack evolution in a direction that is going to meet the hardware and maximize the potential of the hardware that’s going to be delivered to us this half, this quarter, this year, next year, and the following year.

We’re always trying to evolve them in similar directions. Sometimes there are hardware breakthroughs and we evolve our software stack to take advantage of those hardware breakthroughs. Sometimes there’s new software breakthroughs and we try to influence the hardware design in that direction to support those software breakthroughs. There’s a big discussion about this constantly across the industry. It’s particularly important given the rapid pace of innovation in the AI space — how quickly machine learning models are evolving, how quickly they are improving their performance and cost characteristics. It’s a wild time to work in the hardware-software intersection space.

MTIA: Recommender Systems vs LLMs

Totally. And obviously with transformers coming into existence, you’ve probably gone from more traditional ML into evolving toward transformer-based ones, and we’ll get there. But first, take me to MTIA. You talked about CPUs, you talked about GPUs, and how with the Grace Hopper that fit nicely into particular workloads. What leads Meta toward MTIA? There’s been a lot of announcements on that front lately — showing a roadmap, partnering with Broadcom. Can you tell us the business and economic rationale for moving in that direction?

MS: We tend to think about this in terms of the evolution of our heterogeneous hardware fleet over time. We can see the offerings that are available from our hardware partners that have various configurations of memory and compute and memory channels and different ratios. Some of them work really well for a particular use case. Some of them work really well for a different use case. There are different trade-offs with running different machine learning models on each of those hardware configurations. Sometimes the trade-off is latency. Sometimes the trade-off is cost. Sometimes the trade-off is power. In this very complex constraint satisfaction and optimization space, you’re trying to figure out what is the best offering that maximizes your returns for your advertising partners and for your business as well.

That’s where sometimes we have a use case that is different from your standard use case in the space. That was the initial impetus for the Meta Training and Inference Accelerators. Ads is a recommender systems class of problem, which is a little bit different domain than your large language model class of problems. The large language model problem is what’s known in the industry as an embarrassingly parallel problem. You can process a bunch of stuff in parallel. It doesn’t have to have super-effective high-bandwidth communication to be able to sync up the weights at periodic intervals.

At the same time, in the recommender systems space, all of the data is personalized. In the large language model space, if I was to say to somebody, “complete the sentence ‘to be or not to…’” there’s an objective correct answer — the highest probability answer that almost everybody who speaks English and has taken high school English classes could guess what the next word is going to be. A machine learning model similarly can learn there’s an objective highest probability answer to that blank in that sentence.

Now in recommender systems, the world is not objective and highest probability like that. The question is: what is the next best ad to show Matt? And it is not “what is the next best ad to show,” because who’s looking at the ad slot dramatically determines whether the ad is going to matter to them. There’s no objectively correct answer to what is the next ad to show, but there is a highest probability answer to what is the next ad to show Matt. Every example that is fed into our training systems for recommender systems has to have that personalization attached to the example.

What does that personalization look like? Well, Matt likes gardening and cycling and seems to buy a lot of stuff for toddlers, a lot of cleaning products. As a result, things that fit in those domains may be much more appealing to Matt than things that are outside of those domains. I used to have hobbies, now I have young children. That’s changed what I purchase quite a bit. The machine learning model can encode that, and it changes what the correct answer is to that question of what ad should be shown to Matt next.

That changes the size of the data packet associated with each of those examples. You have to pass in this personalization blob for the example of “we showed this ad to Matt and Matt clicked on it,” or “we showed this ad to Matt and Matt didn’t click on it. Here’s Matt’s big personalization blob of things he’s interested in.” The machine learning model can learn, “with this kind of personalization blob associated with Matt, he likes cycling and toddler toys and gardening equipment. These kinds of ads are good ads to show Matt and these kinds of ads are not good ads to show Matt.”

But that literally changes the hardware characteristics that you want when you have a very different I/O ratio associated with each example. If your examples carry a lot more data with each example, then you have to have a much fatter network pipe to keep the chip fed. You have to have more memory on the hardware SKU to keep the chip fed. You have to have a lower ratio of compute to memory — and high-bandwidth memory at that — to be able to effectively utilize the compute. So the optimal hardware SKU for training recommender systems may not be the same as a GPU that is optimized for training large language models. There’s obviously pros and cons there, but you may want to build a SKU that fits that particular workload really well.

Now that’s not all of our workloads. We obviously use GPUs in a lot of places. We use them for a lot of different parts of the recommender systems problem. But for some types of models, we have a use case for a hardware SKU that has a different configuration than what’s commonly offered as a GPU-packaged SKU. For some circumstances, a custom SKU with a different compute-to-memory ratio makes a lot of sense. For other applications, the GPU SKU is much more performant or much more cost-effective for that workload. We’re really trying to optimize the available compute and memory to the available models that need to be trained and the data size with each of those models. It’s a fascinating, challenging technology optimization problem.

Heterogeneous Hardware and LLM-Written Kernels

Yeah, that was really helpful. I like how you illustrated the problem to show that there’s specific I/O requirements and memory requirements, and how that could lead you to think about what, of all the possibilities out there, what SKU would fit best for this particular type of workload — and that might involve making your own. Now that’s talking about recommendation systems, which is really useful, and it’s a good reminder that the business involves training and inferencing recommender systems. Now, you did talk about GEM as a foundation model and needing to train that, and it being so big that it’s not cost-effective to serve. Can you tell us more about the compute challenges and the infrastructure demands on creating GEM and serving GEM?

MS: GEM as our foundation model is the largest model that we train in the ads recommender space. We try to feed it as much of our data as we can feed into the model to produce the largest, most complex, and best-predicting model that we have available. Some of the parts of the model are not super efficient, and that makes it not very effective to serve, particularly if you’re latency constrained. That’s why we had previously done this distillation process.

Now we’re using this distilled GEM variant that we’re calling the adaptive ranking model, where it’s distilled to be efficient enough to be served, but it’s not nearly as distilled as prior models, which were much smaller. The adaptive ranking model is an LLM-scale and complexity recommender model for Meta, with roughly one trillion parameters in this inference-time model. And it gets evaluated at sub-second latencies, which is a pretty fun and interesting software and hardware challenge.

Sub-second latencies — that’s amazing. You’re talking about different SKUs and different workloads, and I’m tracking all that, and you mentioned at the end of the day you have a heterogeneous silicon environment — different vendors, some home-brews, some off-the-shelf, some custom. You talked about software, and obviously having to work internally to make sure your software is going to work with the hardware and vice versa. Can you tell me more about how you manage software across all that hardware? Because to the layman, that sounds like a lot of added complexity — but I don’t know how many different levels of abstraction you can have that makes it easier.

MS: In general, heterogeneous hardware is a challenging problem to solve because you have to make sure that each of your binaries not only is capable of running on that hardware, but is performance and cost effective on that hardware. This is where folks have historically been forced to choose between custom optimization of a binary on a particular hardware type, or translation layers, which abstract away a lot of the custom features of the hardware but also abstract away a lot of the performance improvements of the hardware as well. There was a very clear spectrum of performance trade-offs between abstraction layers, which make it simpler to deploy hardware but less cost effective, and customization of binaries for hardware, which is slow and costly to implement but much more performant and cost effective once implementation is done.

Recently, machine learning models have enabled really cool abilities to customize specific binaries for hardware such that you can now at scale deploy binaries that are custom modified and performance optimized for specific types of hardware rapidly and easily, without having an expert software engineer do those performance optimizations for you. We recently put out a paper, I believe we called it Alpha Evolve or Alpha Kernel, where a machine learning model — a large language model — will write a custom performance-optimized kernel for a particular binary or machine learning model and a particular hardware pair.

If we have a large number of machine learning models and a large number of heterogeneous hardware types, writing the custom hardware kernel that would optimize the performance of this binary on the hardware was very time consuming before. It’s effectively a matrix of custom software that had to be written and hand-tuned by an expert software engineer. Now we’ve entered an era where large language models with coding capabilities can produce these optimized kernels at extremely low cost, way, way, way cheaper than having someone sit there and meticulously pick through the various optimizations necessary to make this binary or model run on this particular type of hardware.

It’s a real breakthrough in the technology industry and it’s going to enable a lot more of that cost-effective optimization that allows you to take much more advantage of all of the hardware available to you. Now we’re thinking through all of our deployments of all of our binaries to all of our hardware. Whereas before we wouldn’t necessarily move a binary that was adapted to a particular type of hardware to another type of hardware because that would be high cost and maybe it wouldn’t be worth it — now we can ask the machine learning model to produce an optimized kernel for this binary or machine learning model on this hardware, and we can do a lot more active management of software running on hardware. Which is going to both lead to better performance for our advertising partners, better experiences for people, and of course lower costs for Meta, as we get to take more advantage of the hardware we have available to us and really right-size the hardware and software use cases together. It’s a long journey, we’re not done by any stretch, but some of the new breakthroughs here in AI are having really beneficial effects on our ability to optimize our hardware and our software for our business.

GenAI Cross-Pollination and the Road Ahead

Amazing. What a world we live in. Reflecting back a bit — where my head is at is, back in the day it used to be software engineers were very expensive, and obviously Meta has probably always bought a lot of compute. But I could see the rationale for not having heterogeneous silicon because then you have to hire a bunch of software engineers if you want to optimize it for every different piece of silicon. Or on the other hand you just say, “software engineering is expensive, so we’re not going to perfectly optimize.” But at your scale you want to perfectly optimize everything so that you can eke out lower latency or better results. And interestingly, now we’re in a world where you need to buy lots and lots of hardware for your business, but the cost of software engineering has gone down to some extent with the help of generative AI LLMs, letting you still have a fleet — a matrix of different tasks and different hardware — and yet you can use LLMs to help optimize and fill out that spreadsheet in a cost-effective way. Which is very awesome. That leads me to the question about generative AI. How is Meta thinking about the relationship between its core recommendation systems and infrastructure and the investments in generative AI? Not only using generative AI in your core business, which alone is really cool and interesting, but also I know that you are training generative AI and offering that to customers.

MS: There is a lot of crosstalk between our various AI experts in the generative AI / large language model world and in our recommender systems world. Not only is there crosstalk and collaboration on hardware and data center design and performance optimization for the distributed systems, including things like the model trainer — we are both really focused on optimizing the machine learning model trainer and optimizing various aspects of the performance that the system needs to be able to train much larger models and serve much larger models. There’s a huge amount of joint investment that effectively benefits both sides of the house, the large language model side of the house and the recommender system side of the house.

We have experts in both types of ranking on both sides of the house so that we can improve the performance using both domains’ techniques and capabilities. We are — maybe as evidenced by the pace of breakthroughs that we’re able to deploy in our services here — really seeing the benefits of the innovation in the AI space across both parts of the business today. That’s obviously very exciting. This is the weirdest, wackiest, most fun time to be a software engineer ever.

Yes, seriously. It’s fascinating to think about those different sides of the house and how they cross-pollinate and impact each other, and just how fast both are moving. What an awesome time to be at Meta, and what a crazy time. Last question — looking forward, maybe two years because the rate of change makes it hard to look further than that — what do you see as the primary infrastructure needs for the next generation of AI-driven advertising?

MS: You can see we are all investing very heavily in building out data centers and purchasing large quantities of compute and memory and storage so that we can build better machine learning models, so we can find better machine learning models. The process of identifying performance improvements is really training a lot of machine learning models, tweaking various optimization parameters, coming up with new architectures and testing those to really drive maximum performance benefits. So, large investments in machine learning model training, machine learning model research that leads to performance improvements for training, that lead to performance improvements at inference time, substantial investments to make sure that we can infer these large language models and other generative models and ranking models both more cost effectively, but also driving more compute available at serve time and more memory available at serve time so we can feed things like longer sequence histories and larger context windows into these models.

The overarching theme here is end-to-end optimization. We’re trying to optimize the data center designs with the networking designs and the SKU designs and the software infrastructure designs for the distributed systems and the machine learning model infrastructure, the machine learning models themselves, the data that goes into them — all jointly, so we can drive maximum performance together.

Maybe to your point earlier, the demand for software engineering has effectively gone through the roof as the price has gone down. Whereas before we would invest in a limited number of hardware optimization kernels to run software on, now we want 100 times as many software optimization kernels for each piece of hardware because it’s available now. We can have machine learning models produce that, and now we have our expert hardware performance tuners supervising these models instead of writing the optimizations themselves. The same thing is true at every layer of the stack where we’re doing this optimization now. The demand for custom software that is more performant than a generic abstraction layer has gone through the roof. Every team at every layer is trying to do much better optimization to produce better results per dollar, better results per watt of power used in these data centers. That’s really leading to these meaningful breakthroughs that you’re seeing in terms of performance all across the industry, but particularly for the business as well.

Yeah, what a wild cross-optimization problem, being vertically integrated in some respects from hardware through data center design all the way to the software, to the training and the inference. And then being able to use LLMs to help with all this super fast. What I like about what you’re talking about here is: you have to make all of these trade-off decisions, but there’s a clear optimization function that you’re solving for when you’re thinking of an ad-space business — an ROI, how much are you willing to spend, how much are they willing to pay, and how can better results lead to potentially paying more or the pie growing bigger. I’m just thinking out loud, contrasting that to maybe other players in the generative AI space where the economics aren’t quite as straightforward in making these decisions. Anyway — you guys have a lot to think through. My very final question for you personally: how do you stay on top of it all as it’s changing so fast up and down the stack?

MS: That’s a great question. I don’t think I have a fantastic answer. The rate of change is amazing. I try to use all of the AI tools available, including large language models, to summarize papers, produce a list of all the latest papers that have come out with breakthroughs that are relevant to the domain that I work in. I rely on a brilliant team of expert AI researchers to summarize the progress that’s happening in the space, how that should influence the roadmap that we’re building for the future. But the amount of information and the progress in the space is just wild. It’s really amazing and something to behold.

Yes, totally. Well, you don’t sound bored, that’s for sure. Awesome. That’s it for today. Thanks so much, Matt, for taking the time to educate us. I’ve learned a lot and I know everyone will really get something out of this, so thank you.

MS: Definitely not. Thank you for having me, Austin. Great to chat with you.

Substrate is controversial. The debate tends to focus on individual objections, such as a lack of industry experience or the impracticality of particle accelerators. But I want to zoom out and look at the elephant as a whole:

To do that, let’s use a simple framework to test whether Substrate’s strategy is actually sound.

How to Solve Problems

A while back, I was listening to an old Lex Fridman podcast with Ray Dalio. And Ray was talking about his “5-Step Process”:

1. Set your goal

2. Identify the problems blocking it

3. Diagnose the root cause

4. Design around it

5. Follow through

Not rocket science. But there’s power in simple frameworks as a lens to view the world.

These five steps resonated with me, as they pattern-match well with my lived experiences (entrepreneurship, engineering, academic research, home projects, and so on). Once I heard it, I started seeing it everywhere. Listening through the How I Built This backlog, I found it in the Crate & Barrel founding story from the 1960s. Let me show you how it applies there, and then we’ll try it on Substrate.

Crate & Barrel

Quick context: Summer 1961. Gordon and Carol Segal are getting married. Both 23. They registered for stylish European housewares, but nobody in their lives had the money or taste to buy it for them. The Segals couldn’t afford it either.

But on their honeymoon in the Caribbean, they found the same products at a fraction of US prices:

Gordon Segal: There was one Scandinavian store in the Virgin Islands. My wife picked up some of the items and said, “How can you have Danish 18/8 stainless at the $2.95 a place setting? It’s so much more expensive in America!”

And the Danish merchant there said, “We have salesmen from Europe come here, and we buy direct from factories.”

Ah! A goal, a problem, a root cause. Gordon’s wheels started spinning:

We got back to Chicago, and I was in the real estate business. She was teaching school. We were both sort of bored. And then, one night in February of ‘62, I was washing these dishes we had bought. I said, “You know, Carol, there had to be other young people like ourselves with good taste and no money. We should open a store.”

Through the Dalio lens (and listening to the rest of the podcast):

• Goal: Sell beautiful European housewares to young people with good taste and no money.

• Problem: Those goods were priced out of reach in the US, even though they were affordable in Europe and the Caribbean.

• Root cause: Middlemen in the US import distribution chain.

• Design: Skip the importers. Buy direct from European factories.

• Follow through: Hard work and grit.

The Segals didn’t pencil out this strategy cleanly from Day 1. But the Dalio process was at work.

Now let’s apply it to Substrate.

Substrate

Substrate CEO James Proud made the goal very clear in this Stratechery interview: revive American chip manufacturing leadership.

WTF? REVIVE AMERICAN CHIP MANUFACTURING? WHO DOES HE THINK HE IS?

Before you grab pitchforks, let’s work through the reasoning. We’ll address “yeah but no industry experience” and the rest later. First, the 5-step process.

Goal: American leading-edge semiconductor manufacturing

What obstacles stand in the way?

Money and talent come to mind first. But those aren’t fundamental bottlenecks. Think about Elon and Terafab. Money and talent are tractable.

Dig deeper. Assume you’re well-capitalized and talent-rich. OK, this sounds like Rapidus. We can cover them in another article.

Now what?

You know what’s actually hard? Creating a customer.

Problem: No one will work with you

Customer acquisition is the problem.

How could you possibly incentivize chip designers to use a brand-new, leading-edge foundry? You need a reason so compelling that it overcomes the risk premium.

On what plane can you even outperform TSMC?! They have 30+ years of process knowledge and relationships with every major chip company on Earth. And don’t say supply chain security. TSMC has n-1 capacity in Arizona and Intel Foundry is getting its swagger back…

Hmm…

Well, what pain points do chip designers have with TSMC today?

Cost is a big one.

The Cost Problem

Many companies can’t afford leading-edge nodes, and even those that can only use them for a select few SKUs. Design costs run in the hundreds of millions. Mask sets cost tens of millions. Only products with massive volume (smartphones) or high ASPs (Nvidia GPUs) can amortize that cost. And variable costs compound it. Leading-edge wafers are north of $20K and all signs point toward $100K by the end of the decade. So even for the highest-volume products, where fixed costs amortize to near zero, wafer price still matters.

Why is the leading edge so expensive? Lithography.

EUV tools cost hundreds of millions each and the cost is only going up. You need dozens per fab, a significant driver of leading edge fab requires tens of billions to build. Deeper background reading here:

But… what if you could somehow reduce the cost of lithography drastically?

What if you could offer a value proposition of “2nm wafers at 28nm prices”?

You could create customers. Existing medium and low volume SKUs that would love to use smaller, more power efficient transistors or build their own custom ASICs instead of using less efficient/broader off-the-shelf options.

Yeah, but EUV LITHOGRAPHY IS MAGIC! TIN DROPLETS! 30+ YEARS! ASML! NO WAY!

Yes, I know. Suspend disbelief for a bit and just follow Dalio’s process. Let’s pull on the thread more.

What is the root cause of the lithography economics problem?

XLight has rightly pointed out that Laser-Produced Plasma (LPP) is very expensive. The “shoot tin droplets and hit them twice with a laser to generate EUV light” part. And every ASML EUV machine needs one.

XLight says, “Why not use a much higher power free electron laser (FEL) and share that light source amongst many EUV scanners?” This unlocks much better economics, not only from decoupling the light source from the scanner but also by increase the dose which impacts productivity and wafer economics.

See more here:

But what if you pull on the root cause even further? Could it lead to a different solution?

The entire EUV system (light source, optics, scanner) is incredibly complex with known inefficiencies (many mirrors, lots of lost light…) all driving extreme cost.

Is that cost due to physics, meaning this is the globally optimal solution, and there is a physical limit preventing lithography from ever working differently?

Or did path dependence lead us to a local minimum, not a global one?

Is the root cause physics, or path dependence?

Retracing The Path

Substrate believes it’s path dependence. And you can actually retrace the history to test that claim. In the ‘80s and ‘90s, the industry was actively researching X-ray lithography, which has a natural resolution advantage from its shorter wavelength.

Hmm… maybe we could learn what the shortcomings were and if they are till true today!

There are great old papers on this, especially from IBM Research. Check out this paper “X-ray lithography in IBM, 1980-1992, the development years”. It’s super enlightening. Let’s keep pulling on the thread, from Alan D. Wilson’s paper:

Optical lithography, in 1980, was very poorly understood by the experts. On the basis of historical trends and current difficulties with existing tooling and technology, the limits of lithography were thought to be about 1–1.25 µm for optics. X-ray lithography, consequently, was targeted for entry around 1 µm, the perceived limit of optical lithography.

At the onset of the program the strategic advantages of X-ray lithography were stated to be high resolution (better than optical lithography), throughput superior to that of e-beam technology, better resist-processing characteristics, and potentially lower defects (no multilayer resists).

X-ray seemed promising at the time. What did they learn? How did they learn? Were they thinking about manufacturability or just science?

We spent the remainder of 1980 developing a financial and technical program plan for X-ray lithography based on synchrotrons, considering a number of basic questions: Where were exposures going to be done? What should the mask be made from, and how would it look? How would we develop a stepper/aligner system, and what would be the role of vendor assistance in this regard? What would be the staffing needs (the initial group included only six people) as the program progressed? And finally, what should the test vehicle be?

It was recognized early in the drafting of our program that we were targeting manufacturing, not device prototyping, but full manufacturing. Our manufacturing divisions would eventually be our customer.

That’s really sound and reasonable thinking. They were thinking about full production scale and manufacturability, not just the science. And if you read the rest of the paper, the X-ray lithography (XRL) technology actually worked:

We had made complex, fully scaled CMOS devices with 0.5-μm ground rules before our optical counterparts had reached the same level using a long-established technology. Our yield was also acceptable: not 100%, but acceptable. The principal goal of the X-ray program had been achieved.

There are even tips for Substrate and us to think consider regarding a practical particle accelerator:

For X-ray lithography to be viable in IBM, we needed to explore acquiring our own X-ray source… Early in 1982, Grobman and I were starting to learn about the physics of synchrotrons. Our interest in a synchrotron ring for IBM was kindled by reports from Munich of the design of a tabletop machine called Kleine-Erna…. We began this serious inquiry by visits to established rings in this country and in Europe. Perhaps not being part of the synchrotron establishment was beneficial: We could ask questions and find out what really made rings good and what did not, as well as who the real experts were.

A quick aside… Agree. Perhaps not being part of the establishment is a benefit at times. … And figure out who the *real* experts are 😂

This IBM researcher Alan D. Wilson is my dude.

Anyway, he continues

Armed with this information, we asked ourselves what the specifications for the ring should be… I was invariably asked “What should a ring for industry be?” My answer was this: It should fit on a truck; plug into a wall socket; be reliable and available to operate 20 out of 21 shifts per week; have sufficient average output capable of sustaining a stepper throughput of more than 30 wafers per hour using an insensitive (having a sensitivity of -100 mJ/cm^) X-ray resist; be capable of being debugged/commissioned and assembled at the vendor, shipped intact to an IBM site, and rendered operational in a reasonable time at full specifications.

Make sure it fits on a truck and plugs into a wall socket. Quite practical!

And that kind of thinking is a lot different than the picture of particle accelerators I had in my head (i.e. CERN).

Oh last quick note from the paper, it seems that folks exploring XRL have always been doubted.

The establishment of a program was a formidable task, since half of this distinguished group seriously questioned the need for and the viability of X-ray lithography.

But IBM actually built a synchrotron (“Helios”) that fit on a truck and worked:

Oxford proposed a superconducting dipole system with a cold bore. The magnet turned out to be difiicult to construct but had excellent performance. The Helios 1 ring was completed and commissioned at Oxford, England, in October 1990. During the design and building of the synchrotron we visited Oxford and the Daresbury team every four to eight weeks over a period of three and a half years. The ring was shipped to IBM in March 1991 and arrived at East Fishkill on March 29, 1991. It fit on a truck, as shown in Figure 12(a), and we slid it into ALF that same day [Figure 12(b)].

The first beam was stored on or about May 20, 1991, and final specifications were met in January 1992. Our goal had been met and, in fact, exceeded, because the ring performs beyond specification [36]. Critics who thought design alone would not work were wrong. IBM and Oxford as a development team commissioned the Oxford ring in record time and with a very high level of performance.

If it worked back then… why don’t we have X-ray lithography today? XRL wasn’t without practical shortcomings at the time. This IBM paper from 1998 said that XRL was mature enough to possibly be introduced at 130nm node, but admitted manufacturing issues, for example with masks.

Nonetheless, there are challenges still to be met. Among the most important are the development and commercial availability of an improved e-beam mask writer; the ability to fabricate defect-free masks satisfying the image placement and critical dimension control requirements with good yields; the stability of the masks in usage (including the issue of possible radiation damage); the ability to correct for magnification errors; and the ability to satisfy the industry’s desire for a technology extendible to 70 nm ground rules. These issues are primarily manufacturing issues, as opposed to issues related to demonstrating proof-of-concept or feasibility, although demonstrating extendibility is still needed before the industry can commit to using XRL at 70 nm ground rules

XRL was physically feasible but had engineering problems to solve at production scale.

Meanwhile, optical lithography kept working far beyond the ~1μm wall Wilson predicted. So the industry kept pushing optical. First DUV, then EUV.

But a lot has changed in 35 years. US National Labs have spent years advancing particle accelerators and sources are now brighter, more reliable, and more compact. Computational lithography has improved significantly, too, and can help overcome the mask and proximity challenges that plagued IBM.

Substrate’s Design

So back to the Dallio process. The bottleneck is EUV-based lithography economics, and Substrate’s approach is to go back to the fork in the road and choose a different path. Could there be a global optimum, and could it be XRL?

It would require co-designing the light source, optics, and scanner as an integrated whole from scratch. But to me, there seem to be many cost savings possibilities on the table:

• One particle accelerator source feeds many scanners, like IBM’s Helios ring which had 16 beamline ports. The light source cost amortizes across many tools.

• No multilayer mirrors means no compounding reflectivity losses. Almost all generated photons are available at the wafer, vs. single-digit percent for EUV.

• No tin-droplet plasma source means no tin contamination, no collector degradation, no droplet generator maintenance.

• Single-patterning at leading-edge nodes. Fewer exposures = fewer masks, fewer etch steps, fewer defect opportunities, faster cycle time.

• In theory, the particle accelerator shouldn’t need cleanroom space. Only the wafer-handling end of the beamline sits in the cleanroom. This is contrary to EUV scanners which are massive and require substantial fab and subfab infrastructure:

So here’s the chain of reasoning so far:

• Goal: American leading-edge semiconductor manufacturing

• Problem: No one will work with a new foundry

• Root cause: The only lever that overcomes the risk premium is dramatically lower cost, and cost is dominated by lithography

• Root cause (a layer deeper): EUV’s cost comes from path dependence, not physics

• Design: Go back to the 1990 fork. X-ray lithography with modern sources.

This is sound. It sure seems XRL could genuinely untangle the lithography cost problem.

It’s believable. And it’s fundable. Substrate raised $100M from Founders Fund, General Catalysts, In-Q-Tel, and more. Founder’s Fund’s thesis is to invest in smart people solving difficult scientific problems where, if they succeed, the technology would be extraordinarily valuable. Substrate is a perfect fit. A 1% chance of reshaping the $1T+ semiconductor industry seems like just the type of asymmetric bet FF was built to make.

Of course, sound strategy doesn’t guarantee success. Execution is everything.

So can they actually pull this off? Behind the paywall I address the biggest objections head-on, work through whether XLight and Substrate can both win, and discuss the impact to TSMC and ASML.

It’s really, really interesting.

Read more

In this interview we discuss why Pope left Google to start a chip company, how to overcome the CUDA lock-in, and why frontier labs are the natural first customers. We get into the chip itself: a hybrid SRAM-HBM memory architecture that combines the low latency of Cerebras and Groq with the throughput of traditional HBM designs, and why that unlocks advantages across training, prefill, and decode. We also cover how agentic AI changes hardware requirements, how MatX uses AI internally in chip design, and the biggest skepticism Pope hears: can a 100-person startup manufacture at datacenter scale?

This interview is lightly edited for clarity.

Subscribe now

Origin Story

Hello listeners, we have a special guest today, co-founder and CEO of MatX, Reiner Pope. Welcome Reiner, for listeners who haven’t heard of you and MatX, who are you, what is MatX, what are you guys trying to do?

RP: Thanks, very happy to be here. As you mentioned, I’m CEO at MatX. What we’re doing is making the best chips for LLMs that is allowable by physics. My co-founder Mike Gunter and I, prior to MatX, were working at Google for a long time. Most recently, I was on the Google Brain Team training one of the LLMs at the time, and Mike was on the TPU team. There were a lot of things we wanted to do to make the TPUs much better for running LLMs. Things like running at much lower precision, having much more compute performance based on large matrix support, and generally optimizing for LLMs, reducing a lot of the other circuitry that was needed for non-LLM workloads. At the time, this was in 2022, and it turned out the best way to do this would be by starting a separate company, which is MatX.

So take me back, you mentioned 2022, you came out of Google, which I will say, it seems like everyone came out of Google that’s at the forefront of AI and hardware.

RP: It’s like the Bell Labs of the time.

Yes! There’ll be a book written 10, 15 years from now that we’ll get to go back and read and it’ll be fun for us to remember the good old days.

But okay, back to 2022. I think it was November 30th when ChatGPT officially launched. How much ahead of that were you guys thinking about this direction? Did you launch before ChatGPT? And how did that inflection point—the general public becoming aware of transformers—how much did that change your life in terms of fundraising, vision casting, hiring?

RP: As it happened, we left Google one week before ChatGPT was released. We did not know it was coming. But the historical context was that GPT-3 had been released more than a year earlier in this developer demo.

It was really hard to use. You had to get in the mindset of “I am writing a document and I want the rest of this document to be the response I’m looking for.” It’s not a chat interface at all, totally different. But if you were paying a lot of attention, you could see the potential. A lot of insiders in the industry were appreciating something big is happening here.

And the question, really, pre-ChatGPT was: these models are incredible, but they’re 100 times more expensive than the models we’re used to running. There are 100 billion parameters instead of under a billion. Can we even afford to run them? The simple economics doesn’t work out if you’re used to running software as a service where every query is free and now you have to spend cents per query. When you’ve got millions of queries per second, it doesn’t pencil out. The big question prior to ChatGPT was: okay, cool demo, but it’s too expensive. Can you actually productize it? And there was a lot of skepticism that you actually could.

ChatGPT demonstrated that you can, and not only that, but the product is incredibly valuable. What that meant for us was we had already seen that prices are going to be high. If prices are high, how do you make them cheaper?

It turned out to be quite difficult for us to fundraise even after ChatGPT. It took about two quarters for that to really land, when the impact on Nvidia’s stock price showed up. Then there was the realization: okay, this is using a ton of GPUs, everyone is buying a ton of GPUs. Eventually Nvidia reported these gangbusters quarters, and at that point investors started seeing the potential.

Okay, interesting. So you started by saying this is really transformational, but on the current hardware, it’s going to be too expensive. So there’s got to be a better hardware solution. Then ChatGPT launches a week after you guys leave, and I would expect investors to say, “I can see this is going to be productized!” But at the same time, Nvidia is the one capturing all the value and selling GPUs. So was the early skepticism just around: why will anyone buy hardware that’s not a GPU? Or did they quickly connect the dots that GPUs aren’t necessarily the most efficient?

RP: Some of the skepticism is definitely about why you would buy hardware that’s not a GPU. And then the other one is just: how do you compete with the world’s biggest company?

On the “why would you buy something that’s not a GPU” question, the big consideration is the software moat that Nvidia has. Everyone writes CUDA. Historically, we’ve seen how much software lock-in there is in so many businesses. Why is this one different? Isn’t there going to be software lock-in here? Would everyone really rewrite their software onto a different hardware platform?

CUDA Lock-In

Is there lock-in? How are you thinking about it from a software perspective?

RP: At this point, I think it’s proven that the lock-in is pretty weak. Barring Google, who has been on TPUs forever, all of the other frontier labs are multi-platform. OpenAI, Anthropic, Meta, X — they are all on Nvidia, many of them are on TPUs. There are Cerebras announcements, AMD, some Broadcom-developed chips as well. All of these players are multi-platform. That is the proof already that the software lock-in is not that great.

If you want to think about the first principles reasons why, it’s because software versus hardware lock-in is really a question of how much spend you’re putting on the hardware versus how much you’re putting on software engineering to support the hardware. This is really the first time that balance has changed, and it has violated a lot of people’s intuitions.

Historically, the whole history of software as a service is you’re paying really large salaries to a large software engineering team, and the compute spend is a small fraction of that. Engineering time is precious is the mantra. Of course there you have to prioritize the ease of software.

But this is totally turned around now. All of the frontier labs are spending tens of billions of dollars on compute. The salaries of the people writing software for that compute are very high, but still small in comparison to the compute spend. So the rational choice is to do anything you can to get hardware costs down, be multi-platform, get the negotiating power that comes from that.

I see, interesting. From first principles, it makes a lot of sense. Now that you’re going to spend so much money on hardware, how can you spend it correctly on software to unlock that? Even if it means you have a team writing kernels specifically for this architecture.

Fast forward from 2022 to now and we’re seeing everyone has multi-vendor silicon and it’s made your point. It’s very easy for you now. But back then, when you’re just starting and trying to raise that Series A, you clearly were trying to articulate that and hope that it came to fruition. Of your early investors, some of them must have believed. What got them to believe you in a world where it looked like Nvidia had all the GPUs and had the lock-in?

RP: Ultimately, all of early investing is primarily a bet on people rather than on technology. There’s a bit of both — you can have the best people in the world and have a business plan which doesn’t make any sense at all. But the premise that there is a physical product that we make that we will sell for dollars is a very easy business plan. It’s clear how you can make margins off of that.

In some sense, that’s even an easier business plan than starting a frontier lab. A frontier lab is like, “we’re going to make a model, we hope we can sell it in a product that hasn’t been defined yet.” With selling hardware, at least the business case is clear.

And for early seed stage investors, it’s primarily going off who we are, our backgrounds, and folks we’ve worked with who have vouched for us.

The Chip

And of course you have the credibility of having been TPU people at Google. Tell me, actually really quick question. I don’t know if I’ve heard you say this anywhere. Explain the name MatX.

RP: Matrix multiply. One angle is you remove “ri” from matrix. Another one is the X is a “times.”

Nice. So now take us into the first chip, the MatX One. I know you raised $100 million to get started, and then just a couple of months ago raised $500 million. We talk about [using that money to build] a chip, but I know you’re actually building a system. The goal is data center deployments. So with all of that context, tell me about the chip, but I want to get into the bigger system.

RP: A few of the core bets of the chip: primarily very high matrix multiply performance, higher than anyone else has announced in the market. There’s a whole story there, but in summary, the marginal returns on having more matrix multiply performance seem to be much higher than marginal returns on more HBM performance or other considerations. So you’ve got to invest in that first.

And then there’s this thing that had been like free money sitting on the table: get your memory system right. That is a combination of seeing two good ideas in the market. Nvidia, Google, Amazon have been all tensors in HBM — HBM first. Cerebras and Groq have been weights in SRAM. That gives you very low latencies, but it has some capacity problems. You can put those two together. It takes careful engineering and you need to balance the system right. It’s hard to balance the system right. But it is totally doable. That is the other thing we’ve done, and it gives some really big advantages in both latency and throughput.

I think a lot of people are now starting to connect with that as they see the Groq LPUs and Cerebras; they see the benefit of weights in SRAM for low latency. But of course you need HBM for high throughput and KV caches. Everyone’s starting to realize that context is awesome — the more context you can give a model, the more interesting insights you get. You made the right bet. Was that an architectural bet made from day one, based on first principles?

RP: Yes. One of the things we’re very good at is workload mapping to hardware, and creative new ways to do that that are more optimal, especially when you consider the space of what potential hardware could be. This combination of different memory systems was a core idea going in.

One of the things it really enables — if you look through the list of parallelism and partitioning techniques: tensor parallelism, expert parallelism, pipeline parallelism. The last one is the ugly stepchild in some sense. It misses a lot of the advantages of optimizing latency and memory footprint that the other ones do. It turns out that’s actually a memory system choice. This combination of SRAM and HBM actually makes pipelining work as well as the other techniques for the first time ever. We understood that, and that was what we were going after.

So back in 2022 when you’re making these early architectural decisions—the big systolic array, the right memory choice—you’re also thinking about mixture of experts and how different parallelism strategies require tuning those memory choices correctly. That’s IP and a differentiator for you versus someone who just says, “oh, weights in SRAM and HBM, let me go do the same thing.”

But reflecting back to 2022, I’m not sure mixture of experts was even out yet. So how much are you reading papers as stuff was happening in ‘22, ‘23, ‘24 and saying, do we need to tweak the architecture?

RP: We’ve been reading papers since 2017. I think the big and disappointing inflection point in 2022 was when Google stopped publishing. We were talking about how Google is where all the researchers came from. They had an incredible team in Google Brain and they were publishing everything, all of the good work they did. Very vibrant place to be. They stopped doing that in 2022 because of seeing the competitive market playing out. You could get all of the trend lines of where the best models are going until then, and then that stopped. DeepSeek publishing has been a pretty good reboot of that, but it’s sad that the volume has not been so large.

Research and Publishing

Totally. I will admit I haven’t read all of your papers on your website, but I see that you guys do some publishing still. How are you thinking about that fine line of what to publish and what not to? Because for talent, it is exciting to get to publish to the world and share what you’re thinking about.

RP: The ability to publish neural net papers is a differentiator for us in terms of hiring. We have two different areas of neural net research. We’re a small company, especially our ML team is very small because that is part of what we do, but it is not the main thing we do. We’re not selling ML, we’re selling GEMMs.

But the agenda of our ML team is twofold. First is attention research, specifically focusing on memory bandwidth efficient attention. That is quite aligned to where we see the future of hardware being. The second is numerics. Numerics has been the single best improvement in chip performance over the last decade. I think we have some of the best numerics talent and IP here.

In terms of what we publish: we don’t currently publish the numerics that goes into our chip. We will probably publish it on a one or two year delay after releasing the chip. But we do publish all of the attention research, because what we’re doing there is advocacy. We’re saying: hey, model designers, you should probably have these considerations in mind, especially when you think of future hardware that’s going to have a ton of flops but is going to be somewhat more memory bandwidth constrained.

Product Positioning

So you’re making hardware to sell at the end of the day, but you have ML researchers working on attention, memory bandwidth-efficient attention, and numerics. That informs your own architecture — extreme co-design. But you’re also trying to show model labs—the end customer—what’s possible. If they adopt your chips, how much will that change how they think about training or inference?

RP: We’re trying to not go too far outside of the comfort zone. If you want product-market fit, you have to mostly meet the customer where they are.

The way to quantify that: you can look at the chip specs and there are maybe five most important ones — HBM bandwidth and capacity, matrix multiply throughput, SRAM bandwidth and capacity, interconnect performance. Our attitude is we want to be at least on par with the best competition like Nvidia on all of these, and then substantially ahead on at least a few. The substantially ahead for us is obviously matrix multiply performance, also interconnect performance and SRAM.

There is no place where we are substantially behind in these big considerations. Maybe in some less LLM-relevant considerations we’re behind, but in these big five, we’re at least on par everywhere. That means the opportunity cost of switching to MatX is never too large.

But then the headroom you can get — if you want to maximize the benefit, you can tune your model. That means things like changing the balance between the MLP layer and the attention, more MLP less attention, or using some of our lower precision arithmetic. We have a range of precisions to get the biggest advantages out.

Gotcha. So you make sure that in these five most important areas, none of them are too weak to prevent a customer from switching. You’ll be there on every front. But then if customers take a step further and optimize for your chips, they’ll have more headroom, they can do more.

RP: Yeah, that’s it.

Customers and Workloads

Let me segue into who are those customers in broad strokes, the target customers for this chip system.

RP: The most interest has been from frontier labs, which is as expected. That is who we are designing for, and why they’re most interested is their spend is biggest. That also means the economics of being willing to tolerate a new software stack is biggest there too.

They also have this longer-term vision of three to five years out, which is where you need to be when you’re buying custom hardware. If you want to do really good co-design with your hardware provider, you need to be thinking on that time scale rather than just “I’ll buy what’s on the shelf today.” That’s where we’ve seen strong interest.

And this has shown up across all of the workloads — training, reinforcement learning, and inference both prefill and decode.

Nice, okay, let’s talk about those workloads.

Let me reflect it back. Your customers are going to be the frontier labs. They have the most compute spend, they are the most incentivized to squeeze as much intelligence as they can out of that. They’re thinking three to five years ahead. They are incentivized to not only work with all their current partners, but to always be listening and see what else is out there.

The market is telling us the defining workload of our time is LLM inference. You can optimize around the transformer, around splitting it into prefill and decode. We see that with Nvidia and with Dynamo. Everyone’s getting used to that concept.

The market narrative has gone from GPUs for everything to actually at the rack scale, maybe it makes sense to have some SKUs that run prefill and some that run decode. This is their way of saying those sub-workloads have different constraints — if it’s memory bound, have the right hardware versus compute bound. But I know you had a great podcast that everyone should go listen to with John Collison and Cheeky Pint. You talked there about being competitive on all those workloads — training, prefill, decode, RL. And it kind of felt like going back to the days of a GPU can do everything. So how are you talking with these partners about their different workloads, and how do you not feel like a salesman just saying “yeah, we can do that, we can do that, we can do that”?

RP: We just have to be honest about what the strengths and weaknesses are. Let’s give that a shot here. Our product has a really large amount of compute. Traditionally, training and inference prefill are the compute-intensive workloads, and decode is memory bandwidth-intensive. So you might think, MatX has a lot of compute, why would we use that on a memory bandwidth intensive workload like decode?

That’s where the joint hybrid SRAM-HBM design really shines. You spend none of your HBM bandwidth on loading weights. All of that bandwidth is spent entirely on KV cache. So you can get better use out of your HBM bandwidth than you can with Nvidia. But you also get the very low latency because the weights are stored in SRAM, like Cerebras and Groq.

Digging into that further: low latency means small batch sizes — that’s just Little’s law. The number of things in flight are smaller. The memory occupancy in HBM is proportional to batch size. So you can actually fit longer contexts in HBM than you could if the latency were larger. Low latency is not just a usability win, but it actually improves your throughput as well.

This is similar to what Nvidia is now doing with the Groq and Nvidia racks side by side, but there are some taxes you pay by them being in different packages. Putting the whole thing in one package is the first principles way to do that and gives you the most advantages.

Sure, that makes sense. You have a lot of compute. You make the right memory choices. Therefore you can do low latency and high throughput. And there are even benefits in the small batch size, low latency with respect to how the HBM is used. You talked about how Nvidia has essentially separate racks, the Groq rack in there, say Vera Rubin. You’re making one chip with benefits to both types of workload. How are you thinking about rack scale, interconnect, scale up, scale out?

RP: We have a lot of interconnect in the product. I think it is the most of any announced product, in fact. The reason: so you can support mixture of expert models with fairly small experts without becoming communication limited. Very sparse mixture of expert models are what primarily drive the interconnect requirements.

We deploy very large scale-up domains as well as supporting scale-out. The sizing of your scale-up domain is really driven by the sparsity and the kind of mixture of expert layers you want to support. You want to do the mixture of expert routing within your scale-up domain as much as possible — that is how everyone does it. Bigger scale-up domains allow bigger mixture of expert layers.

On topology, we do some interesting things with network topology. I won’t go into huge specifics, but contrasting what is in the market: Nvidia has done things like running everything through the NV switches. Google has these torus topologies. If you think about what you really want for mixture of expert layers, you can design something very custom for that.

I see, nice. That again aligns with the idea of designing not just the chip but the whole system for the specific workload, even down to network topology. That makes a lot of sense.

The Team

So how many people, even hand-wavy, do you have at MatX? We’re talking about networking, ML, hardware. Probably you have to think about cooling and operations and all sorts of stuff because it’s really data center design. Tell me more about the company. It must be very cross-functional — what’s it like there?

RP: For a product like this, it’s a relatively small team. It’s over 100 people. But some of these projects — Nvidia has 10,000 or 20,000 people.

Most of the team is hardware, which includes the core chip itself, the logic design, design verification, physical design, and so on. We designed the rack in conjunction with a partner as well. So we have folks looking at what is the insertion force of a board into a rack, cable density, power delivery, thermals. That’s going down the stack.

Going up the stack, we have a really strong software team writing the software stack that runs LLMs on our chip. And then we have the ML team doing exactly the research agenda I described. Very cross-disciplinary. I think it’s a super fun place to work because in one day you’ll have a conversation about physical insertion forces and at the same time functional programming or SAT solvers for compilers.

Agentic AI

Nice, sounds fun. So I’m thinking about your interdisciplinary team, everything you’re trying to build in your first system. And at the same time, the world is constantly changing. We’ve got agentic AI, Claude Code, OpenAI Codex, maybe an explosion of inference tokens needed. Opus is awesome but expensive. I can’t use my Mac subscription for Claude Code. All of a sudden Mythos has come out. And I’m wondering as a chip designer with ML researchers, how are you staying on top of all this? Are things changing that make you think in the next version of our chip we should do things differently? Or are you seeing it play out and feeling pretty confident, like, we can help this problem of awesome but expensive inference?

RP: Halfway through your question, I was like, is this going to be about how do we use agentic AI versus how do we serve it? Both are interesting.

How do we serve it: there is this ongoing trend where you see the incredibly fast pace of change in models, how people are using them, how they’re training them. But when you filter that through the lens of what does that mean for the hardware, it’s almost all noise — 95% is noise. The rate of change for what you need in hardware is much, much slower.

As that applies to agentic AI: what is it doing? It’s still doing decode. It’s still doing prefill and decode. Some things that are different: it has increased the demand. When the agent goes off and thinks for a long time and the user is sitting there waiting, you would like them to wait for 30 seconds instead of five minutes. So the demand for performance has gotten higher, but that’s within expectations. Demand is always going to get higher. That’s a great place to be.

One place where it’s actually a difference is sizing. Sizing exercises are what we do every day. One example: how long does the model sit idle while it’s waiting for a response from an outside system?

In a chatbot context, the model has responded to you, and then you as a human are thinking, maybe you’re going to type another message, maybe you never do, maybe you leave. That’s on the order of 30 seconds or a minute. The context for the model has to be kept in memory somewhere during that time, and you have to size that memory.

That has changed meaningfully in an agentic context where now the model is mostly waiting for tool calls — run a compiler, do a web search, check your email. The times for those are very different. Checking your email can run in seconds rather than waiting for a human to think. So the memories in service of that end up being smaller.

But then there are things like long-running jobs — running a compiler or running a place and route tool, which can take hours. I think that’s actually the biggest place it’s turned up: there is now increasing demand for storage systems for when the KV cache isn’t actively being used but is waiting for a response from an outside agent.

Yeah, interesting. So tell me, how are you guys using agentic AI?

RP: Most of chip design is actually software development in practice. The way you express a chip is you write Verilog, which is a programming language. It’s an unusual programming language because it’s massively parallel, but it is a programming language. Can you write that better with AI?

One of the things we look at: the places where AIs are most effective is when there is a well-defined objective function. Does this compile? Is the area good? Is the power good? How many tests does it pass? We look at our processes and say, can we do development in a way that puts it in that regime, which is really the sweet spot for AI development.

The other thing we do: in addition to Verilog, we use other languages. There are popular ones like Rust and Python, but also some less popular ones — in our case we really like using BlueSpec. It’s a hardware description language that comes from functional programming. We are looking into how we can make sure AI is really good at BlueSpec even though it’s a niche language.

Cool, interesting. I’ve never heard of it. Is that something you think about as a competitive advantage, or just generally wanting to make AI models better at BlueSpec and share this with the world?

RP: There are so few BlueSpec programmers in the world that we just want a higher pool of them, and then it becomes a competitive advantage.

Go to Market

I love that. Okay, since you’re the CEO, I’m going to go back to talking about customers, route to market. On the one hand, it’s kind of nice because maybe there’s only five or six customers that would be great, so any one of these would be a great anchor customer. On the other hand, probably everyone in this space is wanting to talk with them and work with them. What does it look like to say, we’re a startup, trust us, we’re building this thing, it’s going to be awesome? How do you have those conversations to address their concerns, and ultimately, how will they end up buying your first chip or your roadmap of chips?

RP: “Trust us” goes as far as your word goes, right? Not very far. So you need to prove it.

For us, proof means a lot of detail on the artifacts we have. What is the core architecture? What are the very specific details inside the chip? How do we organize the chip — we talked about this splittable systolic array, these are the different compute units inside the chip, how do they connect to each other? What is the instruction set? What is the software SDK?

We give all of this information to customers under NDA. It is a lot, and it is uncomfortable for us to give that information, but it goes a long way towards proving credibility.

Yeah, that makes a lot of sense. As far as the software, what is the level of effort they’ll have to commit to when they say, here’s yet another vendor, we’re excited about everything they told us, we believe them, but there’s probably still some effort to port?

RP: For sure. If you look at the sizes of teams supporting each of these multiple platforms, it’s on the order of 50 to 100 people per platform. Really good people doing kernel development, maybe building compilers, building debugging tools. I think that’s the ballpark of what folks should expect on our platform as well.

We want to help and we’ll do as much as we can to do that work for you rather than you needing to start it all yourself. But ultimately a frontier lab wants to protect its own IP, especially the model architecture. The last mile of kernel development is always going to remain in the frontier lab so they know specifically what they’re doing rather than giving it to us.

The first miles — giving a strong compiler and debugging infrastructure — is something we can actually do for you though.

One or two last questions. What is the biggest skepticism that you hear from people?

RP: One of the things we’re focusing on over the next few years is: how can we as a relatively new startup manufacture in massive volume?

It’s a really exciting opportunity. The projections for data centers over the next few years are in the many gigawatts, tens of gigawatts. I don’t know when we’re going to hit a hundred gigawatts. Nvidia chips sell for about $15 or $20 billion a gigawatt. You might multiply that by 10 or 100. It’s a really large commitment.

The opportunity is really large, but being able to get very quickly to selling such a large volume is also a substantial challenge, and some big parts of that are ahead of us. I think that’s a really exciting thing for us to do over the next year and a half.

Yeah, that’s a good point. It’s not just about building the system, it’s about can you scale it, can you production ramp it, can you get to huge deployments that people are comfortable with, that work, that are reliable. Okay, last question. Give me a hiring plug. You’re 100-some people, it’s very interdisciplinary. Why should people come work with you?

RP: Ultimately you have to believe in the product vision, and I think we just have the best product in the market. It’s designed from first principles for what LLMs really need, keeping in mind years of know-how and techniques of what is the right way to map applications to hardware. That’s the company vision. But the way we operate, it’s a very friendly and high-trust team with a ton of incredibly smart people. I think that’s the day to day of why it’s a really exciting place to be.

Sure, A-plus people enjoy working with A-plus people. Awesome, Reiner, this was great. I learned a lot. Thank you for the time. I’ll be fascinated to check in over time and see how things are going with you.

RP: Yeah, thanks Austin, it was really fun talking.

One might be arriving, but it’s not what anyone expected. In 2017, Benedict Evans predicted augmented reality would be next. Quite reasonable. But three months later, the transformer paper dropped. It took a while for that to change the world, but nine years on, the new device isn’t something even more mobile than a phone. It’s a box on your desk for your AI agents to live on:

Nvidia sells the DGX Spark as a “personal AI supercomputer.” AMD calls it an Agent Computer. Perplexity is shipping Mac Minis as a Personal Computer service.

Is this the beginning of a new S-curve? Or is it another iPad? If the agentic computer takes hold, it’s additive TAM. It carries meaningful ASP because it needs serious memory and compute. Does every knowledge worker’s desk eventually have two computers on it?

But there are headwinds. Recently, Anthropic banned always-on AI agents from using its Claude subscription plans:

That means running OpenClaw just got a lot more expensive. And that’s not the only headwind.

Read more

• Coherent makes its own EMLs, VCSELs, silicon photonics, and finished transceivers. It’s hard to find another public company that touches this many layers of the optical stack.

• Six-inch InP is ramping across four fabs with yields management says exceed 3-inch. The performance comparison against Lumentum is still playing out.

• Five growth vectors stacking: transceivers, OCS, DCI, CPO, thermal. Management says CY2026 is mostly booked and CY2027 is filling fast.

• Breadth vs. depth is the real question. Does a hyperscaler want one partner for everything, or best-in-class at every layer?

We’ve covered Lumentum’s and Broadcom’s AI optical infra businesses so far. Lumentum is a shooting star thanks to its laser performance plus tailwinds of industrywide supply scarcity. Broadcom dominates the datacenter networking silicon (Tomahawk switches, 1.6T DSPs) and is pushing direct-attach copper in scale-up for as long as physics allows, even as it builds CPO technology (lasers included) for optical scale-up.

Time for Coherent.

Coherent’s angle is vertical integration across the photonics value chain. The company designs and manufactures InP-based EMLs and CW lasers, VCSELs, silicon photonics, detectors, and finished transceiver modules in-house.

That puts it in competition with Lumentum, Broadcom, and Sumitomo at the component layer, and with module vendors like InnoLight and Eoptolink at the transceiver level. The stock has run from $45 to $250 in fifteen months and still trades at a lower forward multiple than Lumentum.

In August 2025, Coherent began production on what management calls the world’s first 6-inch indium phosphide production platform in Sherman, Texas and Jarfalla, Sweden:

If yields hold, that should materially improve Coherent’s cost structure and add meaningful InP supply to an industry that is currently constrained. How that affects pricing dynamics across the laser supply chain is one of the key tensions we’ll explore below.

Coherent’s vertical integration means it can supply components, modules, or systems across virtually every optical architecture a hyperscaler might adopt, from pluggable transceivers today to co-packaged optics and optical circuit switches tomorrow:

The bull case is that this flexibility becomes increasingly valuable as datacenter optical needs diversify. The bear case is that hyperscalers prefer to unbundle and multi-source each layer, buying best-in-class lasers from Lumentum, DSPs from Broadcom, and modules from whoever is cheapest.

Let’s walk through the business, the growth vectors, and the tensions that matter.

NFA, DYDD.

Coherent Corp

Coherent’s backstory begins with II-VI Incorporated, founded in 1971 and named after the II-VI compound semiconductor groups on the periodic table. Naming is hard.

II-VI’s original business focused on supplying engineered semiconductor materials and optical substrates that form the foundation of lasers and other photonic devices. Thus, the company operated at the lowest layer of the value chain, upstream of components and with limited exposure to finished products.

There have been many acquisitions along the way:

A few notable ones.

Finisar was acquired in 2019 for ~$3.2B. It was a leading manufacturer of optical transceiver modules at the time and brought significant VCSEL capacity for 3D sensing (for Face ID). The deal extended II-VI from raw materials into finished modules and added scale in hyperscale networking.

Coherent Inc. was acquired in July 2022 for ~$6.6B. Coherent was a storied laser systems company, founded in 1966, that built the first commercial CO2 laser and grew into a global leader in industrial laser systems, especially after acquiring Rofin-Sinar in 2016. It brought complete laser systems for cutting, welding, semiconductor lithography annealing, and display manufacturing, moving II-VI from components into full systems.

So II-VI started as a materials maker and eventually expanded to add photonic products and laser systems capability. They rebranded the parent company II-VI as Coherent Corp. in September 2022.

The result is a company with two distinct lines of business. ~72% of revenue comes from Datacenter & Communications (DC&C), the high-growth segment riding the AI optical buildout. The remaining 28% is Industrial; lasers for semiconductor equipment makers like ASML and Applied Materials, excimer lasers for OLED display fabs, silicon carbide substrates, and specialty materials.

The industrial business is slow growth, but it’s profitable, sticky, and generates nice margins. The slide above suggests Industrial is shrinking QoQ, but the 2025 Analyst day suggests Coherent thinks it can still grow 5-10% CAGR over the next 3-4 years.

Drag on the AI story? Or a diversified base? I tend to be comfortable with the latter — good source of margin dollars.

Coherent CEO Jim Anderson’s mandate is to reshape Coherent from a leveraged, margin-constrained conglomerate into a focused AI photonics platform, and the company is making solid progress. Revenue has grown from $4.73B in FY2024 to a ~$6.7B annualized run rate, while gross margins have expanded by roughly 500 bps to ~39% and EPS has scaled from $1.21 to over $5 on a run-rate basis.

At the same time, leverage has been reduced from over 3x to 1.7x, alongside a series of divestitures and footprint reductions to simplify the business:

Coherent bonus points: Was recently included in the S&P 500 and a $2B equity investment plus multibillion dollar supply agreement with Nvidia.

Vertical Integration

Coherent claims to have the broadest photonics portfolio in the industry.

Let’s walk through it.

First, indium phosphide (InP). Coherent has over 20 years of in-house InP capability spanning epitaxial growth, laser fabrication, modulators, photodiodes, and integrated subsystems.

One important clarification: Coherent does not grow its own raw InP crystals. It purchases InP substrate wafers from external vendors under 3-to-5-year supply agreements, then performs epitaxy and device fabrication in-house. This is the same fundamental supply chain dependency that Lumentum and Broadcom face, though Coherent has locked in multi-year contracts with multiple 6-inch substrate vendors to mitigate it. Worth noting that Coherent does grow its own SiC and GaAs crystals internally; it is specifically InP substrates that are sourced externally:

Gianmarco Conti, Analyst: Gianmarco from Deutsche Bank. You’re expanding indium phosphide capacity, but the raw indium feedstock is roughly 70% sourced from Chinese zinc smelters, which are now subject to export permit requirements with multi-month processing times. I guess my question is, how much visibility do you have on indium supply for the next 12 to 24 months? And are you actively diversifying sourcing away from China? Or do you hold strategic inventory buffer?

James Anderson, CEO: We actually have a very diversified supply chain for indium phosphide substrates. We have -- and I think I’ve shared this in the past, we have over five different substrate suppliers today, and we work with those suppliers, not just on the next -- you mentioned next 12 or 24 months. We don’t work on just next 12 to 24 months. We work on the next like three to five years of capacity that we’re going to need. So we have, in some cases, very long-term agreements in place. And that includes not just the substrates, but all the key inputs that go into that. So we believe that we have very good visibility into substrate supply. And so that capacity expansion that Beck showed is we have commitments from our suppliers to supply the necessary indium phosphide substrates to support that.

The 6-inch InP production platform is an important topic. Management calls it the world’s first, and it is now ramping across four sites: Fremont, California (3-inch legacy); Sherman, Texas; Järfälla, Sweden; and Zürich, Switzerland (newest addition). The economics, per management, are roughly 4x the devices per wafer at about half the cost compared to legacy 3-inch lines, with capacity doubling this year. If yields hold at scale, this could result in competitive cost structure relative to Lumentum (which is migrating from 3-inch to 4-inch) and Broadcom (believed to be on 3-to-4-inch). The timing of those yields is one of the main questions we’ll look at below.

On EML laser chips, Coherent manufactures 100G EMLs for 400G and 800G transceivers, 200G EMLs for 800G and 1.6T, and demonstrated a 400G Differential EML for 3.2T and 6.4T at OFC 2026.

Coherent currently uses a mix of internal and external laser sourcing. Anderson confirmed at the Morgan Stanley conference that Lumentum is both a customer and a supplier, suggesting that Coherent’s internal EML capability has not fully displaced external supply across all performance tiers. The 6-inch cost advantage may be closing the gap, but the performance comparison against Lumentum’s epitaxy is still playing out.

For CW lasers, which power silicon photonics transceivers and are critical for co-packaged optics, Coherent is in full production and ramping on 6-inch InP in Sherman. At OFC 2026, it showed a 400mW high-power version for CPO applications.

CW lasers are one of the key product families covered by the Nvidia supply agreement. On the OFC status update call, Beck Mason (EVP Semiconductor Devices) set out to reassure everyone that CW laser yields on 6-inch wafers are looking promising:

BM: We’re currently running three main categories of devices on 6-inch indium phosphide, EMLs, high-power CW lasers and high-speed photodetectors. And
all three of those categories, we’re seeing higher yield and better throughput efficiency on our 6-inch lines than we’ve been able to achieve even on our very mature 3-inch production lines.

On VCSELs, Coherent manufactures GaAs-based vertical-cavity surface-emitting lasers. These serve Apple under a new multiyear 3D sensing agreement, and Coherent plans to launch a VCSEL-based 1.6T transceiver in the second half of calendar 2026.

VCSELs are a lower-power alternative to InP-based solutions, but with shorter reach. CTO Julie Eng explained the tradeoff at the OFC briefing:

“The VCSEL actually is an interesting potential for silicon photonics because the power is very, very low… it’s basically between 4x and 5x lower power than the silicon photonics solution. But it doesn’t go as far. It’s shorter reach.”

That means VCSELs could be used for in-rack and near-rack scale-up, where power and density are prioritized over distance, typically under 100 meters on multimode fiber.

Management says Coherent has ample gallium arsenide capacity, which is relevant because GaAs circumvents the industrywide InP bottleneck. Anderson expects VCSELs and InP-based approaches to coexist rather than compete:

“In the VCSEL-based solution, you usually have the laser in it. And so there’s some pluses and minuses of that. But I do think that these will coexist in CPO/NPO just as they have in pluggable transceivers.”

Coherent also has its own silicon photonics PIC platform and demonstrated a 400G pure silicon PN-junction Mach-Zehnder Modulator at OFC 2026. This is a pathway to 3.2T transceivers via silicon photonics rather than InP, giving Coherent optionality across both technology approaches.

At the transceiver module level, Coherent ships full OSFP modules at 800G and 1.6T. At OFC, it showed 1.6T transceivers built with three different DSP solutions from three different industry leaders. That is notable because it positions Coherent as technology-agnostic at the DSP layer, in contrast to Broadcom, which makes its own DSPs and can offer a vertically integrated laser-plus-DSP package. Coherent is effectively saying it will work with any DSP partner, giving hyperscalers flexibility to choose.

Beyond transceivers, Coherent makes optical circuit switches (OCS) using digital liquid crystal technology, which is non-mechanical and has no moving parts.

Anderson said at Morgan Stanley that Coherent is engaged with over 10 customers and that “multiple customers have already deployed in real data center applications”. Revenue shipments began in Q4 FY2025. This is a different technology from Lumentum’s MEMS-based OCS, and the two approaches are competing for the same emerging market.

Finally, on the industrial side, Coherent has some interesting datacenter-adjacent materials. Thermadite is a proprietary material with what management describes as exceptional heat-transfer characteristics, which is being evaluated by large customers as a replacement for copper heat sinks in data centers. Coherent also has a thermoelectric material that can convert waste heat back into electricity.

Both are early-stage, but Anderson highlighted them at Morgan Stanley as longer-term growth opportunities that bridge industrial materials expertise with datacenter demand.

The bull case for all of this vertical integration is greater internal control over cost, supply, and iteration speed. Customers value having a single partner that can build EML-based, VCSEL-based, or silicon-photonics-based solutions, depending on the application.

Plus supply chain resilience across 60-plus manufacturing sites in 14 countries, more than 20 of which are in the United States (Q3 FY25 transcript).

JA: But the other -- the second point I would make in terms of supply chain resiliency is around vertical integration. And this applies to not just our data center business, but also to our industrial business, for instance, our laser business is if you look at a lot of the very key technology in feeds for whether it’s a data center transceiver or an industrial laser, we make ourselves, manufacture ourselves a lot of the very key components that go into our transceivers or laser systems or other products. And so that’s an important part of our supply chain resiliency and flexibility. So to the extent that there are changes in the landscape, the tariff landscape and to the extent we need to adapt manufacturing, move manufacturing to different places for the benefit of our customers, we certainly feel like we’ve got a very good, resilient, adaptable supply chain to leverage.

The bear case is essentially “doing everything means doing nothing best”. Lumentum’s epitaxy appears to be ahead (Coherent still sources some lasers externally). Broadcom’s system integration is deeper (laser plus DSP plus switch on-package). The risk is that Coherent ends up as a jack of all trades competing against specialists at every layer.

Growth Vectors

Coherent frames its growth story in two layers. The existing engines, pluggable transceivers (800G through 3.2T), DCI coherent transceivers, transport/transmission equipment, and optical components, collectively address a $50 billion-plus SAM per management’s estimates. These are shipping now and growing.

On top of that base, Coherent identifies four new growth engines that add over $20 billion in incremental SAM by 2030:

The stacking of these new engines on top of an already-growing base is why management says CY2026 is mostly booked, CY2027 is “filling very, very quickly,” and FY2027 revenue growth will exceed FY2026. Each new engine is at a different stage of maturity, so inflection points are spread over the next 18 months rather than concentrated in a single quarter.

Open Questions

So… Coherent has a very broad photonics stack with promising growth vectors at various stages of inflection. The stock has run from $45 to $250. The sell-side is overwhelmingly bullish.

Yet Coherent still sources some lasers externally, including from Lumentum. Broadcom has the full-stack CPO lead, even if its CEO says CPO is “not anytime soon.” Chinese module makers are winning volume at 800G. And the BIS Huawei investigation is still unresolved.

Which of these growth vectors holds up under scrutiny? Where is management credible and where are they hand-waving? How does Coherent stack up head-to-head against Lumentum and Broadcom across each product category?

I went through all of this against Q2 FY2026 earnings, the Morgan Stanley TMT Conference (March 3), OFC 2026 announcements, and sell-side research, then put together a three-way comparison with Lumentum and Broadcom.

Here’s what’s behind the paywall:

• COHR vs. LITE vs. AVGO: Head-to-head across every product category

• Six things to watch, each with a bull case, bear case, and what to look for next: the 6-inch InP bet, OCS liquid crystal vs. MEMS, CPO positioning, the margin path to 42%, BIS risk, and valuation

• How the latest quarter stacks up against each of those

• What the Street is saying and where analysts disagree

• Catalysts for the rest of CY2026 and into CY2027

and more!

Read more

It’s amazing. They just describe what they want, and Claude code asks some clarifying questions, and then runs off and builds it.

If you haven’t tried having AI build software for you yet... you gotta try it. Claude Code or Claude Cowork, Cursor, OpenAI’s Codex, Perplexity Computer, whatever. It’s easy to get started with something simple on your laptop.

We often spin up several simultaneous agents and build many things at once. Dad, can we create a black hole simulator while we wait for this game to build? Sure, buddy!

Now, if I spin up a bunch of agents and they are all doing heavy token generation on my behalf, are the deterministic tasks like API fetching and code execution running on server CPUs or just locally on my machine? Great question.

I personally use the command line version of Claude to spin up agents, so let’s see if we can figure out what it does.

This CLI is open source, so we can read the code to figure out how it works. lol jk, we won’t read the code, we’ll just have Claude Code read its own source code for us. It turns out the CLI runs almost everything locally, which, in my case, is on an Apple M4 chip. So when Claude Code on my MacBook writes and executes code to generate the ASCII art in the image above, the non-GenAI bits run locally on my MacBook.

Now, that doesn’t mean EVERYTHING agentic will run locally. As cool as Claude Code and OpenClaw are, they aren’t the only way to run agentic AI. A lot of agentic AI will be kicked off via phone and web apps. The user will taps a button or type a sentence in a normal app, and behind the scenes an agent spins up in the cloud, runs tools, makes API calls, executes code, fetches data — all CPU work on server racks. The apps themselves can use the Claude API, which offers the same primitives the CLI offers locally — code execution, web search, programmatic tool calling, and so on. But they run in the cloud and can be scaled.

Cloud-based agents will contribute a ton of agentic AI inference in the coming years, and will continue to be a key driver of the majority of Anthropic’s revenue.

In GPU inference servers, the head node's CPU is there to keep GPUs fully utilized. Agentic workloads introduce orchestration and tool execution overhead that erodes CPU headroom and constrains GPU utilization. The fix is to extend beyond the head node into a proximate CPU rack, positioned close to the GPU racks to minimize latency and maintain throughput.

Ok then, if we need racks of AI CPUs for agentic AI... which racks? Like AMD Epyc or Intel Xeon racks? Or Graviton / Cobalt / Axion racks? Or what?

Well, it probably makes sense to start with the same kind of CPUs that are already running the Gen AI related CPU workloads today, right? The existing head node CPUs already handle orchestration, multi-modal fan-out, data pre- and post-processing, and so on.

Maybe just buy more of those? Like my dad always said, if it ain’t broke, don’t fix it.

And interesting, those are often Arm CPUs these days.

As a reminder, Hopper systems typically paired GPUs with Intel Xeon CPUs, most often Sapphire Rapids or Emerald Rapids. Nvidia had little incentive to drive AMD CPU share given Instinct competition, though OEMs still offered server configurations with AMD EPYC head nodes.

But, Blackwell-era GPUs were mostly paired with Nvidia’s Grace CPU (Arm Neoverse V2). Which means the most important workload of our lifetime to date, LLM inference, was initially deployed on x86 but quickly moved to Arm.

Now I’m curious. What about XPUs?

Claude inference on Trainium 2 uses x86 head nodes, purportedly Intel Sapphire Rapids. At roughly one CPU socket per eight Trn2 chips, the one-million-accelerator Trn2 deployment implies about 125K x86 CPUs. Nice. But wait! According to SemiAnalysis, Trainium3 is moving to Graviton4 (Arm Neoverse V2).

Hmm. Another instance of inference clusters swapping CPU sockets to Arm.

And maybe TPUs are heading that direction too? Per SA, “in the future, Google will design Axion CPUs for use as head nodes in their TPU clusters powering Gemini”.

It sure seems like a lot of the CPU workloads that support LLM inference are already running on Arm, or heading in that direction.

And thus adding nearby racks of Arm CPUs to create headroom for agentic AI seems very sensible.

OK then, where does one buy such racks of Arm CPUs anyway?

Option A: Build Your Own

The cloud providers (AWS, Google, Microsoft) are already building custom Arm CPUs (Graviton, Axion, Cobalt). But these existing Arm CPUs were designed for traditional cloud server workloads. Ya know, APIs, databases, web servers, and all that jazz used to build SaaS empires. Agentic AI workloads have different requirements than cloud-native ones. They want much more memory bandwidth per core. Low tail latency. Etc.

So CSPs could add new custom CPU SKUs to the roadmap, tuned appropriately. Same team can reuse a lot of the same IP, it wouldn’t be that bad. But the agentic AI CPU demand/supply imbalance is hitting RIGHT NOW. No one has time to wait for an agentic-flavored Graviton to be designed, validated, and taped out.

In the meantime, sure, CSPs could just use their existing custom CPUs. They might not be perfectly optimized for the workload, but given we’re in agentic takeoff, all CPUs on deck. Perfect is the enemy of good. But, of course, at massive scale, every inefficiency adds up, so there’s still going to be a need for the right CPUs designed for the agentic AI workload in the fullness of time.

Btw, this “good enough” argument could be applied to Arm-based server SKUs from Qualcomm and Ampere, which were not designed with Agentic AI in mind. But silicon is in short supply... if you’ve got ‘em, sell ‘em.

One final observation: future agentic CPUs from hyperscalers will likely adopt Arm’s Neoverse V3 CSS, which carries higher royalty rates than V2. If so, agentic AI drives both unit volume and Arm’s average royalty per chip.

Option B: Buy Nvidia Vera

If you want to stay on Arm and you need racks now, you’re in luck — Nvidia is selling them. The Vera CPU, which Nvidia calls “the world’s first processor purpose-built for the age of agentic AI,” is built on 88 custom Olympus cores (Arm Neoverse V2). The liquid-cooled Vera rack has “256 liquid-cooled Vera CPUs to sustain more than 22,500 concurrent CPU environments, each running independently at full performance”.

Nvidia’s gonna sell a lot of these Vera racks for agentic AI.

Profit pools don’t stay uncontested, though. If there’s money to be made, competitors will appear. And who might that be? The CSPs are the obvious candidates, but their model centers on deploying infrastructure for internal use and rental, not selling merchant silicon. Well, maybe not GCP if the rumors about selling TPUs are true.

OK. I’ve buried the lede... you’ve surely put this together by now…

Who else can sell Arm-based agentic AI CPU racks?

Option C: Buy from Arm

Arm can!

They’ve already done most of the heavy lifting with CSS, the Compute Subsystem that stitches together CPU cores and system IP, and handles the backend physical design too. At that point, why not just take it all the way across the finish line and build the full chip?

And that is exactly what happened.

This week, Arm announced the Arm AGI CPU, the first merchant silicon offering from Arm.

For the first time in its 35+ year history, Arm is a merchant silicon CPU vendor.

Some details from the announcement:

For the workloads driving CPU demand in AI like agentic orchestration (web searches, API calls, agent fan-out), RL training sandboxes (code compilation, verification, tool use), and data processing, you need lots of performant cores.

That’s exactly what the AGI CPU claims to deliver. In a liquid-cooled 200kW rack it has over 45,000 cores:

Note that these CPUs to GPU racks over the network, not via direct chip-to-chip links. Nvidia’s NVLink C2C only applies in the integrated Vera Rubin NVL72, where CPUs and GPUs live in the same rack. I had heard some confusion here so wanted to clarify.

So… Arm is now selling chips. If you need agentic Arm racks you can build, buy from Nvidia, or buy from Arm.

But there are still many questions to think through, including:

• Vera vs. AGI. Different strengths, different price points.

• Where are the x86 agentic AI CPU racks?

• Why are Meta and OpenAI launch customers for both Vera and AGI CPU?

I’ll hit on these and more for paid subscribers.

Read more

• Disaggregation is now official Nvidia doctrine. Not just a startup pitch.

• Agentic AI is the killer app driving all of this. Vera CPU racks because CPUs were bottlenecking GPUs. LPUs for ultra-low latency coding agents.

• Nvidia’s strategy hasn’t changed. The whole system must beat any mix-and-match alternative. Does InferenceX sufficiently measure Agentic AI Factory performance?

• The unbundling of inference workloads is the unbundling of the datacenter. If Groq can slot in, so can Cerebras, Etched, MatX, AMD. Conceptually, anyway…

Without further ado, the most important slide from GTC 2026, demonstrating that Nvidia GPUs plus AI ASICs lead to a better and expanded Pareto frontier:

Want to unlock super low latency (or very high tokens/sec) for insanely fast Claude coding? ABC – Always Be Claudin’, amiright?

You got it! GPU + LPU. See the far right side of the chart.

Disaggregation unlocks “right silicon for the workload”. And the right silicon isn’t always GPUs; a bunch of Groq’s chips can tally up much more bandwidth for memory-bandwidth bound workloads:

Yet Rubin has way more FLOPs for compute-bound workloads. So put the two together, and you can outperform just GPUs. Of course it will cost you $$$, but for certain customers and points on the Pareto curve it can make economic sense.

To be fair, disagg has been out of the bag for over a year now, but the full ramifications are becoming clear. In the past year we learned of Dynamo and prefill/decode disaggregation. And we even saw Nvidia unveil the Rubin CPX as a SKU specifically for prefill. But that was still just splitting the workload amongst GPUs. But now we see further disaggregation, and AI ASICs have entered the picture:

So GPUs aren’t enough!

The narrative has officially moved past GPU for everything, as said on Nvidia’s blog:

Hardware tuned for peak throughput under large batches isn’t ideal for the most latency-sensitive execution paths, while hardware optimized for low-latency execution is less efficient for the most compute-intensive phases.

Multi-Vendor Inference

It’s not a stretch to call this a multi-vendor inference system. Or a heterogeneous system if you prefer. Sure, it’s the Groq 3 LPU with an Nvidia label on it, but conceptually its an AI ASIC startup rack with an Nvidia GPU rack.

Hence, the corollary: If Groq racks can be slotted in, so can Cerebras, MatX, Etched, AMD, Intel, and so on.

The unbundling of the workload is the unbundling the AI inference datacenter.

That said, although the narrative has changed from “GPUs for everything” to “right silicon for the workload”, I’d argue that Nvidia’s strategy hasn’t changed one bit.

I’d sum up Nvidia’s strategy as ensuring that the whole inference system is greater than the sum of the parts. Said another way, Nvidia is betting that full Nvidia inference AI clusters will outperform competitive clusters piecemealed together from different vendors.

And Nvidia is betting that outperformance is unlocked through vertical integration. Think Apple, Tesla.

Take Nvidia GPUs vs AMD Instinct GPUs as an example. When AMD first came on the scene, even though MI300X was a good inference chip, it didn’t have performant enough software to extract all the value from the chip. So Nvidia’s whole system (H100s + CUDA) was better than AMD (MI300 + ROCm).

Grace Blackwell NVL72 took it up a notch. The scale-up NVLink switch enabled Nvidia to have 72 GPUs acting as one big GPU. Thus, Nvidia’s whole system (accelerator + scale-up networking + software) outperformed AMD MI350, which didn’t have competitive scale-up networking technology. AMD won’t have a scale-up domain size of 72 until MI450 with Helios, and even then, it’s UALoE.

And Nvidia is already showing they’ll be running further ahead with a portfolio of inference-centric offerings like the Vera CPU racks for agentic AI and the STX Storage racks:

Just look at that row of CPU + GPU + LPU + switches + storage. Whatever agentic AI factory you can put together, can it outperform this fully integrated system? And was the opportunity cost of sourcing it all, hooking it up, validating it, and making sure the software works across it all worth it?

Again, even though disagg is out of the bag, Nvidia believes it can outperform an “open” modular system with components from different vendors, e.g. CPU and GPU from AMD, AI ASIC from a startup, scale-up switch from Celestica / HPE / Astera Labs / etc.

Agentic AI is GenAI’s Killer App

Given that Nvidia is competing on “full AI factory performance”, it’s no wonder Nvidia is shipping Vera CPU racks.

No, Nvidia isn’t trying to take down Intel or AMD.

It’s much simpler. What was the North Star? The whole is greater than the sum of the parts.

And the “killer app” for GenAI has appeared; it’s agentic AI. Claude Code. OpenClaw. The world will never be the same.

And Nvidia wants to make the best Agentic AI Factory.

Agentic AI involves a lot of tool calling, data processing, and so on. The head node CPUs can’t handle it all. So Nvidia added two racks of Vera CPUs in the same row as all the GPUs, optimized for agentic AI. From Ben Thompson’s interview with Jensen after the keynote:

JH: ... you want the fastest single-threaded computer you can possibly get… the most important thing is single-threaded performance and the I/O has to be really great… if the CPU gets throttled, then we’re holding back a whole bunch of GPUs.

Clearly, Jensen is thinking about making the whole Agentic AI Factory greater than the sum of the parts. CPU was a bottleneck, so CPU racks were included.

And we want agents to be fast. Generating code is awesome. Generating hours worth of code in minutes is even more awesome. High interactivity is necessary for agentic AI, hence LPUs.

Implications

So disagg is out of the bag, and now many implications and questions are piling up:

• What does “right silicon for the workload” mean for the CPU market when agentic AI is the killer app?

• If the datacenter can be disaggregated, the door is open for other SKUs… but can they actually slot in? Will Dynamo allow it? Will an open alternative emerge?

• Where do XPUs fit?

• What about other Pareto frontiers? Why only the largest models at 1000+ token throughput? What about medium or small models at 1000+ tokens?

I have strong views on all of these, and more. Let’s go deeper.

Read more

Three things from Jensen’s GTC 2026 keynote were notable:

1. Scale-up is going copper and optical

2. Nvidia is selling lots of standalone CPUs, calling it a multi-billion-dollar business

3. The Groq 3 LPX chip is real, fabbed by Samsung and shipping Q3

We cover the full Vera Rubin data center system, tiered inference economics, CPO in production, and take a detour into whether the vibe-coding era actually kills SaaS (we think not).

You can find it on podcast players too, e.g. on Spotify and Apple Podcasts.

This interview is lightly edited for clarity. Transcript is available for paid subscribers who prefer to read vs watch.

Read more

• Four MTIA chips in two years, all inference optimized

• ROIC story is straightforward

• Further validates the industry is past “GPUs for everything” value prop

• Implications for Broadcom, Nvidia, AMD, HBM suppliers, TSMC, Arista, and inference startups

The narrative around AI silicon and GPUs has changed. Six months ago, the default assumption was that GPUs are the answer for everything. But as I wrote about last October in Right-Sized AI Infrastructure, as AI workloads mature and become better understood, the economics favor purpose-built hardware over general-purpose GPUs. I’ve been long saying this, see GPU bloat stifles AI from Feb 2024. But that was long before reasoning models and agentic AI, so the industry was still stuck on “is GenAI even useful?” and “what if LLM architectures change?

The demand for transformer-based LLM inference is exploding. As Ben Thompson stated this week, we’re in a new era and demand is increasing exponentially :

This [functional agents] increases the market in two directions: first, humans can run multiple agents, and secondly, agents can leverage reasoning models multiple times to accomplish a task. This isn’t just an exponential increase in the addressable market for tokens, it’s two exponential increases squared.

In this era of intense demand (NEED MOAR TOKENS FOR MY AGENTS!), developers care about latency (MAKE AGENTS FASTER!) and throughput (DANG OPUS IS EXPENSIVE!)

The market is responding rationally. In just the past several months we’ve seen NVIDIA+Groq and OpenAI+Cerebras. And those were pre-ChatGPT chips, aka they made design decisions not optimized for transformer LLM inference.

We’ve also seen maturing XPUs and roadmaps, including Trainium 3/4/5, Microsoft Maia 200/300, and now Meta MTIA 300/400/450/500.

Meta’s annoucement was this detailed technical blog articulating four iterations of MTIA that have shipped or are planned within roughly two years.

And they were very transparent with specs, timelines, etc. This is really cool. Of course, a company like AWS with Trainium is going to share XPU specs because it’s cloud rental business might rent out the XPUs too. But Meta’s only using them internally; no cloud biz. So Meta doesn’t HAVE to share as much info. But they shared anyway. This is good for investor transparency, for current employees and potential hires, and for us nerds.

Here are said specs from Meta:

And the roadmap makes sense.

As I wrote in Meta’s ROIC Strategy: GEM Now, LLMs Later, Meta’s business has always been powered by ML/AI infrastructure. Facebook launched the algorithmic feed in 2011, and ever since then ML recommendation systems have driven what 3.5B daily users see and which ads they’re shown. To the tune of $150B+ annual advertising revenue. Hence, the custom inference silicon roadmap is driven by workload, starting with clearest ROI (recommendation systems) and moving toward GenAI.

By the way, Meta and many others are showing that multi-vendor hardware portfolios are no problem. We need compute. We can deal with the software implications of building across stacks.

From the Meta Q4 earnings call:

“We extended our Andromeda ads retrieval engine, so it can now run on NVIDIA, AMD, and MTIA. This, along with model innovations, enabled us to nearly triple Andromeda’s compute efficiency.”

Done right, custom silicon for recommendations is margin expansion on the core business. Custom silicon for GenAI inference is cost reduction on the fastest-growing workloads. A penny saved is a penny earned. Same story AWS tells with Trainium 3+, same story as Microsoft and Maia 200+.

And again, it’s inference-first. From Meta’s blog:

“Mainstream GPUs are typically built for the most demanding workload — large-scale GenAI pre-training — and then applied, often less cost-effectively, to other workloads such as GenAI inference. We take a different approach: MTIA 450 and 500 are optimized first for GenAI inference, and can then be used to support other workloads as needed”

IMO this pushes back on concerns around The Information’s recent article: Meta’s Internal Chip Design Efforts Hit Roadblocks

Meta last week scrapped the most advanced chip it was developing for training AI models, after struggling with the chip’s design.

This generated some hand-wringing for Broadcom and Hock had to address it on the earnings call. But in context, it’s a bit of a nothing burger, right? Given where we are today, internal custom silicon efforts like Meta’s ought to prioritize inference. Use Nvidia GPUs for training. AMD Helios racks too. IMO there’s no urgent need to build custom silicon for training.

The ~6-month cadence is possible because… chiplets!

Because each chiplet can be upgraded separately, we can implement improvements in months rather than years. Moreover, different chiplets can be manufactured at different process nodes that are most cost-effective while meeting performance and power requirements.

Disclosure, big chiplet fan here.

AMD has long punched above it’s weight thanks to chiplets. And here’s Meta doing the same.

And as you would expect, all the systems use the same chassis, rack, and network infra. Yay OCP.

Oh, and so much related news.

Remember Meta’s Sep 2025 acquisition of Rivos, a “CUDA compatible RISC-V AI startup”? From Reuters

“Our custom silicon work is progressing quickly and this will further accelerate our efforts,” a Meta spokesperson said when contacted by Reuters.

Rivos was already collaborating with Meta on MTIA before the acquisition, according to reporting from The Next Platform:

Rivos, which was founded in May 2021, was pretty secretive about what it was up to and it had a partnership with Meta Platforms where it apparently helped in the design of the MTIA 1i and MTIA 2i compute engines (using the more recent and descriptive way of talking about them). The exact nature of this collaboration was unknown. Separate from this, Rivos was working on its own RISC-V CPU and GPU designs.

And four chips in two years requires a large silicon team. Which apparently Rivos had:

With the backing of Walden International with the help of Dell Capital Ventures and Matrix Capital Management, Rivos started off with more than a hundred employees on day one, and Tan was named chairman of the board. This has, in part, given Rivos access to advanced EDA tools and to foundry expertise and capacity at Taiwan Semiconductor Manufacturing Co. Hiring nearly 50 engineers from Apple in 2023 landed it in a lawsuit with Apple, and Tan negotiated a settlement.

Oh interesting. Also, Lip-Bu Tan is EXPERIENCED.

Software seems not to be a problem. Meta’s MTIA blog emphasizes PyTorch/Triton/vLLM compatibility at the ML framework layer. And Rivos also claimed to have built a “CUDA-compatible software stack”. And Meta is good at software.

Sure seems like MTIA will be successful.

Behind the paywall:

• HBM vs SRAM: two planes of inference competition, and where MTIA sits

• Supply chain impact: Broadcom, NVIDIA, AMD, HBM suppliers, TSMC, Arista, inference chip startups

Read more

If you haven’t read those, they are easy reads. I recommend skimming through them first.

Each step trades flexibility for efficiency. And the root source of the inefficiency is that long, noisy copper trace between the switch and the optics:

At modern lane rates (e.g 50G, 100G per lane), electrical signals pick up a ton of noise and distortion crossing the copper trace between the switch and the transceiver. Pluggable optics handle this with a DSP that overcomes the noise during transmit and receive. And LRO and LPO save power by relocating that DSP into the switch:

But the system becomes less modular and harder to mix-and-match.But why deal with that copper trace at all? What happens when you just... put the optics right next to the silicon?

NPO

Near package optics (NPO) brings the optics module on the same substrate or very close to the switch package, but not inside it:

It’s close enough to reduce most copper impairments. This is a pragmatic middle ground, but the major players are largely leapfrogging it and going straight to CPO.

Might as well just reduce the copper distance to nearly zero right?

CPO

Finally! Co-packaged optics (CPO).

The optics move onto (or into) the switch package itself:

The electrical path between the switch die and the optical engine is now very short (millimeters or less). Since there’s no long copper trace, we needn’t have a DSP to compensate for it! Less silicon content, and less power.

There’s also much less SerDes power overhead as you only need extra short-reach (XSR) SerDes, the simplest, lowest-power tier:

The simplest way to think about CPO is that the transceiver disappears and the optical engine moves onto the switch package itself.

How It Works

SemiAnalysis has an in-depth CPO article with a helpful diagram:

The optical engine is the core of CPO; it converts between the optical and electrical domains. Since the OE is on-package, fiber runs directly to the package edge. And now the electrical path to the switch is so short that signals stay clean without heavy conditioning. The switch ASIC’s SerDes handles what little remains.

Another helpful diagram from Nvidia’s blog. The red circles highlight noisy copper channels. Notice how CPO eliminates most of them:

CPO cuts down on the overall power consumption, too. Per this example from Nvidia, power consumption cuts down from 30W for pluggables to 9W for CPO:

As I always say, in this power-constrained era, every watt saved is a watt that can be used for computation.

At this point, you should watch this Nvidia CPO video again, as it will make a lot of sense now:

Lasers and Silicon Photonics

Oh yeah, an important call out. Look at the Nvidia diagrams above again. The pluggable transceiver uses externally modulated lasers (EMLs) for 1.6Tb. These are discrete InP lasers + modulators.

CPO uses lasers differently. Instead of modulating the laser itself, it uses a simple continuous wave (CW) laser (just a constant beam of light) and performs the modulation on a silicon photonics chip on the switch substrate:

Silicon photonics is an optical circuit built in silicon using CMOS-compatible fabrication of waveguides, modulators, photodetectors:

What about serviceability though? Don’t lasers fail?

Yes, and this was one of the earliest objections to CPO. In pluggable optics, a failed laser means swapping the transceiver module which is easy to access. But if the laser is near the switch… that’s a lot harder right?

Well, don’t put the laser near the switch! The CW laser is external. So if it fails, you can still easily replace the laser source and not the switch.

And as Nvidia shows here, the silicon photonic engines themselves are designed as detachable sub-assemblies. Not as easy as swapping a front-panel pluggable, but far better than scrapping the switch:

Thermal concerns are similar. Lasers are temperature-sensitive and switch ASICs run hot; pluggable EMLs combine a laser and modulator in a single InP device running at high speed, and they run hot and are among the more failure-prone components.

But with CPO, the laser is just a simple CW source and the high-speed modulation moves to silicon photonics. If the laser sits off-package, you’ve removed the most temperature-sensitive component from the equation!

Moving optics closer to the switch looked like a reliability problem, but it may end up improving reliability instead. In fact, SemiAnalysis shared this nice slide from Meta that suggests Meta had fewer failures using CPO than with pluggables:

Trade-offs

Everything in engineering is about trade-offs, and CPO is no different.

Manufacturing is non-trivial. The silicon photonics engines sit very close to the switch ASIC, which means integrating optical and electrical components with different materials, process flows, and reliability characteristics. That complicates packaging, thermal design, and testing compared with traditional pluggable optics.

And CPO also tightens the coupling between the optics and the switch platform. With pluggables, operators can mix transceiver vendors or change optical reaches independently of the switch. In a CPO system, the optical engines are designed and qualified as part of the switch platform, which reduces that flexibility.

But that’s a manageable trade for hyperscalers, who co-design systems with their silicon partners (Broadcom, Marvell), control board layout and qualification, and deploy into environments they fully manage.

And Nvidia is bringing CPO to its merchant switch lineup (Spectrum-X Photonics, Quantum-X Photonics), which could eventually bring CPO within reach of non-hyperscalers who don’t have that kind of vertical integration. Well, they do have that kind of vertical integration… via Nvidia.

Pluggables Are Not Dead

Pluggable transceivers are EML-based InP modules with DSPs. As CPO scales, we need more silicon photonics engines and CW laser sources, and fewer DSP chips for pluggables.

People like to jump to conclusions. Remember: Copper is dead! Long live optical!

But as Vik and I discussed recently, I’d sum it up as: The answer is both. The question is when.

Yes, the direction of travel is toward tighter integration of optics and silicon. But the debate is how fast. After all, Hock Tan claimed 400G SerDes on the last Broadcom earnings call, which would extend the pluggable runway. But Broadcom is also shipping early-access CPO switches. Nvidia is shipping CPO in 2026. Marvell acquired Celestial AI to get in the game.

Today, pluggable optics concentrate value in the transceiver module. InP EML lasers, DSPs, driver and TIA chips, optical packaging. Companies like Lumentum, Coherent, Fabrinet, and DSP suppliers like Marvell and Broadcom sit in that value chain.

But with CPO, there’s an unbundling in the value chain. The optical functions move onto the switch package as silicon photonics engines, the laser becomes a separate CW source, and the DSP largely goes away.

Hence, value shifts toward silicon photonics, CW laser production, and advanced packaging, and away from standalone transceiver DSPs and pluggable module assembly.

But again, these shifts are happening over time. As I said in the video above, the transition between technologies isn’t just a binary thing on a particular date, but more of an adoption curve. Even a single hyperscaler can be deploying different technologies at different places at roughly the same time. So pluggables are still a great business.

I’ve already started pulling on these value chain threads. If you want to understand the laser side in depth, check out Lumentum and the Laser Bottleneck and Broadcom Makes Lasers, which are both directly connected to the CW laser and silicon photonics shifts we covered here. And more to come! Coherent, Applied Optoelectronics, Astera, Poet, etc. Getting lots of requests from readers :)

If you found this useful, subscribe. And if you want the deeper company-level analysis as CPO scales, consider going paid.

Subscribe now

Rivian designing its own chip was a genuine surprise. The company built its own zonal ECU architecture, so electronics has been a core competency, but custom silicon is a different level of ambition.

We start the interview by building up an understanding of the autonomy workload. What does it actually mean to do multimodal inference at the edge, when you have 11 cameras, radar, and lidar all streaming into a chip that has to make driving decisions in a 100-millisecond loop?

From there we get into RAP1 itself, including the custom neural network engine, the safety architecture, and a chip-to-chip interconnect called RivLink that hints at ambitions beyond autonomous vehicles. We discuss build vs. buy decisions, like why Rivian designed its own neural network engine and interconnect but licensed Arm cores. And a very interesting discussion about how silicon economics drove the 800 TOPS compute target.

Finally, we get to the heart of the matter. Why custom silicon over merchant? How does vertical integration unlock optimizations you simply can’t get off the shelf?

We close with the team and timeline, about 2.5 years from project go to silicon in hand. This was fun and educational. Enjoy!

This interview is lightly edited for clarity. Transcript is available for paid subscribers who prefer to read vs watch.

An Interview with Rivian’s Mukund Chavan About RAP1

Read more

Most people hear Broadcom and think Tomahawk switches, XPUs, VMware, Hock Tan. Not lasers. But Broadcom has actually been the primary 200G EML supplier to date, given its position in Nvidia’s 1.6T transceivers. It turns out there’s an interesting history and a great hidden business here.

Heck, Broadcom owns the former Bell Labs InP fab in Breinigsville, Pennsylvania. And they are hiring!

So Broadcom makes its own EMLs, VCSELs, and CW lasers, and pairs them with the industry’s leading DSPs and switch ASICs.

I’m genuinely impressed.

That’s arguably the most complete photonics stack in the industry. I had no idea. Broadcom, you should tell this story!

Broadcom’s Laser Business

Digging in, the laser business looks great:

• Total EML capacity is projected to grow from over 40 million units in 2025 to 50 million in 2026. This is across all speeds. 100G for 800G transceivers, 200G for 1.6T, and other applications

• CW laser capacity is expanding from mid-teens millions in 2025 to roughly 30 million in 2026

• Broadcom was first to ship high-volume 100G-per-lane EMLs and VCSEL technology

• Management claims leadership in 200G EMLs for 1.6T transceivers

• The networking segment, which includes these components, carries Broadcom’s highest margins outside of software

Note that Broadcom doesn’t break out its photonics revenue separately. They should! Lumentum trades at ~60x forward earnings as a pure-play laser company. Coherent trades at ~44x on vertical integration. Broadcom’s laser business... not sure how the Street values it. It’s buried inside a networking segment within a semiconductor solutions business unit. inside a $1.5T market cap conglomerate. Investors can’t price what they can’t see. I said this about Nvidia autonomy business recently too, and previously about the Nvidia networking business.

On the Q4 FY2025 earnings call, CEO Hock Tan said:

“The demand for our latest 1.6 terabit per second DSPs that enables optical interconnects for scale-out, particularly. It’s just very, very strong. And by extension, demand for the optical components like lasers, PIN diodes, just going nuts.”

Going nuts.

With EML capacity ramping 25% year-over-year and demand going nuts, photonics breakout would give the Street something to actually model in a sum-of-the-parts fashion. And Jefferies notes that Broadcom’s networking segment carries the highest margins outside software.

InP lasers are high-ASP, high-margin components, and the 100G-to-200G mix shift roughly doubles the ASP per laser. Vik and I over at Semi Doped have been very interested in exploring components companies with stacking s-curves, where each generation of new product has higher ASPs.

How does Broadcom’s laser position compare to Lumentum and Coherent?

Today I dug into the 1.6T competitive dynamics, CPO timeline, the substrate supply risk (and a good update this week from Lumentum derisking this substrate supply risk), and some bull/bear tensions.

Plus a three-way comparison between AVGO, LITE, and COHR during this optical supercycle.

Went down the rabbit hole over the past few days, thanks for coming along on the ride and learning with me!

Not financial advice. Do your own due diligence.

Read more