In this interview, we walk through Meta’s ads infrastructure from first principles. A few things that surprised me:

• Recommender workloads have a different compute-to-memory ratio than a standard LLM GPU, and this difference gave rise to MTIA custom silicon

• Retrieval isn’t a generic workload either. Meta’s scale makes it memory-bound, which is why Andromeda got its own custom NVIDIA Grace Hopper SKU that Meta co-designed

• Meta’s adaptive ranking model is an LLM-scale recommender (~1 trillion parameters) served at sub-second latency. It’s distilled from GEM, Meta’s Generative Ads Recommendation foundation model, and scales compute per user based on interaction history length

• Consolidating N ad ranking models into one (Lattice) improved performance, not just cost. A single model trained across varied objectives outperformed the specialized ones

• LLM-written kernels (Meta’s KernelEvolve) flip the economics of heterogeneous fleets. Demand for software engineering is going up as the price comes down, and Meta now wants ~100x more optimized kernels per chip

We also cover how Meta’s GenAI and recommender systems teams cross-pollinate inside Meta, and what Meta’s infrastructure looks like two years out.

This interview is lightly edited for clarity.

How Meta’s Ad System Works

Hello everyone. Today we have a special guest, Matt Steiner, VP of Monetization Infrastructure, Ranking, and AI Foundations at Meta. Welcome, Matt.

MS: Thanks, great to be here with you, Austin. Thanks for having me.

What I wanted to get out of this conversation is to better understand Meta’s core advertising business and then how that drives infrastructure decisions. I’m going to assume listeners know nothing and we’ll walk through from first principles. At the highest level, how do ads work? What are the backend models that power Meta’s ad stack?

MS: Maybe let’s start with a quick overview of how the ad system works. On a very high level, an advertiser shows up and they say, “I have some creatives with some copy and I want to show them to some people.” Sometimes they pick explicitly who they want to show them to. Sometimes they say to our ad system, “show them to whoever is most likely to convert for the objective that I specify” — whether the objective is the person visits my website, the person adds something to a shopping cart on my website, or the person actually clicks buy on my website. Those are all different objectives. Advertisers can optimize for different things.

Once the ads are created, it is our job to record who these ads should be shown to. So we produce a big database and it says, “here are all the people that the advertiser would have wanted their ad to be shown to,” and we record in each person’s little mini database, “this is an ad that could be shown to Matt the next time Matt logs in.” Of course, that list of ads that could be shown to Matt the next time Matt logs in is very, very long.

So when Matt logs in and our front end asks for an ad, whether that’s on your mobile device on Instagram or Facebook, on the web — each front end queries our backend system and says, “give me the best ads to show Matt next.” The request goes through our systems and arrives at our indexing system, and our indexing system fetches all the ads that could be shown to Matt. That is where a piece of technology that we’ve talked about recently called Meta Andromeda comes into play.

A long time ago, we had a much shorter list of ads that could be shown to Matt. Today that list is extremely long, and to be able to process all of the ads that exist in that list we need to use a fairly powerful system. We worked with our hardware partners at NVIDIA and designed a custom hardware SKU with some GPUs in it, and we co-designed a machine learning model that runs specifically on that hardware SKU for the purposes of best assessing which ads are the top N ads to rank for Matt.

In the ads serving process, the two large steps are basically: find ads that could be shown to Matt, and then rank them to produce the top ads to be shown to Matt.

Andromeda operates in the first stage, which we call retrieval, and it uses a powerful machine learning model that has embedded some of my interests and past interactions to personalize which ads should be retrieved for me. Because not every product that is advertised to me is going to be a product that is interesting to me. So we’re basically sub-selecting the products and creatives that might be interesting to me in order to return to the ranking system to rank those.

The next step is ranking, where we apply these large and powerful machine learning models to figure out what is the right order of these ads in terms of highest conversion probability times expected value for advertisers. The ad system has a number of ranking models and they rank different ads based on the objective functions for the advertiser, and we have been on a long journey to consolidate those into a single ranking model using a technology we call Lattice.

The advantage of combining ads ranking models into a single larger model is of course cost savings. You don’t have to keep N copies of user interests in each machine learning model. You can keep one copy of a person’s interests in that machine learning model, which saves memory. You can compute the subnets for a machine learning model once instead of repeatedly computing the same subnets across a bunch of different models. You just do one computation. It’s more computationally efficient to have a single model. And then the other advantage is performance — a machine learning model trained on more data with more varied objectives performs better than a smaller machine learning model trained on all the data for that objective, partly because of the compute advantages, partly because of the memory pressure advantages, partly because each piece of data has some additional signal associated with it that the machine learning model can use to improve its own performance.

So: Lattice, consolidation. And then further along in the consolidation journey, we have built GEM, our Generative Ads Recommendation Model, which is our foundation model that we’ve tried to train on all of the data that’s available for Meta’s ad system to use to improve the probability of accurately predicting what somebody’s going to be interested in, what they’re going to convert when we show them an ad for achieving an advertiser’s objective. This large foundation model was then used to distill into smaller models that we could serve for specific purposes, encoding as much information as we can from the larger foundation model.

Now, like with any system, some people use it less and some people use it more. There are people that are very interactive with brands and content and ads. They’re commenting on the ads, they’re liking the ads, they’re interacting with the brand, they’re buying things from the brand. Those power users actually have much longer interaction histories with a brand or with all the brands together. It turns out that in our original architecture design, we did not have enough compute available to process all of those interactions given our extremely limited latency budget. For example, when a person shows up in a Meta property, we want to make sure that their feed loads and their ad loads in that feed in a certain fixed latency budget — let’s call it roughly one second. We want to have sub-second latency for all of our average retrieval requests. That means we can only process so many interactions when evaluating or inferring that machine learning model.

Recently, we’ve built a new ranking model called the adaptive ranking model that substantially varies the amount of compute used to evaluate the model based on how long a sequence from a user is of their interaction history with a brand or all the brands that are advertising on Meta systems. That way we can use a dramatic amount more compute for users with longer interaction histories and meaningfully increase the accuracy of our predictions about what they’re going to interact with next. That drives better results for advertising partners and much better experiences for the people that are seeing those ads. It’s all through the magic of right-sizing the compute and memory associated with each one of those requests, and right-sizing the model based on the amount of data that’s available to evaluate for a particular person.

Okay, this is so fascinating, there’s so much here. At the highest level, you broke it down to retrieval and ranking — retrieval was Andromeda, ranking was Lattice. With Lattice, you talked about having lots of models but trying to simplify that down into one model for many reasons. And meanwhile, the whole backdrop here is — what kind of scale are we talking about again? Three-plus billion daily active users?

MS: That’s exactly right. More than three billion daily active users across Meta’s properties worldwide. A lot of people seeing a lot of organic content in their feed, a lot of paid content in their feed, and interacting with both.

Take me back to GEM and remind me — we have retrieval and ranking, and where does GEM fit in?

MS: GEM is our foundation model. It’s the model that we train with all of the data that we can use for training to produce the largest, most sophisticated, most prediction-accurate model possible. At the same time, the model is so large it’s not servable effectively. So the model has to go through a distillation stage where a lot of the core learnings of the model are distilled into smaller models that are servable.

The next step after that was to try and make the largest possible servable model on the most powerful inference hardware we have available, to produce the most accurate predictions specifically for those users who are power users. They have long interaction histories with brands and content and interests that we can really do a lot better for — deliver them much better experiences and deliver advertisers much better predictions and consequently return on advertiser spend.

Long User Histories and Adaptive Ranking

Nice, and that’s where adaptive ranking fits in. This is really interesting, because I think people are starting to get used to the idea of a foundation model that’s so big you can’t serve it, and then the consequences and trade-offs of having smaller models that are servable. For listeners thinking of generative AI, they might be thinking of smaller models that respond faster but aren’t as “intelligent.” Broadly when people are thinking about generative AI, they’re thinking about optimizing for intelligence or for interactivity — how quickly does it respond. You talked about latency, but you also talked about being willing to spend more compute at inference time to get a better outcome for the advertiser and a better experience for the user. Can you talk more about the outcomes? Why does adaptive ranking and spending more compute because you have that longer history yield a better outcome?

MS: Maybe one way to think about this is: imagine that you’re married and you have an anniversary, and every year you buy something for your spouse — something that they like that’s in their interest set that’s not necessarily in your interest set. If you can look at a long interaction history for a particular person, and you see, “every September they buy this particular class of item,” you don’t have to even know that it’s their anniversary, but you can see in that long interaction history, every September they buy something in this category. Then you can use that information to make a better prediction for what they’re likely to purchase in September.

That’s one example, but maybe you have a history of purchasing specific things in specific months corresponding to your children’s birthdays or a holiday or an anniversary. You can see how looking at longer sequences of interactions can deliver much-improved predictions about what a person is likely to want and then what a person is likely to purchase based on those longer interaction sequences.

But you can only process those longer interaction sequences if first, you’ve stored longer interaction sequences, and second, you have the computational power available at serve time to be able to process that whole interaction sequence when a person logs in. Not everybody has long interaction sequences. Not everybody interacts every month with an advertiser, but some people do, and where the data is available to deliver dramatically improved experiences for those people, you of course want to give them the best possible experience you can. That is a function of whether you have the compute available to be able to process all that information within that latency budget through parallelization, etc., that GPUs and large-scale GPUs in the inference stack now allow us to provide for people. Better providing which products and services people are interested in delivers better results for our advertising partners as well, because we’re just matchmaking. We are matching the person who wants to purchase a thing with an advertiser who has the thing to purchase.

Yes, that makes a ton of sense. For me, you’re saying: if I only look temporally at the last month of what you’ve been doing, I could give you some ads. But you’ve been on Facebook since back when you had to get invited — so if I could look all the way back, maybe there’s interesting trends. But of course the trade-off — I’m thinking about an analogy to generative AI, which everyone can relate to. It’s kind of like context. I want a big model, I want to give it a ton of context, but that’s expensive and takes time. And with user-centric social apps, you’re thinking a lot about latency. So you’ve got that constraint of what is the most context I can give it, the biggest model I can give it, but still do it in sub-one-second. That’s a perfect segue to ask you more. You talked about co-designing with NVIDIA, you talked about GPUs. Take me back — did this stuff run on CPUs at one point? How has that evolved?

From CPUs to Custom ASICs

MS: Back in the day, retrieval of course ran on CPUs. And back in the day, even ranking ran on CPUs. There was always a push to deliver more compute for both retrieval and ranking, because the more compute available, the larger, more complex machine learning model we can evaluate, the larger the user history long-sequence context windows can be passed into those models, delivering better predictions.

We’ve been on a long march through smaller CPUs, medium-sized CPUs, larger CPUs, custom ASICs, GPUs, more sophisticated and powerful GPUs, more sophisticated and powerful custom ASICs. This is all in service of delivering better results for our customers at a reasonable cost to our business so that the ROI works out on both ends for both our advertising partners and Meta.

Okay, that’s amazing. What I heard you saying was: it’s been a long history for Meta of asking “how can we get more compute to serve better ads?”, which is a win-win — you’re in a marketplace with users and businesses and you’re sitting in the middle. This idea of using compute to do predictions better has been the story of Meta’s business for quite some time.

MS: At least the last ten years, we’ve been investing really deeply in performance-optimizing the hardware, the networks, the data center designs, the silicon chips themselves, the machine learning models, the software infrastructure, the tooling associated with them. It’s a very large, complex optimization CP-SAT problem that we have to satisfy to deliver the best results for our customers and for the people that use our products and services. It’s a really fascinating technology problem in addition to a business problem.

Co-Designing with NVIDIA

Yes, indeed. It’s an intersection of both. What did the practical process of hardware-software co-design look like when you were developing the retrieval engine, like with the NVIDIA Grace Hopper?

MS: We sit down with our partners and we say, “this is the amount of compute that we want to target for this particular use case. This is the latency budget. What are the configurable blocks you have in your portfolio that you could considerably make into a SKU, whether it’s a chip level or a hardware level, that would work for this particular use case?”

Our hardware partners have various configurations of machines and chips and boards available that they are willing to build in certain configurations. We looked at that and we said, “given the retrieval problem itself, it’s going to require a huge amount of memory. It’s maybe a little bit more memory bound than it is compute bound. So we need a lot of memory. We need a lot of specifically high-bandwidth memory, so there’s enough memory channels to keep those GPUs saturated when they’re doing that computation.” We wind up with a SKU design that is optimized for the retrieval space where it has the right amount of memory, the right amount of high-bandwidth channels between the memory and the compute, and the right amount of compute that is effectively balancing that for that particular use case.

That design is maybe different than the hardware SKUs that you would use in ranking broadly or in serving a web page. But we had some great partners to work with on the hardware side. And of course, we have truly brilliant AI researchers on the modeling side, and software engineers for distributed systems that are optimizing the software infrastructure layer, and networking engineers who are optimizing how these machines talk to each other so that we can minimize end-to-end latency while maximizing the parallelism and compute we have available to deliver the best results for people and businesses.

So you sit down with your partner and say, “hey, we are a large customer. We have particular workloads that we run at scale and we know the shape of those workloads really well. This one with retrieval has these characteristics — memory bound, needs high memory capacity and bandwidth.” Does that lead you to look at those certain workloads that you have and ask, “what is the right shape of compute? What is the right SKU for retrieval versus ranking versus GEM training versus adaptive ranking?”

MS: That’s exactly right. We are always trying to work both sides of this problem. One problem is: how do we influence evolution of the hardware to better meet the needs of the software stack, and where we anticipate the software and AI stack is evolving over the next couple of years? Because — you’re probably familiar — hardware has relatively long lead times compared to software. On the other side of the problem, we are trying to influence the software stack evolution in a direction that is going to meet the hardware and maximize the potential of the hardware that’s going to be delivered to us this half, this quarter, this year, next year, and the following year.

We’re always trying to evolve them in similar directions. Sometimes there are hardware breakthroughs and we evolve our software stack to take advantage of those hardware breakthroughs. Sometimes there’s new software breakthroughs and we try to influence the hardware design in that direction to support those software breakthroughs. There’s a big discussion about this constantly across the industry. It’s particularly important given the rapid pace of innovation in the AI space — how quickly machine learning models are evolving, how quickly they are improving their performance and cost characteristics. It’s a wild time to work in the hardware-software intersection space.

MTIA: Recommender Systems vs LLMs

Totally. And obviously with transformers coming into existence, you’ve probably gone from more traditional ML into evolving toward transformer-based ones, and we’ll get there. But first, take me to MTIA. You talked about CPUs, you talked about GPUs, and how with the Grace Hopper that fit nicely into particular workloads. What leads Meta toward MTIA? There’s been a lot of announcements on that front lately — showing a roadmap, partnering with Broadcom. Can you tell us the business and economic rationale for moving in that direction?

MS: We tend to think about this in terms of the evolution of our heterogeneous hardware fleet over time. We can see the offerings that are available from our hardware partners that have various configurations of memory and compute and memory channels and different ratios. Some of them work really well for a particular use case. Some of them work really well for a different use case. There are different trade-offs with running different machine learning models on each of those hardware configurations. Sometimes the trade-off is latency. Sometimes the trade-off is cost. Sometimes the trade-off is power. In this very complex constraint satisfaction and optimization space, you’re trying to figure out what is the best offering that maximizes your returns for your advertising partners and for your business as well.

That’s where sometimes we have a use case that is different from your standard use case in the space. That was the initial impetus for the Meta Training and Inference Accelerators. Ads is a recommender systems class of problem, which is a little bit different domain than your large language model class of problems. The large language model problem is what’s known in the industry as an embarrassingly parallel problem. You can process a bunch of stuff in parallel. It doesn’t have to have super-effective high-bandwidth communication to be able to sync up the weights at periodic intervals.

At the same time, in the recommender systems space, all of the data is personalized. In the large language model space, if I was to say to somebody, “complete the sentence ‘to be or not to…’” there’s an objective correct answer — the highest probability answer that almost everybody who speaks English and has taken high school English classes could guess what the next word is going to be. A machine learning model similarly can learn there’s an objective highest probability answer to that blank in that sentence.

Now in recommender systems, the world is not objective and highest probability like that. The question is: what is the next best ad to show Matt? And it is not “what is the next best ad to show,” because who’s looking at the ad slot dramatically determines whether the ad is going to matter to them. There’s no objectively correct answer to what is the next ad to show, but there is a highest probability answer to what is the next ad to show Matt. Every example that is fed into our training systems for recommender systems has to have that personalization attached to the example.

What does that personalization look like? Well, Matt likes gardening and cycling and seems to buy a lot of stuff for toddlers, a lot of cleaning products. As a result, things that fit in those domains may be much more appealing to Matt than things that are outside of those domains. I used to have hobbies, now I have young children. That’s changed what I purchase quite a bit. The machine learning model can encode that, and it changes what the correct answer is to that question of what ad should be shown to Matt next.

That changes the size of the data packet associated with each of those examples. You have to pass in this personalization blob for the example of “we showed this ad to Matt and Matt clicked on it,” or “we showed this ad to Matt and Matt didn’t click on it. Here’s Matt’s big personalization blob of things he’s interested in.” The machine learning model can learn, “with this kind of personalization blob associated with Matt, he likes cycling and toddler toys and gardening equipment. These kinds of ads are good ads to show Matt and these kinds of ads are not good ads to show Matt.”

But that literally changes the hardware characteristics that you want when you have a very different I/O ratio associated with each example. If your examples carry a lot more data with each example, then you have to have a much fatter network pipe to keep the chip fed. You have to have more memory on the hardware SKU to keep the chip fed. You have to have a lower ratio of compute to memory — and high-bandwidth memory at that — to be able to effectively utilize the compute. So the optimal hardware SKU for training recommender systems may not be the same as a GPU that is optimized for training large language models. There’s obviously pros and cons there, but you may want to build a SKU that fits that particular workload really well.

Now that’s not all of our workloads. We obviously use GPUs in a lot of places. We use them for a lot of different parts of the recommender systems problem. But for some types of models, we have a use case for a hardware SKU that has a different configuration than what’s commonly offered as a GPU-packaged SKU. For some circumstances, a custom SKU with a different compute-to-memory ratio makes a lot of sense. For other applications, the GPU SKU is much more performant or much more cost-effective for that workload. We’re really trying to optimize the available compute and memory to the available models that need to be trained and the data size with each of those models. It’s a fascinating, challenging technology optimization problem.

Heterogeneous Hardware and LLM-Written Kernels

Yeah, that was really helpful. I like how you illustrated the problem to show that there’s specific I/O requirements and memory requirements, and how that could lead you to think about what, of all the possibilities out there, what SKU would fit best for this particular type of workload — and that might involve making your own. Now that’s talking about recommendation systems, which is really useful, and it’s a good reminder that the business involves training and inferencing recommender systems. Now, you did talk about GEM as a foundation model and needing to train that, and it being so big that it’s not cost-effective to serve. Can you tell us more about the compute challenges and the infrastructure demands on creating GEM and serving GEM?

MS: GEM as our foundation model is the largest model that we train in the ads recommender space. We try to feed it as much of our data as we can feed into the model to produce the largest, most complex, and best-predicting model that we have available. Some of the parts of the model are not super efficient, and that makes it not very effective to serve, particularly if you’re latency constrained. That’s why we had previously done this distillation process.

Now we’re using this distilled GEM variant that we’re calling the adaptive ranking model, where it’s distilled to be efficient enough to be served, but it’s not nearly as distilled as prior models, which were much smaller. The adaptive ranking model is an LLM-scale and complexity recommender model for Meta, with roughly one trillion parameters in this inference-time model. And it gets evaluated at sub-second latencies, which is a pretty fun and interesting software and hardware challenge.

Sub-second latencies — that’s amazing. You’re talking about different SKUs and different workloads, and I’m tracking all that, and you mentioned at the end of the day you have a heterogeneous silicon environment — different vendors, some home-brews, some off-the-shelf, some custom. You talked about software, and obviously having to work internally to make sure your software is going to work with the hardware and vice versa. Can you tell me more about how you manage software across all that hardware? Because to the layman, that sounds like a lot of added complexity — but I don’t know how many different levels of abstraction you can have that makes it easier.

MS: In general, heterogeneous hardware is a challenging problem to solve because you have to make sure that each of your binaries not only is capable of running on that hardware, but is performance and cost effective on that hardware. This is where folks have historically been forced to choose between custom optimization of a binary on a particular hardware type, or translation layers, which abstract away a lot of the custom features of the hardware but also abstract away a lot of the performance improvements of the hardware as well. There was a very clear spectrum of performance trade-offs between abstraction layers, which make it simpler to deploy hardware but less cost effective, and customization of binaries for hardware, which is slow and costly to implement but much more performant and cost effective once implementation is done.

Recently, machine learning models have enabled really cool abilities to customize specific binaries for hardware such that you can now at scale deploy binaries that are custom modified and performance optimized for specific types of hardware rapidly and easily, without having an expert software engineer do those performance optimizations for you. We recently put out a paper, I believe we called it Alpha Evolve or Alpha Kernel, where a machine learning model — a large language model — will write a custom performance-optimized kernel for a particular binary or machine learning model and a particular hardware pair.

If we have a large number of machine learning models and a large number of heterogeneous hardware types, writing the custom hardware kernel that would optimize the performance of this binary on the hardware was very time consuming before. It’s effectively a matrix of custom software that had to be written and hand-tuned by an expert software engineer. Now we’ve entered an era where large language models with coding capabilities can produce these optimized kernels at extremely low cost, way, way, way cheaper than having someone sit there and meticulously pick through the various optimizations necessary to make this binary or model run on this particular type of hardware.

It’s a real breakthrough in the technology industry and it’s going to enable a lot more of that cost-effective optimization that allows you to take much more advantage of all of the hardware available to you. Now we’re thinking through all of our deployments of all of our binaries to all of our hardware. Whereas before we wouldn’t necessarily move a binary that was adapted to a particular type of hardware to another type of hardware because that would be high cost and maybe it wouldn’t be worth it — now we can ask the machine learning model to produce an optimized kernel for this binary or machine learning model on this hardware, and we can do a lot more active management of software running on hardware. Which is going to both lead to better performance for our advertising partners, better experiences for people, and of course lower costs for Meta, as we get to take more advantage of the hardware we have available to us and really right-size the hardware and software use cases together. It’s a long journey, we’re not done by any stretch, but some of the new breakthroughs here in AI are having really beneficial effects on our ability to optimize our hardware and our software for our business.

GenAI Cross-Pollination and the Road Ahead

Amazing. What a world we live in. Reflecting back a bit — where my head is at is, back in the day it used to be software engineers were very expensive, and obviously Meta has probably always bought a lot of compute. But I could see the rationale for not having heterogeneous silicon because then you have to hire a bunch of software engineers if you want to optimize it for every different piece of silicon. Or on the other hand you just say, “software engineering is expensive, so we’re not going to perfectly optimize.” But at your scale you want to perfectly optimize everything so that you can eke out lower latency or better results. And interestingly, now we’re in a world where you need to buy lots and lots of hardware for your business, but the cost of software engineering has gone down to some extent with the help of generative AI LLMs, letting you still have a fleet — a matrix of different tasks and different hardware — and yet you can use LLMs to help optimize and fill out that spreadsheet in a cost-effective way. Which is very awesome. That leads me to the question about generative AI. How is Meta thinking about the relationship between its core recommendation systems and infrastructure and the investments in generative AI? Not only using generative AI in your core business, which alone is really cool and interesting, but also I know that you are training generative AI and offering that to customers.

MS: There is a lot of crosstalk between our various AI experts in the generative AI / large language model world and in our recommender systems world. Not only is there crosstalk and collaboration on hardware and data center design and performance optimization for the distributed systems, including things like the model trainer — we are both really focused on optimizing the machine learning model trainer and optimizing various aspects of the performance that the system needs to be able to train much larger models and serve much larger models. There’s a huge amount of joint investment that effectively benefits both sides of the house, the large language model side of the house and the recommender system side of the house.

We have experts in both types of ranking on both sides of the house so that we can improve the performance using both domains’ techniques and capabilities. We are — maybe as evidenced by the pace of breakthroughs that we’re able to deploy in our services here — really seeing the benefits of the innovation in the AI space across both parts of the business today. That’s obviously very exciting. This is the weirdest, wackiest, most fun time to be a software engineer ever.

Yes, seriously. It’s fascinating to think about those different sides of the house and how they cross-pollinate and impact each other, and just how fast both are moving. What an awesome time to be at Meta, and what a crazy time. Last question — looking forward, maybe two years because the rate of change makes it hard to look further than that — what do you see as the primary infrastructure needs for the next generation of AI-driven advertising?

MS: You can see we are all investing very heavily in building out data centers and purchasing large quantities of compute and memory and storage so that we can build better machine learning models, so we can find better machine learning models. The process of identifying performance improvements is really training a lot of machine learning models, tweaking various optimization parameters, coming up with new architectures and testing those to really drive maximum performance benefits. So, large investments in machine learning model training, machine learning model research that leads to performance improvements for training, that lead to performance improvements at inference time, substantial investments to make sure that we can infer these large language models and other generative models and ranking models both more cost effectively, but also driving more compute available at serve time and more memory available at serve time so we can feed things like longer sequence histories and larger context windows into these models.

The overarching theme here is end-to-end optimization. We’re trying to optimize the data center designs with the networking designs and the SKU designs and the software infrastructure designs for the distributed systems and the machine learning model infrastructure, the machine learning models themselves, the data that goes into them — all jointly, so we can drive maximum performance together.

Maybe to your point earlier, the demand for software engineering has effectively gone through the roof as the price has gone down. Whereas before we would invest in a limited number of hardware optimization kernels to run software on, now we want 100 times as many software optimization kernels for each piece of hardware because it’s available now. We can have machine learning models produce that, and now we have our expert hardware performance tuners supervising these models instead of writing the optimizations themselves. The same thing is true at every layer of the stack where we’re doing this optimization now. The demand for custom software that is more performant than a generic abstraction layer has gone through the roof. Every team at every layer is trying to do much better optimization to produce better results per dollar, better results per watt of power used in these data centers. That’s really leading to these meaningful breakthroughs that you’re seeing in terms of performance all across the industry, but particularly for the business as well.

Yeah, what a wild cross-optimization problem, being vertically integrated in some respects from hardware through data center design all the way to the software, to the training and the inference. And then being able to use LLMs to help with all this super fast. What I like about what you’re talking about here is: you have to make all of these trade-off decisions, but there’s a clear optimization function that you’re solving for when you’re thinking of an ad-space business — an ROI, how much are you willing to spend, how much are they willing to pay, and how can better results lead to potentially paying more or the pie growing bigger. I’m just thinking out loud, contrasting that to maybe other players in the generative AI space where the economics aren’t quite as straightforward in making these decisions. Anyway — you guys have a lot to think through. My very final question for you personally: how do you stay on top of it all as it’s changing so fast up and down the stack?

MS: That’s a great question. I don’t think I have a fantastic answer. The rate of change is amazing. I try to use all of the AI tools available, including large language models, to summarize papers, produce a list of all the latest papers that have come out with breakthroughs that are relevant to the domain that I work in. I rely on a brilliant team of expert AI researchers to summarize the progress that’s happening in the space, how that should influence the roadmap that we’re building for the future. But the amount of information and the progress in the space is just wild. It’s really amazing and something to behold.

Yes, totally. Well, you don’t sound bored, that’s for sure. Awesome. That’s it for today. Thanks so much, Matt, for taking the time to educate us. I’ve learned a lot and I know everyone will really get something out of this, so thank you.

MS: Definitely not. Thank you for having me, Austin. Great to chat with you.

Substrate is controversial. The debate tends to focus on individual objections, such as a lack of industry experience or the impracticality of particle accelerators. But I want to zoom out and look at the elephant as a whole:

To do that, let’s use a simple framework to test whether Substrate’s strategy is actually sound.

How to Solve Problems

A while back, I was listening to an old Lex Fridman podcast with Ray Dalio. And Ray was talking about his “5-Step Process”:

1. Set your goal

2. Identify the problems blocking it

3. Diagnose the root cause

4. Design around it

5. Follow through

Not rocket science. But there’s power in simple frameworks as a lens to view the world.

These five steps resonated with me, as they pattern-match well with my lived experiences (entrepreneurship, engineering, academic research, home projects, and so on). Once I heard it, I started seeing it everywhere. Listening through the How I Built This backlog, I found it in the Crate & Barrel founding story from the 1960s. Let me show you how it applies there, and then we’ll try it on Substrate.

Crate & Barrel

Quick context: Summer 1961. Gordon and Carol Segal are getting married. Both 23. They registered for stylish European housewares, but nobody in their lives had the money or taste to buy it for them. The Segals couldn’t afford it either.

But on their honeymoon in the Caribbean, they found the same products at a fraction of US prices:

Gordon Segal: There was one Scandinavian store in the Virgin Islands. My wife picked up some of the items and said, “How can you have Danish 18/8 stainless at the $2.95 a place setting? It’s so much more expensive in America!”

And the Danish merchant there said, “We have salesmen from Europe come here, and we buy direct from factories.”

Ah! A goal, a problem, a root cause. Gordon’s wheels started spinning:

We got back to Chicago, and I was in the real estate business. She was teaching school. We were both sort of bored. And then, one night in February of ‘62, I was washing these dishes we had bought. I said, “You know, Carol, there had to be other young people like ourselves with good taste and no money. We should open a store.”

Through the Dalio lens (and listening to the rest of the podcast):

• Goal: Sell beautiful European housewares to young people with good taste and no money.

• Problem: Those goods were priced out of reach in the US, even though they were affordable in Europe and the Caribbean.

• Root cause: Middlemen in the US import distribution chain.

• Design: Skip the importers. Buy direct from European factories.

• Follow through: Hard work and grit.

The Segals didn’t pencil out this strategy cleanly from Day 1. But the Dalio process was at work.

Now let’s apply it to Substrate.

Substrate

Substrate CEO James Proud made the goal very clear in this Stratechery interview: revive American chip manufacturing leadership.

WTF? REVIVE AMERICAN CHIP MANUFACTURING? WHO DOES HE THINK HE IS?

Before you grab pitchforks, let’s work through the reasoning. We’ll address “yeah but no industry experience” and the rest later. First, the 5-step process.

Goal: American leading-edge semiconductor manufacturing

What obstacles stand in the way?

Money and talent come to mind first. But those aren’t fundamental bottlenecks. Think about Elon and Terafab. Money and talent are tractable.

Dig deeper. Assume you’re well-capitalized and talent-rich. OK, this sounds like Rapidus. We can cover them in another article.

Now what?

You know what’s actually hard? Creating a customer.

Problem: No one will work with you

Customer acquisition is the problem.

How could you possibly incentivize chip designers to use a brand-new, leading-edge foundry? You need a reason so compelling that it overcomes the risk premium.

On what plane can you even outperform TSMC?! They have 30+ years of process knowledge and relationships with every major chip company on Earth. And don’t say supply chain security. TSMC has n-1 capacity in Arizona and Intel Foundry is getting its swagger back…

Hmm…

Well, what pain points do chip designers have with TSMC today?

Cost is a big one.

The Cost Problem

Many companies can’t afford leading-edge nodes, and even those that can only use them for a select few SKUs. Design costs run in the hundreds of millions. Mask sets cost tens of millions. Only products with massive volume (smartphones) or high ASPs (Nvidia GPUs) can amortize that cost. And variable costs compound it. Leading-edge wafers are north of $20K and all signs point toward $100K by the end of the decade. So even for the highest-volume products, where fixed costs amortize to near zero, wafer price still matters.

Why is the leading edge so expensive? Lithography.

EUV tools cost hundreds of millions each and the cost is only going up. You need dozens per fab, a significant driver of leading edge fab requires tens of billions to build. Deeper background reading here:

But… what if you could somehow reduce the cost of lithography drastically?

What if you could offer a value proposition of “2nm wafers at 28nm prices”?

You could create customers. Existing medium and low volume SKUs that would love to use smaller, more power efficient transistors or build their own custom ASICs instead of using less efficient/broader off-the-shelf options.

Yeah, but EUV LITHOGRAPHY IS MAGIC! TIN DROPLETS! 30+ YEARS! ASML! NO WAY!

Yes, I know. Suspend disbelief for a bit and just follow Dalio’s process. Let’s pull on the thread more.

What is the root cause of the lithography economics problem?

XLight has rightly pointed out that Laser-Produced Plasma (LPP) is very expensive. The “shoot tin droplets and hit them twice with a laser to generate EUV light” part. And every ASML EUV machine needs one.

XLight says, “Why not use a much higher power free electron laser (FEL) and share that light source amongst many EUV scanners?” This unlocks much better economics, not only from decoupling the light source from the scanner but also by increase the dose which impacts productivity and wafer economics.

See more here:

But what if you pull on the root cause even further? Could it lead to a different solution?

The entire EUV system (light source, optics, scanner) is incredibly complex with known inefficiencies (many mirrors, lots of lost light…) all driving extreme cost.

Is that cost due to physics, meaning this is the globally optimal solution, and there is a physical limit preventing lithography from ever working differently?

Or did path dependence lead us to a local minimum, not a global one?

Is the root cause physics, or path dependence?

Retracing The Path

Substrate believes it’s path dependence. And you can actually retrace the history to test that claim. In the ‘80s and ‘90s, the industry was actively researching X-ray lithography, which has a natural resolution advantage from its shorter wavelength.

Hmm… maybe we could learn what the shortcomings were and if they are till true today!

There are great old papers on this, especially from IBM Research. Check out this paper “X-ray lithography in IBM, 1980-1992, the development years”. It’s super enlightening. Let’s keep pulling on the thread, from Alan D. Wilson’s paper:

Optical lithography, in 1980, was very poorly understood by the experts. On the basis of historical trends and current difficulties with existing tooling and technology, the limits of lithography were thought to be about 1–1.25 µm for optics. X-ray lithography, consequently, was targeted for entry around 1 µm, the perceived limit of optical lithography.

At the onset of the program the strategic advantages of X-ray lithography were stated to be high resolution (better than optical lithography), throughput superior to that of e-beam technology, better resist-processing characteristics, and potentially lower defects (no multilayer resists).

X-ray seemed promising at the time. What did they learn? How did they learn? Were they thinking about manufacturability or just science?

We spent the remainder of 1980 developing a financial and technical program plan for X-ray lithography based on synchrotrons, considering a number of basic questions: Where were exposures going to be done? What should the mask be made from, and how would it look? How would we develop a stepper/aligner system, and what would be the role of vendor assistance in this regard? What would be the staffing needs (the initial group included only six people) as the program progressed? And finally, what should the test vehicle be?

It was recognized early in the drafting of our program that we were targeting manufacturing, not device prototyping, but full manufacturing. Our manufacturing divisions would eventually be our customer.

That’s really sound and reasonable thinking. They were thinking about full production scale and manufacturability, not just the science. And if you read the rest of the paper, the X-ray lithography (XRL) technology actually worked:

We had made complex, fully scaled CMOS devices with 0.5-μm ground rules before our optical counterparts had reached the same level using a long-established technology. Our yield was also acceptable: not 100%, but acceptable. The principal goal of the X-ray program had been achieved.

There are even tips for Substrate and us to think consider regarding a practical particle accelerator:

For X-ray lithography to be viable in IBM, we needed to explore acquiring our own X-ray source… Early in 1982, Grobman and I were starting to learn about the physics of synchrotrons. Our interest in a synchrotron ring for IBM was kindled by reports from Munich of the design of a tabletop machine called Kleine-Erna…. We began this serious inquiry by visits to established rings in this country and in Europe. Perhaps not being part of the synchrotron establishment was beneficial: We could ask questions and find out what really made rings good and what did not, as well as who the real experts were.

A quick aside… Agree. Perhaps not being part of the establishment is a benefit at times. … And figure out who the *real* experts are 😂

This IBM researcher Alan D. Wilson is my dude.

Anyway, he continues

Armed with this information, we asked ourselves what the specifications for the ring should be… I was invariably asked “What should a ring for industry be?” My answer was this: It should fit on a truck; plug into a wall socket; be reliable and available to operate 20 out of 21 shifts per week; have sufficient average output capable of sustaining a stepper throughput of more than 30 wafers per hour using an insensitive (having a sensitivity of -100 mJ/cm^) X-ray resist; be capable of being debugged/commissioned and assembled at the vendor, shipped intact to an IBM site, and rendered operational in a reasonable time at full specifications.

Make sure it fits on a truck and plugs into a wall socket. Quite practical!

And that kind of thinking is a lot different than the picture of particle accelerators I had in my head (i.e. CERN).

Oh last quick note from the paper, it seems that folks exploring XRL have always been doubted.

The establishment of a program was a formidable task, since half of this distinguished group seriously questioned the need for and the viability of X-ray lithography.

But IBM actually built a synchrotron (“Helios”) that fit on a truck and worked:

Oxford proposed a superconducting dipole system with a cold bore. The magnet turned out to be difiicult to construct but had excellent performance. The Helios 1 ring was completed and commissioned at Oxford, England, in October 1990. During the design and building of the synchrotron we visited Oxford and the Daresbury team every four to eight weeks over a period of three and a half years. The ring was shipped to IBM in March 1991 and arrived at East Fishkill on March 29, 1991. It fit on a truck, as shown in Figure 12(a), and we slid it into ALF that same day [Figure 12(b)].

The first beam was stored on or about May 20, 1991, and final specifications were met in January 1992. Our goal had been met and, in fact, exceeded, because the ring performs beyond specification [36]. Critics who thought design alone would not work were wrong. IBM and Oxford as a development team commissioned the Oxford ring in record time and with a very high level of performance.

If it worked back then… why don’t we have X-ray lithography today? XRL wasn’t without practical shortcomings at the time. This IBM paper from 1998 said that XRL was mature enough to possibly be introduced at 130nm node, but admitted manufacturing issues, for example with masks.

Nonetheless, there are challenges still to be met. Among the most important are the development and commercial availability of an improved e-beam mask writer; the ability to fabricate defect-free masks satisfying the image placement and critical dimension control requirements with good yields; the stability of the masks in usage (including the issue of possible radiation damage); the ability to correct for magnification errors; and the ability to satisfy the industry’s desire for a technology extendible to 70 nm ground rules. These issues are primarily manufacturing issues, as opposed to issues related to demonstrating proof-of-concept or feasibility, although demonstrating extendibility is still needed before the industry can commit to using XRL at 70 nm ground rules

XRL was physically feasible but had engineering problems to solve at production scale.

Meanwhile, optical lithography kept working far beyond the ~1μm wall Wilson predicted. So the industry kept pushing optical. First DUV, then EUV.

But a lot has changed in 35 years. US National Labs have spent years advancing particle accelerators and sources are now brighter, more reliable, and more compact. Computational lithography has improved significantly, too, and can help overcome the mask and proximity challenges that plagued IBM.

Substrate’s Design

So back to the Dallio process. The bottleneck is EUV-based lithography economics, and Substrate’s approach is to go back to the fork in the road and choose a different path. Could there be a global optimum, and could it be XRL?

It would require co-designing the light source, optics, and scanner as an integrated whole from scratch. But to me, there seem to be many cost savings possibilities on the table:

• One particle accelerator source feeds many scanners, like IBM’s Helios ring which had 16 beamline ports. The light source cost amortizes across many tools.

• No multilayer mirrors means no compounding reflectivity losses. Almost all generated photons are available at the wafer, vs. single-digit percent for EUV.

• No tin-droplet plasma source means no tin contamination, no collector degradation, no droplet generator maintenance.

• Single-patterning at leading-edge nodes. Fewer exposures = fewer masks, fewer etch steps, fewer defect opportunities, faster cycle time.

• In theory, the particle accelerator shouldn’t need cleanroom space. Only the wafer-handling end of the beamline sits in the cleanroom. This is contrary to EUV scanners which are massive and require substantial fab and subfab infrastructure:

So here’s the chain of reasoning so far:

• Goal: American leading-edge semiconductor manufacturing

• Problem: No one will work with a new foundry

• Root cause: The only lever that overcomes the risk premium is dramatically lower cost, and cost is dominated by lithography

• Root cause (a layer deeper): EUV’s cost comes from path dependence, not physics

• Design: Go back to the 1990 fork. X-ray lithography with modern sources.

This is sound. It sure seems XRL could genuinely untangle the lithography cost problem.

It’s believable. And it’s fundable. Substrate raised $100M from Founders Fund, General Catalysts, In-Q-Tel, and more. Founder’s Fund’s thesis is to invest in smart people solving difficult scientific problems where, if they succeed, the technology would be extraordinarily valuable. Substrate is a perfect fit. A 1% chance of reshaping the $1T+ semiconductor industry seems like just the type of asymmetric bet FF was built to make.

Of course, sound strategy doesn’t guarantee success. Execution is everything.

So can they actually pull this off? Behind the paywall I address the biggest objections head-on, work through whether XLight and Substrate can both win, and discuss the impact to TSMC and ASML.

It’s really, really interesting.

Read more

In this interview we discuss why Pope left Google to start a chip company, how to overcome the CUDA lock-in, and why frontier labs are the natural first customers. We get into the chip itself: a hybrid SRAM-HBM memory architecture that combines the low latency of Cerebras and Groq with the throughput of traditional HBM designs, and why that unlocks advantages across training, prefill, and decode. We also cover how agentic AI changes hardware requirements, how MatX uses AI internally in chip design, and the biggest skepticism Pope hears: can a 100-person startup manufacture at datacenter scale?

This interview is lightly edited for clarity.

Subscribe now

Origin Story

Hello listeners, we have a special guest today, co-founder and CEO of MatX, Reiner Pope. Welcome Reiner, for listeners who haven’t heard of you and MatX, who are you, what is MatX, what are you guys trying to do?

RP: Thanks, very happy to be here. As you mentioned, I’m CEO at MatX. What we’re doing is making the best chips for LLMs that is allowable by physics. My co-founder Mike Gunter and I, prior to MatX, were working at Google for a long time. Most recently, I was on the Google Brain Team training one of the LLMs at the time, and Mike was on the TPU team. There were a lot of things we wanted to do to make the TPUs much better for running LLMs. Things like running at much lower precision, having much more compute performance based on large matrix support, and generally optimizing for LLMs, reducing a lot of the other circuitry that was needed for non-LLM workloads. At the time, this was in 2022, and it turned out the best way to do this would be by starting a separate company, which is MatX.

So take me back, you mentioned 2022, you came out of Google, which I will say, it seems like everyone came out of Google that’s at the forefront of AI and hardware.

RP: It’s like the Bell Labs of the time.

Yes! There’ll be a book written 10, 15 years from now that we’ll get to go back and read and it’ll be fun for us to remember the good old days.

But okay, back to 2022. I think it was November 30th when ChatGPT officially launched. How much ahead of that were you guys thinking about this direction? Did you launch before ChatGPT? And how did that inflection point—the general public becoming aware of transformers—how much did that change your life in terms of fundraising, vision casting, hiring?

RP: As it happened, we left Google one week before ChatGPT was released. We did not know it was coming. But the historical context was that GPT-3 had been released more than a year earlier in this developer demo.

It was really hard to use. You had to get in the mindset of “I am writing a document and I want the rest of this document to be the response I’m looking for.” It’s not a chat interface at all, totally different. But if you were paying a lot of attention, you could see the potential. A lot of insiders in the industry were appreciating something big is happening here.

And the question, really, pre-ChatGPT was: these models are incredible, but they’re 100 times more expensive than the models we’re used to running. There are 100 billion parameters instead of under a billion. Can we even afford to run them? The simple economics doesn’t work out if you’re used to running software as a service where every query is free and now you have to spend cents per query. When you’ve got millions of queries per second, it doesn’t pencil out. The big question prior to ChatGPT was: okay, cool demo, but it’s too expensive. Can you actually productize it? And there was a lot of skepticism that you actually could.

ChatGPT demonstrated that you can, and not only that, but the product is incredibly valuable. What that meant for us was we had already seen that prices are going to be high. If prices are high, how do you make them cheaper?

It turned out to be quite difficult for us to fundraise even after ChatGPT. It took about two quarters for that to really land, when the impact on Nvidia’s stock price showed up. Then there was the realization: okay, this is using a ton of GPUs, everyone is buying a ton of GPUs. Eventually Nvidia reported these gangbusters quarters, and at that point investors started seeing the potential.

Okay, interesting. So you started by saying this is really transformational, but on the current hardware, it’s going to be too expensive. So there’s got to be a better hardware solution. Then ChatGPT launches a week after you guys leave, and I would expect investors to say, “I can see this is going to be productized!” But at the same time, Nvidia is the one capturing all the value and selling GPUs. So was the early skepticism just around: why will anyone buy hardware that’s not a GPU? Or did they quickly connect the dots that GPUs aren’t necessarily the most efficient?

RP: Some of the skepticism is definitely about why you would buy hardware that’s not a GPU. And then the other one is just: how do you compete with the world’s biggest company?

On the “why would you buy something that’s not a GPU” question, the big consideration is the software moat that Nvidia has. Everyone writes CUDA. Historically, we’ve seen how much software lock-in there is in so many businesses. Why is this one different? Isn’t there going to be software lock-in here? Would everyone really rewrite their software onto a different hardware platform?

CUDA Lock-In

Is there lock-in? How are you thinking about it from a software perspective?

RP: At this point, I think it’s proven that the lock-in is pretty weak. Barring Google, who has been on TPUs forever, all of the other frontier labs are multi-platform. OpenAI, Anthropic, Meta, X — they are all on Nvidia, many of them are on TPUs. There are Cerebras announcements, AMD, some Broadcom-developed chips as well. All of these players are multi-platform. That is the proof already that the software lock-in is not that great.

If you want to think about the first principles reasons why, it’s because software versus hardware lock-in is really a question of how much spend you’re putting on the hardware versus how much you’re putting on software engineering to support the hardware. This is really the first time that balance has changed, and it has violated a lot of people’s intuitions.

Historically, the whole history of software as a service is you’re paying really large salaries to a large software engineering team, and the compute spend is a small fraction of that. Engineering time is precious is the mantra. Of course there you have to prioritize the ease of software.

But this is totally turned around now. All of the frontier labs are spending tens of billions of dollars on compute. The salaries of the people writing software for that compute are very high, but still small in comparison to the compute spend. So the rational choice is to do anything you can to get hardware costs down, be multi-platform, get the negotiating power that comes from that.

I see, interesting. From first principles, it makes a lot of sense. Now that you’re going to spend so much money on hardware, how can you spend it correctly on software to unlock that? Even if it means you have a team writing kernels specifically for this architecture.

Fast forward from 2022 to now and we’re seeing everyone has multi-vendor silicon and it’s made your point. It’s very easy for you now. But back then, when you’re just starting and trying to raise that Series A, you clearly were trying to articulate that and hope that it came to fruition. Of your early investors, some of them must have believed. What got them to believe you in a world where it looked like Nvidia had all the GPUs and had the lock-in?

RP: Ultimately, all of early investing is primarily a bet on people rather than on technology. There’s a bit of both — you can have the best people in the world and have a business plan which doesn’t make any sense at all. But the premise that there is a physical product that we make that we will sell for dollars is a very easy business plan. It’s clear how you can make margins off of that.

In some sense, that’s even an easier business plan than starting a frontier lab. A frontier lab is like, “we’re going to make a model, we hope we can sell it in a product that hasn’t been defined yet.” With selling hardware, at least the business case is clear.

And for early seed stage investors, it’s primarily going off who we are, our backgrounds, and folks we’ve worked with who have vouched for us.

The Chip

And of course you have the credibility of having been TPU people at Google. Tell me, actually really quick question. I don’t know if I’ve heard you say this anywhere. Explain the name MatX.

RP: Matrix multiply. One angle is you remove “ri” from matrix. Another one is the X is a “times.”

Nice. So now take us into the first chip, the MatX One. I know you raised $100 million to get started, and then just a couple of months ago raised $500 million. We talk about [using that money to build] a chip, but I know you’re actually building a system. The goal is data center deployments. So with all of that context, tell me about the chip, but I want to get into the bigger system.

RP: A few of the core bets of the chip: primarily very high matrix multiply performance, higher than anyone else has announced in the market. There’s a whole story there, but in summary, the marginal returns on having more matrix multiply performance seem to be much higher than marginal returns on more HBM performance or other considerations. So you’ve got to invest in that first.

And then there’s this thing that had been like free money sitting on the table: get your memory system right. That is a combination of seeing two good ideas in the market. Nvidia, Google, Amazon have been all tensors in HBM — HBM first. Cerebras and Groq have been weights in SRAM. That gives you very low latencies, but it has some capacity problems. You can put those two together. It takes careful engineering and you need to balance the system right. It’s hard to balance the system right. But it is totally doable. That is the other thing we’ve done, and it gives some really big advantages in both latency and throughput.

I think a lot of people are now starting to connect with that as they see the Groq LPUs and Cerebras; they see the benefit of weights in SRAM for low latency. But of course you need HBM for high throughput and KV caches. Everyone’s starting to realize that context is awesome — the more context you can give a model, the more interesting insights you get. You made the right bet. Was that an architectural bet made from day one, based on first principles?

RP: Yes. One of the things we’re very good at is workload mapping to hardware, and creative new ways to do that that are more optimal, especially when you consider the space of what potential hardware could be. This combination of different memory systems was a core idea going in.

One of the things it really enables — if you look through the list of parallelism and partitioning techniques: tensor parallelism, expert parallelism, pipeline parallelism. The last one is the ugly stepchild in some sense. It misses a lot of the advantages of optimizing latency and memory footprint that the other ones do. It turns out that’s actually a memory system choice. This combination of SRAM and HBM actually makes pipelining work as well as the other techniques for the first time ever. We understood that, and that was what we were going after.

So back in 2022 when you’re making these early architectural decisions—the big systolic array, the right memory choice—you’re also thinking about mixture of experts and how different parallelism strategies require tuning those memory choices correctly. That’s IP and a differentiator for you versus someone who just says, “oh, weights in SRAM and HBM, let me go do the same thing.”

But reflecting back to 2022, I’m not sure mixture of experts was even out yet. So how much are you reading papers as stuff was happening in ‘22, ‘23, ‘24 and saying, do we need to tweak the architecture?

RP: We’ve been reading papers since 2017. I think the big and disappointing inflection point in 2022 was when Google stopped publishing. We were talking about how Google is where all the researchers came from. They had an incredible team in Google Brain and they were publishing everything, all of the good work they did. Very vibrant place to be. They stopped doing that in 2022 because of seeing the competitive market playing out. You could get all of the trend lines of where the best models are going until then, and then that stopped. DeepSeek publishing has been a pretty good reboot of that, but it’s sad that the volume has not been so large.

Research and Publishing

Totally. I will admit I haven’t read all of your papers on your website, but I see that you guys do some publishing still. How are you thinking about that fine line of what to publish and what not to? Because for talent, it is exciting to get to publish to the world and share what you’re thinking about.

RP: The ability to publish neural net papers is a differentiator for us in terms of hiring. We have two different areas of neural net research. We’re a small company, especially our ML team is very small because that is part of what we do, but it is not the main thing we do. We’re not selling ML, we’re selling GEMMs.

But the agenda of our ML team is twofold. First is attention research, specifically focusing on memory bandwidth efficient attention. That is quite aligned to where we see the future of hardware being. The second is numerics. Numerics has been the single best improvement in chip performance over the last decade. I think we have some of the best numerics talent and IP here.

In terms of what we publish: we don’t currently publish the numerics that goes into our chip. We will probably publish it on a one or two year delay after releasing the chip. But we do publish all of the attention research, because what we’re doing there is advocacy. We’re saying: hey, model designers, you should probably have these considerations in mind, especially when you think of future hardware that’s going to have a ton of flops but is going to be somewhat more memory bandwidth constrained.

Product Positioning

So you’re making hardware to sell at the end of the day, but you have ML researchers working on attention, memory bandwidth-efficient attention, and numerics. That informs your own architecture — extreme co-design. But you’re also trying to show model labs—the end customer—what’s possible. If they adopt your chips, how much will that change how they think about training or inference?

RP: We’re trying to not go too far outside of the comfort zone. If you want product-market fit, you have to mostly meet the customer where they are.

The way to quantify that: you can look at the chip specs and there are maybe five most important ones — HBM bandwidth and capacity, matrix multiply throughput, SRAM bandwidth and capacity, interconnect performance. Our attitude is we want to be at least on par with the best competition like Nvidia on all of these, and then substantially ahead on at least a few. The substantially ahead for us is obviously matrix multiply performance, also interconnect performance and SRAM.

There is no place where we are substantially behind in these big considerations. Maybe in some less LLM-relevant considerations we’re behind, but in these big five, we’re at least on par everywhere. That means the opportunity cost of switching to MatX is never too large.

But then the headroom you can get — if you want to maximize the benefit, you can tune your model. That means things like changing the balance between the MLP layer and the attention, more MLP less attention, or using some of our lower precision arithmetic. We have a range of precisions to get the biggest advantages out.

Gotcha. So you make sure that in these five most important areas, none of them are too weak to prevent a customer from switching. You’ll be there on every front. But then if customers take a step further and optimize for your chips, they’ll have more headroom, they can do more.

RP: Yeah, that’s it.

Customers and Workloads

Let me segue into who are those customers in broad strokes, the target customers for this chip system.

RP: The most interest has been from frontier labs, which is as expected. That is who we are designing for, and why they’re most interested is their spend is biggest. That also means the economics of being willing to tolerate a new software stack is biggest there too.

They also have this longer-term vision of three to five years out, which is where you need to be when you’re buying custom hardware. If you want to do really good co-design with your hardware provider, you need to be thinking on that time scale rather than just “I’ll buy what’s on the shelf today.” That’s where we’ve seen strong interest.

And this has shown up across all of the workloads — training, reinforcement learning, and inference both prefill and decode.

Nice, okay, let’s talk about those workloads.

Let me reflect it back. Your customers are going to be the frontier labs. They have the most compute spend, they are the most incentivized to squeeze as much intelligence as they can out of that. They’re thinking three to five years ahead. They are incentivized to not only work with all their current partners, but to always be listening and see what else is out there.

The market is telling us the defining workload of our time is LLM inference. You can optimize around the transformer, around splitting it into prefill and decode. We see that with Nvidia and with Dynamo. Everyone’s getting used to that concept.

The market narrative has gone from GPUs for everything to actually at the rack scale, maybe it makes sense to have some SKUs that run prefill and some that run decode. This is their way of saying those sub-workloads have different constraints — if it’s memory bound, have the right hardware versus compute bound. But I know you had a great podcast that everyone should go listen to with John Collison and Cheeky Pint. You talked there about being competitive on all those workloads — training, prefill, decode, RL. And it kind of felt like going back to the days of a GPU can do everything. So how are you talking with these partners about their different workloads, and how do you not feel like a salesman just saying “yeah, we can do that, we can do that, we can do that”?

RP: We just have to be honest about what the strengths and weaknesses are. Let’s give that a shot here. Our product has a really large amount of compute. Traditionally, training and inference prefill are the compute-intensive workloads, and decode is memory bandwidth-intensive. So you might think, MatX has a lot of compute, why would we use that on a memory bandwidth intensive workload like decode?

That’s where the joint hybrid SRAM-HBM design really shines. You spend none of your HBM bandwidth on loading weights. All of that bandwidth is spent entirely on KV cache. So you can get better use out of your HBM bandwidth than you can with Nvidia. But you also get the very low latency because the weights are stored in SRAM, like Cerebras and Groq.

Digging into that further: low latency means small batch sizes — that’s just Little’s law. The number of things in flight are smaller. The memory occupancy in HBM is proportional to batch size. So you can actually fit longer contexts in HBM than you could if the latency were larger. Low latency is not just a usability win, but it actually improves your throughput as well.

This is similar to what Nvidia is now doing with the Groq and Nvidia racks side by side, but there are some taxes you pay by them being in different packages. Putting the whole thing in one package is the first principles way to do that and gives you the most advantages.

Sure, that makes sense. You have a lot of compute. You make the right memory choices. Therefore you can do low latency and high throughput. And there are even benefits in the small batch size, low latency with respect to how the HBM is used. You talked about how Nvidia has essentially separate racks, the Groq rack in there, say Vera Rubin. You’re making one chip with benefits to both types of workload. How are you thinking about rack scale, interconnect, scale up, scale out?

RP: We have a lot of interconnect in the product. I think it is the most of any announced product, in fact. The reason: so you can support mixture of expert models with fairly small experts without becoming communication limited. Very sparse mixture of expert models are what primarily drive the interconnect requirements.

We deploy very large scale-up domains as well as supporting scale-out. The sizing of your scale-up domain is really driven by the sparsity and the kind of mixture of expert layers you want to support. You want to do the mixture of expert routing within your scale-up domain as much as possible — that is how everyone does it. Bigger scale-up domains allow bigger mixture of expert layers.

On topology, we do some interesting things with network topology. I won’t go into huge specifics, but contrasting what is in the market: Nvidia has done things like running everything through the NV switches. Google has these torus topologies. If you think about what you really want for mixture of expert layers, you can design something very custom for that.

I see, nice. That again aligns with the idea of designing not just the chip but the whole system for the specific workload, even down to network topology. That makes a lot of sense.

The Team

So how many people, even hand-wavy, do you have at MatX? We’re talking about networking, ML, hardware. Probably you have to think about cooling and operations and all sorts of stuff because it’s really data center design. Tell me more about the company. It must be very cross-functional — what’s it like there?

RP: For a product like this, it’s a relatively small team. It’s over 100 people. But some of these projects — Nvidia has 10,000 or 20,000 people.

Most of the team is hardware, which includes the core chip itself, the logic design, design verification, physical design, and so on. We designed the rack in conjunction with a partner as well. So we have folks looking at what is the insertion force of a board into a rack, cable density, power delivery, thermals. That’s going down the stack.

Going up the stack, we have a really strong software team writing the software stack that runs LLMs on our chip. And then we have the ML team doing exactly the research agenda I described. Very cross-disciplinary. I think it’s a super fun place to work because in one day you’ll have a conversation about physical insertion forces and at the same time functional programming or SAT solvers for compilers.

Agentic AI

Nice, sounds fun. So I’m thinking about your interdisciplinary team, everything you’re trying to build in your first system. And at the same time, the world is constantly changing. We’ve got agentic AI, Claude Code, OpenAI Codex, maybe an explosion of inference tokens needed. Opus is awesome but expensive. I can’t use my Mac subscription for Claude Code. All of a sudden Mythos has come out. And I’m wondering as a chip designer with ML researchers, how are you staying on top of all this? Are things changing that make you think in the next version of our chip we should do things differently? Or are you seeing it play out and feeling pretty confident, like, we can help this problem of awesome but expensive inference?

RP: Halfway through your question, I was like, is this going to be about how do we use agentic AI versus how do we serve it? Both are interesting.

How do we serve it: there is this ongoing trend where you see the incredibly fast pace of change in models, how people are using them, how they’re training them. But when you filter that through the lens of what does that mean for the hardware, it’s almost all noise — 95% is noise. The rate of change for what you need in hardware is much, much slower.

As that applies to agentic AI: what is it doing? It’s still doing decode. It’s still doing prefill and decode. Some things that are different: it has increased the demand. When the agent goes off and thinks for a long time and the user is sitting there waiting, you would like them to wait for 30 seconds instead of five minutes. So the demand for performance has gotten higher, but that’s within expectations. Demand is always going to get higher. That’s a great place to be.

One place where it’s actually a difference is sizing. Sizing exercises are what we do every day. One example: how long does the model sit idle while it’s waiting for a response from an outside system?

In a chatbot context, the model has responded to you, and then you as a human are thinking, maybe you’re going to type another message, maybe you never do, maybe you leave. That’s on the order of 30 seconds or a minute. The context for the model has to be kept in memory somewhere during that time, and you have to size that memory.

That has changed meaningfully in an agentic context where now the model is mostly waiting for tool calls — run a compiler, do a web search, check your email. The times for those are very different. Checking your email can run in seconds rather than waiting for a human to think. So the memories in service of that end up being smaller.

But then there are things like long-running jobs — running a compiler or running a place and route tool, which can take hours. I think that’s actually the biggest place it’s turned up: there is now increasing demand for storage systems for when the KV cache isn’t actively being used but is waiting for a response from an outside agent.

Yeah, interesting. So tell me, how are you guys using agentic AI?

RP: Most of chip design is actually software development in practice. The way you express a chip is you write Verilog, which is a programming language. It’s an unusual programming language because it’s massively parallel, but it is a programming language. Can you write that better with AI?

One of the things we look at: the places where AIs are most effective is when there is a well-defined objective function. Does this compile? Is the area good? Is the power good? How many tests does it pass? We look at our processes and say, can we do development in a way that puts it in that regime, which is really the sweet spot for AI development.

The other thing we do: in addition to Verilog, we use other languages. There are popular ones like Rust and Python, but also some less popular ones — in our case we really like using BlueSpec. It’s a hardware description language that comes from functional programming. We are looking into how we can make sure AI is really good at BlueSpec even though it’s a niche language.

Cool, interesting. I’ve never heard of it. Is that something you think about as a competitive advantage, or just generally wanting to make AI models better at BlueSpec and share this with the world?

RP: There are so few BlueSpec programmers in the world that we just want a higher pool of them, and then it becomes a competitive advantage.

Go to Market

I love that. Okay, since you’re the CEO, I’m going to go back to talking about customers, route to market. On the one hand, it’s kind of nice because maybe there’s only five or six customers that would be great, so any one of these would be a great anchor customer. On the other hand, probably everyone in this space is wanting to talk with them and work with them. What does it look like to say, we’re a startup, trust us, we’re building this thing, it’s going to be awesome? How do you have those conversations to address their concerns, and ultimately, how will they end up buying your first chip or your roadmap of chips?

RP: “Trust us” goes as far as your word goes, right? Not very far. So you need to prove it.

For us, proof means a lot of detail on the artifacts we have. What is the core architecture? What are the very specific details inside the chip? How do we organize the chip — we talked about this splittable systolic array, these are the different compute units inside the chip, how do they connect to each other? What is the instruction set? What is the software SDK?

We give all of this information to customers under NDA. It is a lot, and it is uncomfortable for us to give that information, but it goes a long way towards proving credibility.

Yeah, that makes a lot of sense. As far as the software, what is the level of effort they’ll have to commit to when they say, here’s yet another vendor, we’re excited about everything they told us, we believe them, but there’s probably still some effort to port?

RP: For sure. If you look at the sizes of teams supporting each of these multiple platforms, it’s on the order of 50 to 100 people per platform. Really good people doing kernel development, maybe building compilers, building debugging tools. I think that’s the ballpark of what folks should expect on our platform as well.

We want to help and we’ll do as much as we can to do that work for you rather than you needing to start it all yourself. But ultimately a frontier lab wants to protect its own IP, especially the model architecture. The last mile of kernel development is always going to remain in the frontier lab so they know specifically what they’re doing rather than giving it to us.

The first miles — giving a strong compiler and debugging infrastructure — is something we can actually do for you though.

One or two last questions. What is the biggest skepticism that you hear from people?

RP: One of the things we’re focusing on over the next few years is: how can we as a relatively new startup manufacture in massive volume?

It’s a really exciting opportunity. The projections for data centers over the next few years are in the many gigawatts, tens of gigawatts. I don’t know when we’re going to hit a hundred gigawatts. Nvidia chips sell for about $15 or $20 billion a gigawatt. You might multiply that by 10 or 100. It’s a really large commitment.

The opportunity is really large, but being able to get very quickly to selling such a large volume is also a substantial challenge, and some big parts of that are ahead of us. I think that’s a really exciting thing for us to do over the next year and a half.

Yeah, that’s a good point. It’s not just about building the system, it’s about can you scale it, can you production ramp it, can you get to huge deployments that people are comfortable with, that work, that are reliable. Okay, last question. Give me a hiring plug. You’re 100-some people, it’s very interdisciplinary. Why should people come work with you?

RP: Ultimately you have to believe in the product vision, and I think we just have the best product in the market. It’s designed from first principles for what LLMs really need, keeping in mind years of know-how and techniques of what is the right way to map applications to hardware. That’s the company vision. But the way we operate, it’s a very friendly and high-trust team with a ton of incredibly smart people. I think that’s the day to day of why it’s a really exciting place to be.

Sure, A-plus people enjoy working with A-plus people. Awesome, Reiner, this was great. I learned a lot. Thank you for the time. I’ll be fascinated to check in over time and see how things are going with you.

RP: Yeah, thanks Austin, it was really fun talking.

One might be arriving, but it’s not what anyone expected. In 2017, Benedict Evans predicted augmented reality would be next. Quite reasonable. But three months later, the transformer paper dropped. It took a while for that to change the world, but nine years on, the new device isn’t something even more mobile than a phone. It’s a box on your desk for your AI agents to live on:

Nvidia sells the DGX Spark as a “personal AI supercomputer.” AMD calls it an Agent Computer. Perplexity is shipping Mac Minis as a Personal Computer service.

Is this the beginning of a new S-curve? Or is it another iPad? If the agentic computer takes hold, it’s additive TAM. It carries meaningful ASP because it needs serious memory and compute. Does every knowledge worker’s desk eventually have two computers on it?

But there are headwinds. Recently, Anthropic banned always-on AI agents from using its Claude subscription plans:

That means running OpenClaw just got a lot more expensive. And that’s not the only headwind.

Read more

• Coherent makes its own EMLs, VCSELs, silicon photonics, and finished transceivers. It’s hard to find another public company that touches this many layers of the optical stack.

• Six-inch InP is ramping across four fabs with yields management says exceed 3-inch. The performance comparison against Lumentum is still playing out.

• Five growth vectors stacking: transceivers, OCS, DCI, CPO, thermal. Management says CY2026 is mostly booked and CY2027 is filling fast.

• Breadth vs. depth is the real question. Does a hyperscaler want one partner for everything, or best-in-class at every layer?

We’ve covered Lumentum’s and Broadcom’s AI optical infra businesses so far. Lumentum is a shooting star thanks to its laser performance plus tailwinds of industrywide supply scarcity. Broadcom dominates the datacenter networking silicon (Tomahawk switches, 1.6T DSPs) and is pushing direct-attach copper in scale-up for as long as physics allows, even as it builds CPO technology (lasers included) for optical scale-up.

Time for Coherent.

Coherent’s angle is vertical integration across the photonics value chain. The company designs and manufactures InP-based EMLs and CW lasers, VCSELs, silicon photonics, detectors, and finished transceiver modules in-house.

That puts it in competition with Lumentum, Broadcom, and Sumitomo at the component layer, and with module vendors like InnoLight and Eoptolink at the transceiver level. The stock has run from $45 to $250 in fifteen months and still trades at a lower forward multiple than Lumentum.

In August 2025, Coherent began production on what management calls the world’s first 6-inch indium phosphide production platform in Sherman, Texas and Jarfalla, Sweden:

If yields hold, that should materially improve Coherent’s cost structure and add meaningful InP supply to an industry that is currently constrained. How that affects pricing dynamics across the laser supply chain is one of the key tensions we’ll explore below.

Coherent’s vertical integration means it can supply components, modules, or systems across virtually every optical architecture a hyperscaler might adopt, from pluggable transceivers today to co-packaged optics and optical circuit switches tomorrow:

The bull case is that this flexibility becomes increasingly valuable as datacenter optical needs diversify. The bear case is that hyperscalers prefer to unbundle and multi-source each layer, buying best-in-class lasers from Lumentum, DSPs from Broadcom, and modules from whoever is cheapest.

Let’s walk through the business, the growth vectors, and the tensions that matter.

NFA, DYDD.

Coherent Corp

Coherent’s backstory begins with II-VI Incorporated, founded in 1971 and named after the II-VI compound semiconductor groups on the periodic table. Naming is hard.

II-VI’s original business focused on supplying engineered semiconductor materials and optical substrates that form the foundation of lasers and other photonic devices. Thus, the company operated at the lowest layer of the value chain, upstream of components and with limited exposure to finished products.

There have been many acquisitions along the way:

A few notable ones.

Finisar was acquired in 2019 for ~$3.2B. It was a leading manufacturer of optical transceiver modules at the time and brought significant VCSEL capacity for 3D sensing (for Face ID). The deal extended II-VI from raw materials into finished modules and added scale in hyperscale networking.

Coherent Inc. was acquired in July 2022 for ~$6.6B. Coherent was a storied laser systems company, founded in 1966, that built the first commercial CO2 laser and grew into a global leader in industrial laser systems, especially after acquiring Rofin-Sinar in 2016. It brought complete laser systems for cutting, welding, semiconductor lithography annealing, and display manufacturing, moving II-VI from components into full systems.

So II-VI started as a materials maker and eventually expanded to add photonic products and laser systems capability. They rebranded the parent company II-VI as Coherent Corp. in September 2022.

The result is a company with two distinct lines of business. ~72% of revenue comes from Datacenter & Communications (DC&C), the high-growth segment riding the AI optical buildout. The remaining 28% is Industrial; lasers for semiconductor equipment makers like ASML and Applied Materials, excimer lasers for OLED display fabs, silicon carbide substrates, and specialty materials.

The industrial business is slow growth, but it’s profitable, sticky, and generates nice margins. The slide above suggests Industrial is shrinking QoQ, but the 2025 Analyst day suggests Coherent thinks it can still grow 5-10% CAGR over the next 3-4 years.

Drag on the AI story? Or a diversified base? I tend to be comfortable with the latter — good source of margin dollars.

Coherent CEO Jim Anderson’s mandate is to reshape Coherent from a leveraged, margin-constrained conglomerate into a focused AI photonics platform, and the company is making solid progress. Revenue has grown from $4.73B in FY2024 to a ~$6.7B annualized run rate, while gross margins have expanded by roughly 500 bps to ~39% and EPS has scaled from $1.21 to over $5 on a run-rate basis.

At the same time, leverage has been reduced from over 3x to 1.7x, alongside a series of divestitures and footprint reductions to simplify the business:

Coherent bonus points: Was recently included in the S&P 500 and a $2B equity investment plus multibillion dollar supply agreement with Nvidia.

Vertical Integration

Coherent claims to have the broadest photonics portfolio in the industry.

Let’s walk through it.

First, indium phosphide (InP). Coherent has over 20 years of in-house InP capability spanning epitaxial growth, laser fabrication, modulators, photodiodes, and integrated subsystems.

One important clarification: Coherent does not grow its own raw InP crystals. It purchases InP substrate wafers from external vendors under 3-to-5-year supply agreements, then performs epitaxy and device fabrication in-house. This is the same fundamental supply chain dependency that Lumentum and Broadcom face, though Coherent has locked in multi-year contracts with multiple 6-inch substrate vendors to mitigate it. Worth noting that Coherent does grow its own SiC and GaAs crystals internally; it is specifically InP substrates that are sourced externally:

Gianmarco Conti, Analyst: Gianmarco from Deutsche Bank. You’re expanding indium phosphide capacity, but the raw indium feedstock is roughly 70% sourced from Chinese zinc smelters, which are now subject to export permit requirements with multi-month processing times. I guess my question is, how much visibility do you have on indium supply for the next 12 to 24 months? And are you actively diversifying sourcing away from China? Or do you hold strategic inventory buffer?

James Anderson, CEO: We actually have a very diversified supply chain for indium phosphide substrates. We have -- and I think I’ve shared this in the past, we have over five different substrate suppliers today, and we work with those suppliers, not just on the next -- you mentioned next 12 or 24 months. We don’t work on just next 12 to 24 months. We work on the next like three to five years of capacity that we’re going to need. So we have, in some cases, very long-term agreements in place. And that includes not just the substrates, but all the key inputs that go into that. So we believe that we have very good visibility into substrate supply. And so that capacity expansion that Beck showed is we have commitments from our suppliers to supply the necessary indium phosphide substrates to support that.

The 6-inch InP production platform is an important topic. Management calls it the world’s first, and it is now ramping across four sites: Fremont, California (3-inch legacy); Sherman, Texas; Järfälla, Sweden; and Zürich, Switzerland (newest addition). The economics, per management, are roughly 4x the devices per wafer at about half the cost compared to legacy 3-inch lines, with capacity doubling this year. If yields hold at scale, this could result in competitive cost structure relative to Lumentum (which is migrating from 3-inch to 4-inch) and Broadcom (believed to be on 3-to-4-inch). The timing of those yields is one of the main questions we’ll look at below.

On EML laser chips, Coherent manufactures 100G EMLs for 400G and 800G transceivers, 200G EMLs for 800G and 1.6T, and demonstrated a 400G Differential EML for 3.2T and 6.4T at OFC 2026.

Coherent currently uses a mix of internal and external laser sourcing. Anderson confirmed at the Morgan Stanley conference that Lumentum is both a customer and a supplier, suggesting that Coherent’s internal EML capability has not fully displaced external supply across all performance tiers. The 6-inch cost advantage may be closing the gap, but the performance comparison against Lumentum’s epitaxy is still playing out.

For CW lasers, which power silicon photonics transceivers and are critical for co-packaged optics, Coherent is in full production and ramping on 6-inch InP in Sherman. At OFC 2026, it showed a 400mW high-power version for CPO applications.

CW lasers are one of the key product families covered by the Nvidia supply agreement. On the OFC status update call, Beck Mason (EVP Semiconductor Devices) set out to reassure everyone that CW laser yields on 6-inch wafers are looking promising:

BM: We’re currently running three main categories of devices on 6-inch indium phosphide, EMLs, high-power CW lasers and high-speed photodetectors. And
all three of those categories, we’re seeing higher yield and better throughput efficiency on our 6-inch lines than we’ve been able to achieve even on our very mature 3-inch production lines.

On VCSELs, Coherent manufactures GaAs-based vertical-cavity surface-emitting lasers. These serve Apple under a new multiyear 3D sensing agreement, and Coherent plans to launch a VCSEL-based 1.6T transceiver in the second half of calendar 2026.

VCSELs are a lower-power alternative to InP-based solutions, but with shorter reach. CTO Julie Eng explained the tradeoff at the OFC briefing:

“The VCSEL actually is an interesting potential for silicon photonics because the power is very, very low… it’s basically between 4x and 5x lower power than the silicon photonics solution. But it doesn’t go as far. It’s shorter reach.”

That means VCSELs could be used for in-rack and near-rack scale-up, where power and density are prioritized over distance, typically under 100 meters on multimode fiber.

Management says Coherent has ample gallium arsenide capacity, which is relevant because GaAs circumvents the industrywide InP bottleneck. Anderson expects VCSELs and InP-based approaches to coexist rather than compete:

“In the VCSEL-based solution, you usually have the laser in it. And so there’s some pluses and minuses of that. But I do think that these will coexist in CPO/NPO just as they have in pluggable transceivers.”

Coherent also has its own silicon photonics PIC platform and demonstrated a 400G pure silicon PN-junction Mach-Zehnder Modulator at OFC 2026. This is a pathway to 3.2T transceivers via silicon photonics rather than InP, giving Coherent optionality across both technology approaches.

At the transceiver module level, Coherent ships full OSFP modules at 800G and 1.6T. At OFC, it showed 1.6T transceivers built with three different DSP solutions from three different industry leaders. That is notable because it positions Coherent as technology-agnostic at the DSP layer, in contrast to Broadcom, which makes its own DSPs and can offer a vertically integrated laser-plus-DSP package. Coherent is effectively saying it will work with any DSP partner, giving hyperscalers flexibility to choose.

Beyond transceivers, Coherent makes optical circuit switches (OCS) using digital liquid crystal technology, which is non-mechanical and has no moving parts.

Anderson said at Morgan Stanley that Coherent is engaged with over 10 customers and that “multiple customers have already deployed in real data center applications”. Revenue shipments began in Q4 FY2025. This is a different technology from Lumentum’s MEMS-based OCS, and the two approaches are competing for the same emerging market.

Finally, on the industrial side, Coherent has some interesting datacenter-adjacent materials. Thermadite is a proprietary material with what management describes as exceptional heat-transfer characteristics, which is being evaluated by large customers as a replacement for copper heat sinks in data centers. Coherent also has a thermoelectric material that can convert waste heat back into electricity.

Both are early-stage, but Anderson highlighted them at Morgan Stanley as longer-term growth opportunities that bridge industrial materials expertise with datacenter demand.

The bull case for all of this vertical integration is greater internal control over cost, supply, and iteration speed. Customers value having a single partner that can build EML-based, VCSEL-based, or silicon-photonics-based solutions, depending on the application.

Plus supply chain resilience across 60-plus manufacturing sites in 14 countries, more than 20 of which are in the United States (Q3 FY25 transcript).

JA: But the other -- the second point I would make in terms of supply chain resiliency is around vertical integration. And this applies to not just our data center business, but also to our industrial business, for instance, our laser business is if you look at a lot of the very key technology in feeds for whether it’s a data center transceiver or an industrial laser, we make ourselves, manufacture ourselves a lot of the very key components that go into our transceivers or laser systems or other products. And so that’s an important part of our supply chain resiliency and flexibility. So to the extent that there are changes in the landscape, the tariff landscape and to the extent we need to adapt manufacturing, move manufacturing to different places for the benefit of our customers, we certainly feel like we’ve got a very good, resilient, adaptable supply chain to leverage.

The bear case is essentially “doing everything means doing nothing best”. Lumentum’s epitaxy appears to be ahead (Coherent still sources some lasers externally). Broadcom’s system integration is deeper (laser plus DSP plus switch on-package). The risk is that Coherent ends up as a jack of all trades competing against specialists at every layer.

Growth Vectors

Coherent frames its growth story in two layers. The existing engines, pluggable transceivers (800G through 3.2T), DCI coherent transceivers, transport/transmission equipment, and optical components, collectively address a $50 billion-plus SAM per management’s estimates. These are shipping now and growing.

On top of that base, Coherent identifies four new growth engines that add over $20 billion in incremental SAM by 2030:

The stacking of these new engines on top of an already-growing base is why management says CY2026 is mostly booked, CY2027 is “filling very, very quickly,” and FY2027 revenue growth will exceed FY2026. Each new engine is at a different stage of maturity, so inflection points are spread over the next 18 months rather than concentrated in a single quarter.

Open Questions

So… Coherent has a very broad photonics stack with promising growth vectors at various stages of inflection. The stock has run from $45 to $250. The sell-side is overwhelmingly bullish.

Yet Coherent still sources some lasers externally, including from Lumentum. Broadcom has the full-stack CPO lead, even if its CEO says CPO is “not anytime soon.” Chinese module makers are winning volume at 800G. And the BIS Huawei investigation is still unresolved.

Which of these growth vectors holds up under scrutiny? Where is management credible and where are they hand-waving? How does Coherent stack up head-to-head against Lumentum and Broadcom across each product category?

I went through all of this against Q2 FY2026 earnings, the Morgan Stanley TMT Conference (March 3), OFC 2026 announcements, and sell-side research, then put together a three-way comparison with Lumentum and Broadcom.

Here’s what’s behind the paywall:

• COHR vs. LITE vs. AVGO: Head-to-head across every product category

• Six things to watch, each with a bull case, bear case, and what to look for next: the 6-inch InP bet, OCS liquid crystal vs. MEMS, CPO positioning, the margin path to 42%, BIS risk, and valuation

• How the latest quarter stacks up against each of those

• What the Street is saying and where analysts disagree

• Catalysts for the rest of CY2026 and into CY2027

and more!

Read more

It’s amazing. They just describe what they want, and Claude code asks some clarifying questions, and then runs off and builds it.

If you haven’t tried having AI build software for you yet... you gotta try it. Claude Code or Claude Cowork, Cursor, OpenAI’s Codex, Perplexity Computer, whatever. It’s easy to get started with something simple on your laptop.

We often spin up several simultaneous agents and build many things at once. Dad, can we create a black hole simulator while we wait for this game to build? Sure, buddy!

Now, if I spin up a bunch of agents and they are all doing heavy token generation on my behalf, are the deterministic tasks like API fetching and code execution running on server CPUs or just locally on my machine? Great question.

I personally use the command line version of Claude to spin up agents, so let’s see if we can figure out what it does.

This CLI is open source, so we can read the code to figure out how it works. lol jk, we won’t read the code, we’ll just have Claude Code read its own source code for us. It turns out the CLI runs almost everything locally, which, in my case, is on an Apple M4 chip. So when Claude Code on my MacBook writes and executes code to generate the ASCII art in the image above, the non-GenAI bits run locally on my MacBook.

Now, that doesn’t mean EVERYTHING agentic will run locally. As cool as Claude Code and OpenClaw are, they aren’t the only way to run agentic AI. A lot of agentic AI will be kicked off via phone and web apps. The user will taps a button or type a sentence in a normal app, and behind the scenes an agent spins up in the cloud, runs tools, makes API calls, executes code, fetches data — all CPU work on server racks. The apps themselves can use the Claude API, which offers the same primitives the CLI offers locally — code execution, web search, programmatic tool calling, and so on. But they run in the cloud and can be scaled.

Cloud-based agents will contribute a ton of agentic AI inference in the coming years, and will continue to be a key driver of the majority of Anthropic’s revenue.

In GPU inference servers, the head node's CPU is there to keep GPUs fully utilized. Agentic workloads introduce orchestration and tool execution overhead that erodes CPU headroom and constrains GPU utilization. The fix is to extend beyond the head node into a proximate CPU rack, positioned close to the GPU racks to minimize latency and maintain throughput.

Ok then, if we need racks of AI CPUs for agentic AI... which racks? Like AMD Epyc or Intel Xeon racks? Or Graviton / Cobalt / Axion racks? Or what?

Well, it probably makes sense to start with the same kind of CPUs that are already running the Gen AI related CPU workloads today, right? The existing head node CPUs already handle orchestration, multi-modal fan-out, data pre- and post-processing, and so on.

Maybe just buy more of those? Like my dad always said, if it ain’t broke, don’t fix it.

And interesting, those are often Arm CPUs these days.

As a reminder, Hopper systems typically paired GPUs with Intel Xeon CPUs, most often Sapphire Rapids or Emerald Rapids. Nvidia had little incentive to drive AMD CPU share given Instinct competition, though OEMs still offered server configurations with AMD EPYC head nodes.

But, Blackwell-era GPUs were mostly paired with Nvidia’s Grace CPU (Arm Neoverse V2). Which means the most important workload of our lifetime to date, LLM inference, was initially deployed on x86 but quickly moved to Arm.

Now I’m curious. What about XPUs?

Claude inference on Trainium 2 uses x86 head nodes, purportedly Intel Sapphire Rapids. At roughly one CPU socket per eight Trn2 chips, the one-million-accelerator Trn2 deployment implies about 125K x86 CPUs. Nice. But wait! According to SemiAnalysis, Trainium3 is moving to Graviton4 (Arm Neoverse V2).

Hmm. Another instance of inference clusters swapping CPU sockets to Arm.

And maybe TPUs are heading that direction too? Per SA, “in the future, Google will design Axion CPUs for use as head nodes in their TPU clusters powering Gemini”.

It sure seems like a lot of the CPU workloads that support LLM inference are already running on Arm, or heading in that direction.

And thus adding nearby racks of Arm CPUs to create headroom for agentic AI seems very sensible.

OK then, where does one buy such racks of Arm CPUs anyway?

Option A: Build Your Own

The cloud providers (AWS, Google, Microsoft) are already building custom Arm CPUs (Graviton, Axion, Cobalt). But these existing Arm CPUs were designed for traditional cloud server workloads. Ya know, APIs, databases, web servers, and all that jazz used to build SaaS empires. Agentic AI workloads have different requirements than cloud-native ones. They want much more memory bandwidth per core. Low tail latency. Etc.

So CSPs could add new custom CPU SKUs to the roadmap, tuned appropriately. Same team can reuse a lot of the same IP, it wouldn’t be that bad. But the agentic AI CPU demand/supply imbalance is hitting RIGHT NOW. No one has time to wait for an agentic-flavored Graviton to be designed, validated, and taped out.

In the meantime, sure, CSPs could just use their existing custom CPUs. They might not be perfectly optimized for the workload, but given we’re in agentic takeoff, all CPUs on deck. Perfect is the enemy of good. But, of course, at massive scale, every inefficiency adds up, so there’s still going to be a need for the right CPUs designed for the agentic AI workload in the fullness of time.

Btw, this “good enough” argument could be applied to Arm-based server SKUs from Qualcomm and Ampere, which were not designed with Agentic AI in mind. But silicon is in short supply... if you’ve got ‘em, sell ‘em.

One final observation: future agentic CPUs from hyperscalers will likely adopt Arm’s Neoverse V3 CSS, which carries higher royalty rates than V2. If so, agentic AI drives both unit volume and Arm’s average royalty per chip.

Option B: Buy Nvidia Vera

If you want to stay on Arm and you need racks now, you’re in luck — Nvidia is selling them. The Vera CPU, which Nvidia calls “the world’s first processor purpose-built for the age of agentic AI,” is built on 88 custom Olympus cores (Arm Neoverse V2). The liquid-cooled Vera rack has “256 liquid-cooled Vera CPUs to sustain more than 22,500 concurrent CPU environments, each running independently at full performance”.

Nvidia’s gonna sell a lot of these Vera racks for agentic AI.

Profit pools don’t stay uncontested, though. If there’s money to be made, competitors will appear. And who might that be? The CSPs are the obvious candidates, but their model centers on deploying infrastructure for internal use and rental, not selling merchant silicon. Well, maybe not GCP if the rumors about selling TPUs are true.

OK. I’ve buried the lede... you’ve surely put this together by now…

Who else can sell Arm-based agentic AI CPU racks?

Option C: Buy from Arm

Arm can!

They’ve already done most of the heavy lifting with CSS, the Compute Subsystem that stitches together CPU cores and system IP, and handles the backend physical design too. At that point, why not just take it all the way across the finish line and build the full chip?

And that is exactly what happened.

This week, Arm announced the Arm AGI CPU, the first merchant silicon offering from Arm.

For the first time in its 35+ year history, Arm is a merchant silicon CPU vendor.

Some details from the announcement:

For the workloads driving CPU demand in AI like agentic orchestration (web searches, API calls, agent fan-out), RL training sandboxes (code compilation, verification, tool use), and data processing, you need lots of performant cores.

That’s exactly what the AGI CPU claims to deliver. In a liquid-cooled 200kW rack it has over 45,000 cores:

Note that these CPUs to GPU racks over the network, not via direct chip-to-chip links. Nvidia’s NVLink C2C only applies in the integrated Vera Rubin NVL72, where CPUs and GPUs live in the same rack. I had heard some confusion here so wanted to clarify.

So… Arm is now selling chips. If you need agentic Arm racks you can build, buy from Nvidia, or buy from Arm.

But there are still many questions to think through, including:

• Vera vs. AGI. Different strengths, different price points.

• Where are the x86 agentic AI CPU racks?

• Why are Meta and OpenAI launch customers for both Vera and AGI CPU?

I’ll hit on these and more for paid subscribers.

Read more

• Disaggregation is now official Nvidia doctrine. Not just a startup pitch.

• Agentic AI is the killer app driving all of this. Vera CPU racks because CPUs were bottlenecking GPUs. LPUs for ultra-low latency coding agents.

• Nvidia’s strategy hasn’t changed. The whole system must beat any mix-and-match alternative. Does InferenceX sufficiently measure Agentic AI Factory performance?

• The unbundling of inference workloads is the unbundling of the datacenter. If Groq can slot in, so can Cerebras, Etched, MatX, AMD. Conceptually, anyway…

Without further ado, the most important slide from GTC 2026, demonstrating that Nvidia GPUs plus AI ASICs lead to a better and expanded Pareto frontier:

Want to unlock super low latency (or very high tokens/sec) for insanely fast Claude coding? ABC – Always Be Claudin’, amiright?

You got it! GPU + LPU. See the far right side of the chart.

Disaggregation unlocks “right silicon for the workload”. And the right silicon isn’t always GPUs; a bunch of Groq’s chips can tally up much more bandwidth for memory-bandwidth bound workloads:

Yet Rubin has way more FLOPs for compute-bound workloads. So put the two together, and you can outperform just GPUs. Of course it will cost you $$$, but for certain customers and points on the Pareto curve it can make economic sense.

To be fair, disagg has been out of the bag for over a year now, but the full ramifications are becoming clear. In the past year we learned of Dynamo and prefill/decode disaggregation. And we even saw Nvidia unveil the Rubin CPX as a SKU specifically for prefill. But that was still just splitting the workload amongst GPUs. But now we see further disaggregation, and AI ASICs have entered the picture:

So GPUs aren’t enough!

The narrative has officially moved past GPU for everything, as said on Nvidia’s blog:

Hardware tuned for peak throughput under large batches isn’t ideal for the most latency-sensitive execution paths, while hardware optimized for low-latency execution is less efficient for the most compute-intensive phases.

Multi-Vendor Inference

It’s not a stretch to call this a multi-vendor inference system. Or a heterogeneous system if you prefer. Sure, it’s the Groq 3 LPU with an Nvidia label on it, but conceptually its an AI ASIC startup rack with an Nvidia GPU rack.

Hence, the corollary: If Groq racks can be slotted in, so can Cerebras, MatX, Etched, AMD, Intel, and so on.

The unbundling of the workload is the unbundling the AI inference datacenter.

That said, although the narrative has changed from “GPUs for everything” to “right silicon for the workload”, I’d argue that Nvidia’s strategy hasn’t changed one bit.

I’d sum up Nvidia’s strategy as ensuring that the whole inference system is greater than the sum of the parts. Said another way, Nvidia is betting that full Nvidia inference AI clusters will outperform competitive clusters piecemealed together from different vendors.

And Nvidia is betting that outperformance is unlocked through vertical integration. Think Apple, Tesla.

Take Nvidia GPUs vs AMD Instinct GPUs as an example. When AMD first came on the scene, even though MI300X was a good inference chip, it didn’t have performant enough software to extract all the value from the chip. So Nvidia’s whole system (H100s + CUDA) was better than AMD (MI300 + ROCm).

Grace Blackwell NVL72 took it up a notch. The scale-up NVLink switch enabled Nvidia to have 72 GPUs acting as one big GPU. Thus, Nvidia’s whole system (accelerator + scale-up networking + software) outperformed AMD MI350, which didn’t have competitive scale-up networking technology. AMD won’t have a scale-up domain size of 72 until MI450 with Helios, and even then, it’s UALoE.

And Nvidia is already showing they’ll be running further ahead with a portfolio of inference-centric offerings like the Vera CPU racks for agentic AI and the STX Storage racks:

Just look at that row of CPU + GPU + LPU + switches + storage. Whatever agentic AI factory you can put together, can it outperform this fully integrated system? And was the opportunity cost of sourcing it all, hooking it up, validating it, and making sure the software works across it all worth it?

Again, even though disagg is out of the bag, Nvidia believes it can outperform an “open” modular system with components from different vendors, e.g. CPU and GPU from AMD, AI ASIC from a startup, scale-up switch from Celestica / HPE / Astera Labs / etc.

Agentic AI is GenAI’s Killer App

Given that Nvidia is competing on “full AI factory performance”, it’s no wonder Nvidia is shipping Vera CPU racks.

No, Nvidia isn’t trying to take down Intel or AMD.

It’s much simpler. What was the North Star? The whole is greater than the sum of the parts.

And the “killer app” for GenAI has appeared; it’s agentic AI. Claude Code. OpenClaw. The world will never be the same.

And Nvidia wants to make the best Agentic AI Factory.

Agentic AI involves a lot of tool calling, data processing, and so on. The head node CPUs can’t handle it all. So Nvidia added two racks of Vera CPUs in the same row as all the GPUs, optimized for agentic AI. From Ben Thompson’s interview with Jensen after the keynote:

JH: ... you want the fastest single-threaded computer you can possibly get… the most important thing is single-threaded performance and the I/O has to be really great… if the CPU gets throttled, then we’re holding back a whole bunch of GPUs.

Clearly, Jensen is thinking about making the whole Agentic AI Factory greater than the sum of the parts. CPU was a bottleneck, so CPU racks were included.

And we want agents to be fast. Generating code is awesome. Generating hours worth of code in minutes is even more awesome. High interactivity is necessary for agentic AI, hence LPUs.

Implications

So disagg is out of the bag, and now many implications and questions are piling up:

• What does “right silicon for the workload” mean for the CPU market when agentic AI is the killer app?

• If the datacenter can be disaggregated, the door is open for other SKUs… but can they actually slot in? Will Dynamo allow it? Will an open alternative emerge?

• Where do XPUs fit?

• What about other Pareto frontiers? Why only the largest models at 1000+ token throughput? What about medium or small models at 1000+ tokens?

I have strong views on all of these, and more. Let’s go deeper.

Read more

Three things from Jensen’s GTC 2026 keynote were notable:

1. Scale-up is going copper and optical

2. Nvidia is selling lots of standalone CPUs, calling it a multi-billion-dollar business

3. The Groq 3 LPX chip is real, fabbed by Samsung and shipping Q3

We cover the full Vera Rubin data center system, tiered inference economics, CPO in production, and take a detour into whether the vibe-coding era actually kills SaaS (we think not).

You can find it on podcast players too, e.g. on Spotify and Apple Podcasts.

This interview is lightly edited for clarity. Transcript is available for paid subscribers who prefer to read vs watch.

Read more

• Four MTIA chips in two years, all inference optimized

• ROIC story is straightforward

• Further validates the industry is past “GPUs for everything” value prop

• Implications for Broadcom, Nvidia, AMD, HBM suppliers, TSMC, Arista, and inference startups

The narrative around AI silicon and GPUs has changed. Six months ago, the default assumption was that GPUs are the answer for everything. But as I wrote about last October in Right-Sized AI Infrastructure, as AI workloads mature and become better understood, the economics favor purpose-built hardware over general-purpose GPUs. I’ve been long saying this, see GPU bloat stifles AI from Feb 2024. But that was long before reasoning models and agentic AI, so the industry was still stuck on “is GenAI even useful?” and “what if LLM architectures change?

The demand for transformer-based LLM inference is exploding. As Ben Thompson stated this week, we’re in a new era and demand is increasing exponentially :

This [functional agents] increases the market in two directions: first, humans can run multiple agents, and secondly, agents can leverage reasoning models multiple times to accomplish a task. This isn’t just an exponential increase in the addressable market for tokens, it’s two exponential increases squared.

In this era of intense demand (NEED MOAR TOKENS FOR MY AGENTS!), developers care about latency (MAKE AGENTS FASTER!) and throughput (DANG OPUS IS EXPENSIVE!)

The market is responding rationally. In just the past several months we’ve seen NVIDIA+Groq and OpenAI+Cerebras. And those were pre-ChatGPT chips, aka they made design decisions not optimized for transformer LLM inference.

We’ve also seen maturing XPUs and roadmaps, including Trainium 3/4/5, Microsoft Maia 200/300, and now Meta MTIA 300/400/450/500.

Meta’s annoucement was this detailed technical blog articulating four iterations of MTIA that have shipped or are planned within roughly two years.

And they were very transparent with specs, timelines, etc. This is really cool. Of course, a company like AWS with Trainium is going to share XPU specs because it’s cloud rental business might rent out the XPUs too. But Meta’s only using them internally; no cloud biz. So Meta doesn’t HAVE to share as much info. But they shared anyway. This is good for investor transparency, for current employees and potential hires, and for us nerds.

Here are said specs from Meta:

And the roadmap makes sense.

As I wrote in Meta’s ROIC Strategy: GEM Now, LLMs Later, Meta’s business has always been powered by ML/AI infrastructure. Facebook launched the algorithmic feed in 2011, and ever since then ML recommendation systems have driven what 3.5B daily users see and which ads they’re shown. To the tune of $150B+ annual advertising revenue. Hence, the custom inference silicon roadmap is driven by workload, starting with clearest ROI (recommendation systems) and moving toward GenAI.

By the way, Meta and many others are showing that multi-vendor hardware portfolios are no problem. We need compute. We can deal with the software implications of building across stacks.

From the Meta Q4 earnings call:

“We extended our Andromeda ads retrieval engine, so it can now run on NVIDIA, AMD, and MTIA. This, along with model innovations, enabled us to nearly triple Andromeda’s compute efficiency.”

Done right, custom silicon for recommendations is margin expansion on the core business. Custom silicon for GenAI inference is cost reduction on the fastest-growing workloads. A penny saved is a penny earned. Same story AWS tells with Trainium 3+, same story as Microsoft and Maia 200+.

And again, it’s inference-first. From Meta’s blog:

“Mainstream GPUs are typically built for the most demanding workload — large-scale GenAI pre-training — and then applied, often less cost-effectively, to other workloads such as GenAI inference. We take a different approach: MTIA 450 and 500 are optimized first for GenAI inference, and can then be used to support other workloads as needed”

IMO this pushes back on concerns around The Information’s recent article: Meta’s Internal Chip Design Efforts Hit Roadblocks

Meta last week scrapped the most advanced chip it was developing for training AI models, after struggling with the chip’s design.

This generated some hand-wringing for Broadcom and Hock had to address it on the earnings call. But in context, it’s a bit of a nothing burger, right? Given where we are today, internal custom silicon efforts like Meta’s ought to prioritize inference. Use Nvidia GPUs for training. AMD Helios racks too. IMO there’s no urgent need to build custom silicon for training.

The ~6-month cadence is possible because… chiplets!

Because each chiplet can be upgraded separately, we can implement improvements in months rather than years. Moreover, different chiplets can be manufactured at different process nodes that are most cost-effective while meeting performance and power requirements.

Disclosure, big chiplet fan here.

AMD has long punched above it’s weight thanks to chiplets. And here’s Meta doing the same.

And as you would expect, all the systems use the same chassis, rack, and network infra. Yay OCP.

Oh, and so much related news.

Remember Meta’s Sep 2025 acquisition of Rivos, a “CUDA compatible RISC-V AI startup”? From Reuters

“Our custom silicon work is progressing quickly and this will further accelerate our efforts,” a Meta spokesperson said when contacted by Reuters.

Rivos was already collaborating with Meta on MTIA before the acquisition, according to reporting from The Next Platform:

Rivos, which was founded in May 2021, was pretty secretive about what it was up to and it had a partnership with Meta Platforms where it apparently helped in the design of the MTIA 1i and MTIA 2i compute engines (using the more recent and descriptive way of talking about them). The exact nature of this collaboration was unknown. Separate from this, Rivos was working on its own RISC-V CPU and GPU designs.

And four chips in two years requires a large silicon team. Which apparently Rivos had:

With the backing of Walden International with the help of Dell Capital Ventures and Matrix Capital Management, Rivos started off with more than a hundred employees on day one, and Tan was named chairman of the board. This has, in part, given Rivos access to advanced EDA tools and to foundry expertise and capacity at Taiwan Semiconductor Manufacturing Co. Hiring nearly 50 engineers from Apple in 2023 landed it in a lawsuit with Apple, and Tan negotiated a settlement.

Oh interesting. Also, Lip-Bu Tan is EXPERIENCED.

Software seems not to be a problem. Meta’s MTIA blog emphasizes PyTorch/Triton/vLLM compatibility at the ML framework layer. And Rivos also claimed to have built a “CUDA-compatible software stack”. And Meta is good at software.

Sure seems like MTIA will be successful.

Behind the paywall:

• HBM vs SRAM: two planes of inference competition, and where MTIA sits

• Supply chain impact: Broadcom, NVIDIA, AMD, HBM suppliers, TSMC, Arista, inference chip startups

Read more

If you haven’t read those, they are easy reads. I recommend skimming through them first.

Each step trades flexibility for efficiency. And the root source of the inefficiency is that long, noisy copper trace between the switch and the optics:

At modern lane rates (e.g 50G, 100G per lane), electrical signals pick up a ton of noise and distortion crossing the copper trace between the switch and the transceiver. Pluggable optics handle this with a DSP that overcomes the noise during transmit and receive. And LRO and LPO save power by relocating that DSP into the switch:

But the system becomes less modular and harder to mix-and-match.But why deal with that copper trace at all? What happens when you just... put the optics right next to the silicon?

NPO

Near package optics (NPO) brings the optics module on the same substrate or very close to the switch package, but not inside it:

It’s close enough to reduce most copper impairments. This is a pragmatic middle ground, but the major players are largely leapfrogging it and going straight to CPO.

Might as well just reduce the copper distance to nearly zero right?

CPO

Finally! Co-packaged optics (CPO).

The optics move onto (or into) the switch package itself:

The electrical path between the switch die and the optical engine is now very short (millimeters or less). Since there’s no long copper trace, we needn’t have a DSP to compensate for it! Less silicon content, and less power.

There’s also much less SerDes power overhead as you only need extra short-reach (XSR) SerDes, the simplest, lowest-power tier:

The simplest way to think about CPO is that the transceiver disappears and the optical engine moves onto the switch package itself.

How It Works

SemiAnalysis has an in-depth CPO article with a helpful diagram:

The optical engine is the core of CPO; it converts between the optical and electrical domains. Since the OE is on-package, fiber runs directly to the package edge. And now the electrical path to the switch is so short that signals stay clean without heavy conditioning. The switch ASIC’s SerDes handles what little remains.

Another helpful diagram from Nvidia’s blog. The red circles highlight noisy copper channels. Notice how CPO eliminates most of them:

CPO cuts down on the overall power consumption, too. Per this example from Nvidia, power consumption cuts down from 30W for pluggables to 9W for CPO:

As I always say, in this power-constrained era, every watt saved is a watt that can be used for computation.

At this point, you should watch this Nvidia CPO video again, as it will make a lot of sense now:

Lasers and Silicon Photonics

Oh yeah, an important call out. Look at the Nvidia diagrams above again. The pluggable transceiver uses externally modulated lasers (EMLs) for 1.6Tb. These are discrete InP lasers + modulators.

CPO uses lasers differently. Instead of modulating the laser itself, it uses a simple continuous wave (CW) laser (just a constant beam of light) and performs the modulation on a silicon photonics chip on the switch substrate:

Silicon photonics is an optical circuit built in silicon using CMOS-compatible fabrication of waveguides, modulators, photodetectors:

What about serviceability though? Don’t lasers fail?

Yes, and this was one of the earliest objections to CPO. In pluggable optics, a failed laser means swapping the transceiver module which is easy to access. But if the laser is near the switch… that’s a lot harder right?

Well, don’t put the laser near the switch! The CW laser is external. So if it fails, you can still easily replace the laser source and not the switch.

And as Nvidia shows here, the silicon photonic engines themselves are designed as detachable sub-assemblies. Not as easy as swapping a front-panel pluggable, but far better than scrapping the switch:

Thermal concerns are similar. Lasers are temperature-sensitive and switch ASICs run hot; pluggable EMLs combine a laser and modulator in a single InP device running at high speed, and they run hot and are among the more failure-prone components.

But with CPO, the laser is just a simple CW source and the high-speed modulation moves to silicon photonics. If the laser sits off-package, you’ve removed the most temperature-sensitive component from the equation!

Moving optics closer to the switch looked like a reliability problem, but it may end up improving reliability instead. In fact, SemiAnalysis shared this nice slide from Meta that suggests Meta had fewer failures using CPO than with pluggables:

Trade-offs

Everything in engineering is about trade-offs, and CPO is no different.

Manufacturing is non-trivial. The silicon photonics engines sit very close to the switch ASIC, which means integrating optical and electrical components with different materials, process flows, and reliability characteristics. That complicates packaging, thermal design, and testing compared with traditional pluggable optics.

And CPO also tightens the coupling between the optics and the switch platform. With pluggables, operators can mix transceiver vendors or change optical reaches independently of the switch. In a CPO system, the optical engines are designed and qualified as part of the switch platform, which reduces that flexibility.

But that’s a manageable trade for hyperscalers, who co-design systems with their silicon partners (Broadcom, Marvell), control board layout and qualification, and deploy into environments they fully manage.

And Nvidia is bringing CPO to its merchant switch lineup (Spectrum-X Photonics, Quantum-X Photonics), which could eventually bring CPO within reach of non-hyperscalers who don’t have that kind of vertical integration. Well, they do have that kind of vertical integration… via Nvidia.

Pluggables Are Not Dead

Pluggable transceivers are EML-based InP modules with DSPs. As CPO scales, we need more silicon photonics engines and CW laser sources, and fewer DSP chips for pluggables.

People like to jump to conclusions. Remember: Copper is dead! Long live optical!

But as Vik and I discussed recently, I’d sum it up as: The answer is both. The question is when.

Yes, the direction of travel is toward tighter integration of optics and silicon. But the debate is how fast. After all, Hock Tan claimed 400G SerDes on the last Broadcom earnings call, which would extend the pluggable runway. But Broadcom is also shipping early-access CPO switches. Nvidia is shipping CPO in 2026. Marvell acquired Celestial AI to get in the game.

Today, pluggable optics concentrate value in the transceiver module. InP EML lasers, DSPs, driver and TIA chips, optical packaging. Companies like Lumentum, Coherent, Fabrinet, and DSP suppliers like Marvell and Broadcom sit in that value chain.

But with CPO, there’s an unbundling in the value chain. The optical functions move onto the switch package as silicon photonics engines, the laser becomes a separate CW source, and the DSP largely goes away.

Hence, value shifts toward silicon photonics, CW laser production, and advanced packaging, and away from standalone transceiver DSPs and pluggable module assembly.

But again, these shifts are happening over time. As I said in the video above, the transition between technologies isn’t just a binary thing on a particular date, but more of an adoption curve. Even a single hyperscaler can be deploying different technologies at different places at roughly the same time. So pluggables are still a great business.

I’ve already started pulling on these value chain threads. If you want to understand the laser side in depth, check out Lumentum and the Laser Bottleneck and Broadcom Makes Lasers, which are both directly connected to the CW laser and silicon photonics shifts we covered here. And more to come! Coherent, Applied Optoelectronics, Astera, Poet, etc. Getting lots of requests from readers :)

If you found this useful, subscribe. And if you want the deeper company-level analysis as CPO scales, consider going paid.

Subscribe now

Rivian designing its own chip was a genuine surprise. The company built its own zonal ECU architecture, so electronics has been a core competency, but custom silicon is a different level of ambition.

We start the interview by building up an understanding of the autonomy workload. What does it actually mean to do multimodal inference at the edge, when you have 11 cameras, radar, and lidar all streaming into a chip that has to make driving decisions in a 100-millisecond loop?

From there we get into RAP1 itself, including the custom neural network engine, the safety architecture, and a chip-to-chip interconnect called RivLink that hints at ambitions beyond autonomous vehicles. We discuss build vs. buy decisions, like why Rivian designed its own neural network engine and interconnect but licensed Arm cores. And a very interesting discussion about how silicon economics drove the 800 TOPS compute target.

Finally, we get to the heart of the matter. Why custom silicon over merchant? How does vertical integration unlock optimizations you simply can’t get off the shelf?

We close with the team and timeline, about 2.5 years from project go to silicon in hand. This was fun and educational. Enjoy!

This interview is lightly edited for clarity. Transcript is available for paid subscribers who prefer to read vs watch.

An Interview with Rivian’s Mukund Chavan About RAP1

Read more

Most people hear Broadcom and think Tomahawk switches, XPUs, VMware, Hock Tan. Not lasers. But Broadcom has actually been the primary 200G EML supplier to date, given its position in Nvidia’s 1.6T transceivers. It turns out there’s an interesting history and a great hidden business here.

Heck, Broadcom owns the former Bell Labs InP fab in Breinigsville, Pennsylvania. And they are hiring!

So Broadcom makes its own EMLs, VCSELs, and CW lasers, and pairs them with the industry’s leading DSPs and switch ASICs.

I’m genuinely impressed.

That’s arguably the most complete photonics stack in the industry. I had no idea. Broadcom, you should tell this story!

Broadcom’s Laser Business

Digging in, the laser business looks great:

• Total EML capacity is projected to grow from over 40 million units in 2025 to 50 million in 2026. This is across all speeds. 100G for 800G transceivers, 200G for 1.6T, and other applications

• CW laser capacity is expanding from mid-teens millions in 2025 to roughly 30 million in 2026

• Broadcom was first to ship high-volume 100G-per-lane EMLs and VCSEL technology

• Management claims leadership in 200G EMLs for 1.6T transceivers

• The networking segment, which includes these components, carries Broadcom’s highest margins outside of software

Note that Broadcom doesn’t break out its photonics revenue separately. They should! Lumentum trades at ~60x forward earnings as a pure-play laser company. Coherent trades at ~44x on vertical integration. Broadcom’s laser business... not sure how the Street values it. It’s buried inside a networking segment within a semiconductor solutions business unit. inside a $1.5T market cap conglomerate. Investors can’t price what they can’t see. I said this about Nvidia autonomy business recently too, and previously about the Nvidia networking business.

On the Q4 FY2025 earnings call, CEO Hock Tan said:

“The demand for our latest 1.6 terabit per second DSPs that enables optical interconnects for scale-out, particularly. It’s just very, very strong. And by extension, demand for the optical components like lasers, PIN diodes, just going nuts.”

Going nuts.

With EML capacity ramping 25% year-over-year and demand going nuts, photonics breakout would give the Street something to actually model in a sum-of-the-parts fashion. And Jefferies notes that Broadcom’s networking segment carries the highest margins outside software.

InP lasers are high-ASP, high-margin components, and the 100G-to-200G mix shift roughly doubles the ASP per laser. Vik and I over at Semi Doped have been very interested in exploring components companies with stacking s-curves, where each generation of new product has higher ASPs.

How does Broadcom’s laser position compare to Lumentum and Coherent?

Today I dug into the 1.6T competitive dynamics, CPO timeline, the substrate supply risk (and a good update this week from Lumentum derisking this substrate supply risk), and some bull/bear tensions.

Plus a three-way comparison between AVGO, LITE, and COHR during this optical supercycle.

Went down the rabbit hole over the past few days, thanks for coming along on the ride and learning with me!

Not financial advice. Do your own due diligence.

Read more

Electro-absorption modulated lasers (EMLs) are the component inside optical transceivers that converts electrical data into light. It’s a laser and a high-speed modulator on a single indium phosphide chip, and every 800G and 1.6T transceiver needs them.

Lumentum makes more of these lasers than anyone, somewhere between 50-60% of the global market. Yet demand outstrips supply by roughly 30%!

Lumentum CEO Michael Hurlston said as much on the Q2 earnings call:

“We’re undershipping our customers’ demand by somewhere around 30%. Even as we’ve added 20% additional capacity, the demand-supply imbalance has increased.”

Dang. All EML capacity is locked under long-term agreements through calendar 2027. Customers coming back for more than their LTAs cover are paying premium prices. Customers who won’t sign LTAs risk losing supply priority entirely. Must be nice!

As CFO Wajid Ali said,

the LTA structure is “allowing us to have incremental pricing discussions around those incremental units.”

Of course, this pricing power is showing up in the financials. Revenue hit $665M, up 65% YoY, with $805M guided next quarter. Operating margins went from 7.5% to 25.2% in twelve months, and 30%+ guided next quarter. Whew!

Yeah, that explains why the stock trades at more than 60x forward earnings, and the current price has run ahead of the average sell-side target.

But bottlenecks and winning like this encourage competition.

The question thus is how long this temporary advantage holds, and whether the stock already prices in the answer. NFA, DYDD.

To get there, we first need to understand why the entire AI data center is going optical, and then why the laser layer is a bottleneck.

Why the Data Center Is Going Optical

Three shifts are expanding the optical TAM.

First, scale-out networking at 800G and above is increasingly optical. At 100G-per-lane speeds, copper alternatives like Credo’s AECs still work for short reaches (up to ~5-7 meters) and ALCs could push that to 30 meters. But anything beyond that requires optical transceivers, and each one needs EML laser chips. As lane speeds jump to 200G for 1.6T, copper reach compresses further, expanding the optical TAM with every speed generation.

Second, scale-up is going optical. Copper has dominated inside-the-rack connectivity (NVLink), but it’s hitting a physical wall at higher speeds. The industry is actively exploring alternatives. Nvidia demonstrated die-to-die optics at ISSCC in February using TSMC’s COUPE 3D hybrid bonding platform with silicon photonics engines. Marvell acquired Celestial AI for its photonic fabric technology. Ayar Labs (backed by Intel and Nvidia) is building optical I/O chiplets for in-package chip-to-chip communication. Broadcom’s ESUN spec targets Ethernet-based optical scale-up. Even Credo, the copper champion, is hedging with ALCs that use micro-LED light sources for row-scale scale-up.

This creates an entirely new optical TAM that didn’t exist two years ago. Lumentum CEO Hurlston on the Q2 call:

“Copper has long been the gold standard for scale-up... it is hitting a physical wall. An industry pivot is underway. By late calendar 2027, we would expect our first scale-up CPO shipments, replacing longer copper connections.”

One note: I see folks conflating Nvidia’s CPO announcements with scale-up. Nvidia’s current CPO switches (Quantum-X Photonics, Spectrum-X Photonics) are on the scale-out fabric, not scale-up. NVLink remains copper today. Optical scale up is a transition that’s just in its infancy.

Third, Optical Circuit Switching is a new category. OCS reconfigures optical paths in milliseconds, replacing fixed electrical spine switches. Google pioneered this for AI cluster networking. I think this technology is awesome; it’s lower-power because it stays in the optical domain, so there's no need to convert light into electrical signals for switching and routing.

Google historically built its own, but the technology is moving to merchant vendors. Lumentum’s OCS backlog went from zero to over $400 million in roughly a year. Three customers, four use cases (spine replacement, scale-across, scale-up, redundancy). Cignal AI projects the OCS market will exceed $2.5 billion by 2029.

Lumentum’s approach uses MEMS-based mirrors:

So Where Does Lumentum Stand?

Lumentum is a great spot, with three expanding optical TAMs (scale-out, scale-up, and OCS) and shipping lasers into all of them.

And icing on the cake, just today Nvidia invested $2 billion in both Lumentum and Coherent today, with multibillion-dollar purchase commitments for future capacity. See NVIDIA Announces Strategic Partnership With Lumentum to Develop State-of-the-Art Optics Technology

As we said, bottlenecks attract competition though. Coherent is ramping 6-inch indium phosphide wafers, which could yield 4x as many dies per wafer at significantly lower cost. Tower and GlobalFoundries are 5Xing SiPho capacity, which could eventually route around EML-based transceivers entirely. Mesh Optical Technologies, a startup founded by SpaceX alumni, raised $50M to build 1.6T transceivers using semiconductor-style packaging instead of traditional manual optical assembly. The typical big claims from a new startup, but an interesting point to watch because it would simultanesouly increase capacity and decrease cost.

The nearest-term capacity risk is Coherent. If Coherent’s 6-inch yields truly close the gap with Lumentum’s 3-inch performance (as they claim), the scarcity that drives Lumentum’s pricing power will end. There’s still a side-by-side performance comparison to make at that point, not to mention future generation laser supply constraints. Whoever can get there first can enjoy the same pricing dynamics.

I broke Lumentum’s story into some clear points of tension to track, and scored each against Q2 FY2026 earnings and sell-side research.

Here’s what’s behind the paywall:

• Lumentum’s business units: the component and systems businesses, the epitaxy moat, and why even competitors buy LITE’s lasers

• Four Growth Vectors: EML, OCS, CPO, and optical scale-up. What’s shipping now, what inflects in H2 2026, and what’s a 2027+ story

• Tensions: bull vs. bear on the EML moat, OCS market creation, CPO timing, supply chain risk

• Sell-Side Sentiment: what the Street is saying and where analysts disagree

• Earnings Scorecard: each tension scored against Q2 results, management quotes, and sell-side evidence

Read more

I teased xLight and Substrate and promised deeper dives on both.

Today I’m delivering the first one on xLight. Now that we understand the broader economic problem the industry needs to solve, let’s dive into technical details and areas for improvement.

We’ll start with some fundamental limits of LPP EUV scanners.

The Photon Problem

Every ASML EUV scanner generates its light the same way. It’s called Laser Produced Plasma, or LPP. It’s the “shoot tin droplets and hit ‘em each twice with a laser” light source magic trick. I’m assuming most readers are somewhat familiar, but if not here are great YouTubes references get you up to speed plus a short recap

How EUV’s light source LPP works: A high-intensity CO2 laser fires at tiny droplets of molten tin traveling at 100 meters per second, 50,000 times per second. Each droplet is actually hit twice: first a pre-pulse reshapes the 30-micrometer tin sphere into a concave sheet, then the main CO2 pulse vaporizes it into a dense plasma that emits 13.5nm EUV photons. These photons bounce off a series of collector mirrors, pass through a photomask, and expose the wafer below. As you can imagine, it took decades to get this concept to work at reliability and yield needed for scaled manufacturing. It’s freaking nuts that it works. That’s what I love about the semiconductor industry by the way, literally everywhere you look it’s just mind-blowing technology.

One interesting thing to note is that the industry actually first tried particle accelerators called synchrotrons to generate this high-energy, small-wavelength light. But at the time, synchrotrons didn’t have enough directional light. Next they tried xenon gas lasers, but that had a fundamental physics ceiling resulting in less than 1% conversion efficiency. Eventually the industry landed on tin back in 2002. Blasting tin particles into a plasma produces EUV light, but it has a debris problem (contaminating everything). That’s an engineering problem though (how to deal with this debris) but not a fundamental physics roadblock.

Fast forward 20 years and LPP with tin works. Hitting 50,000 falling tin droplets per second with a laser, twice each, 24/7. It’s genuinely an engineering marvel, should be discussed in freshman physics and engineering classes to get people excited about science

But.... the best LPP achieves only about 6% conversion efficiency. 94% of the energy put in is wasted. And the system has fundamental problems that get worse as chips get smaller.

For example, EUV photons carry about 14x more energy than the 193nm photons used in previous-generation DUV lithography. That means for the same power output, an EUV source produces 14x fewer photons.

This is where dose comes in. Dose is the amount of light energy delivered per unit area of the wafer, measured in millijoules per square centimeter (mJ/cm²). Think of it as the photon budget for each exposure. The photoresist (the light-sensitive chemical coating on the wafer) needs a minimum number of photons to trigger the chemical reaction that creates the pattern. Too few photons and edges come out rough or some features are missing:

The required dose isn’t a knob you can turn down to save time, as it’s set by the resist chemistry and the feature size. And as features shrink, dose requirements go up. Why? Because smaller features mean each “pixel” on the wafer covers fewer atoms of resist. Fewer atoms mean fewer photon interactions, leading to greater statistical randomness in whether the pattern prints correctly. To compensate, you need more photons per pixel (a higher dose) to keep the randomness under control. The industry calls these randomness problems stochastic effects, and as Asianometry puts it:

“Stochastic print failures are far smaller, completely random and thus non-repeating, and come via a law of nature. The randomness is unavoidable. So how do we usually deal with randomness caused by the law of small numbers? We increase numbers. Just send more photons.”

So every new node needs a higher dose, but the light source isn’t getting proportionally brighter.

For example, a Low-NA EUV tool is rated for about 220 wafers per hour:

But that rating assumes a dose of 30 mJ/cm². Move to High-NA which prints finer features and needs a higher dose and look what happens:

Notice throughput is only 175 wafers per hour at 50 mJ/cm².

The more advanced tool is slower because the higher dose eats into throughput. High-NA can save you total exposure steps by eliminating double patterning, which helps. But the long-term trend is evident. Every step to smaller features demands more dose, and more dose means fewer wafers per hour unless the light source gets brighter. Today’s LPP sources produce about 500 watts of EUV power. ASML has shown a roadmap to 600-800 watts. Note: per this week’s ASML disclosure in Reuters, ASML says its R&D team has unlocked 1,000+ W which would increase throughput back up to >300 wph.

But it’s still not enough. Why not? Because dose requirements are increasing too. Low-NA needs 30 mJ/cm² and High-NA needs 50+ mJ/cm². Future nodes will push higher as stochastic effects worsen with each shrink.

And that’s just the power problem. Most of the photons you do generate never reach the wafer. Tin debris fogs the collector mirrors. A layer just 1.2nm thick cuts collector efficiency by 20%. And each mirror in the optical chain absorbs about 30% of the light that hits it, so after the full series of reflections, less than 10% of source photons make it to the wafer.

Then the kicker. Every scanner has its own dedicated LPP source. A fab running 15 EUV tools is operating 15 separate tin-burning light sources, each 6% efficient, each losing 90%+ to mirrors, each fighting its own debris.

Whew. Again, it’s a miracle it even works. But it’s pretty inefficient, and the problems mount as we look ahead on the EUV roadmap.

With LPP light sources, lithography costs will continue to increase. It’s inevitable. And as we discussed in the first article in this series, those costs have a real impact on the broader industry.

Kind of reminds me of this chart... we need to bend the cost per chip of lithography downward over time...like good technology curves do…

A Different Way to Make Light

But what if there were a light source that produced orders of magnitude more power, eliminated tin entirely, and could serve an entire fleet of scanners from a single system?

That’s a Free Electron Laser.

(Shout out Asianometry again! A great video on the topic if you want the deep dive.)

Remember that the industry first tried synchrotrons for EUV back in the 1980s, but rejected them because the light was too diffuse.

A Free Electron Laser is different. Instead of passively emitting radiation as electrons circle, an FEL actively amplifies coherent, directional light through a linear accelerator. Directional light! That distinction solves the problem that killed synchrotrons.

Here’s the short version of how an FEL works. A linear accelerator shoots electrons to near light speed, then sends them through an undulator, a chain of alternating magnets that wiggles the beam back and forth. That wiggling causes the electrons to emit photons, which interact with the beam itself, causing the electrons to self-organize into coherent bunches that amplify the light exponentially. The result is intense, coherent, laser-like light:

Exactly what EUV scanners need! No tin. No plasma. No debris. And you can tune it to 13.5nm to plug in this light source directly into existing 13.5nm EUV scanners!

Three properties make FELs really awesome for lithography.

1. Tunable wavelength. By adjusting the electron energy and magnet spacing, you can dial the output to any wavelength: 13.5nm for EUV, smaller for future “Beyond EUV” nodes, or even dial it back and use longer wavelengths for less cutting-edge process nodes.

2. Energy recovery. Instead of dumping spent electrons, you loop them back through the accelerator in reverse, recovering most of their kinetic energy. FEL light sources can be several times more efficient than LPP.

3. No tin, no debris. The entire contamination problem goes away. Cleaner light means longer mirror life and less maintenance downtime.

Enter xLight

xLight is a venture-backed company in Palo Alto building Free Electron Lasers purpose-built for semiconductor manufacturing. Their team comes from the US National Lab ecosystem, with deep experience designing, building, and operating FELs for research and national security. They’ve partnered with Fermilab on the superconducting cavity technology at the heart of the accelerator.

And they’re not doing this in a garage. Their prototype is being built at NY CREATES in Albany, NY, a semiconductor R&D hub where IBM runs a full 2nm process flow and where Tokyo Electron, Applied Materials, and other major equipment companies have a presence. xLight’s prototype was accelerated by investment from the Commerce Department as part of the CHIPS Act R&D program.

Replace the Source, Not the Scanner

So… what’s the product xLight sells? Well, light. xLight doesn’t compete with ASML’s scanners, but rather, replaces the light source. Unplug LPP, plug in xLight’s FEL.

xLight’s FEL is backward compatible with ASML’s existing scanner platforms. From the scanner’s perspective, the swap is transparent; the light comes in at the exact same angle, same wavelength, through the same intermediate focus point:

The only difference is that this approach doesn’t have tin splattering around and doesn’t have out-of-band radiation causing heat in the optics. In fact, because FEL light has a much narrower bandwidth than LPP, there’s less junk light and less thermal noise. Hence, the same scanner that’s designed for LPP light can actually handle more FEL light.

Oh, and did you notice in that image how many scanners are all being fed at the same time? This is the economic differentiator.

120 Kilowatts

xLight’s system generates 120 kilowatts of EUV power at the source.

Remember, current LPP sources produce just over 500 watts. ASML’s roadmap targets 800 watts. xLight is building a source that produces 120 kilowatts. Over 100x more power!

Now, you don’t get all 120 kW to the scanner. As light is distributed through the fab via vacuum beamlines and grazing-incidence optics, there are transmission losses. Depending on whether it’s a greenfield or brownfield fab, you deliver somewhere between 25% and 50% to the scanner fleet. Divided across up to say 16 scanner connections, that’s roughly 2-4 kilowatts per scanner. That’s still several times more than what LPP delivers today…

Abundant light unlocks the full dose at full productivity. No more running $500M High-NA tools at a slower speed because you can’t deliver enough photons. And beyond throughput, more light means better patterning. And that unlocks better uniformity and line edge roughness, resulting in higher yields. win win win.

xLight’s approach seems compelling. But there are still a lot of questions, like

• How do they plan to sell it?

• Who are the customers?

• How to fund it? Building particle accelerators sounds expensive…

• What does this mean for the broader lithography ecosystem?

Interestingly, if xLight’s economics work, it doesn’t necessarily disrupt ASML, and could arguably expand ASML’s addressable market. It’s nuanced though. We’ll get into it, as it has implications for anyone modeling ASML’s long-term revenue.

And the business model itself is fascinating. xLight doesn’t sell light sources. They sell light as a service, like a utility. Your electric bill is really just consumption-based electrons-as-a-service, right? We’ll dig into this.

And finally, adoption seems hard, right? I mean, semiconductor manufacturing is interestingly both super high-tech and yet quite conservative in some respects. Will they adopt?

Let’s get into it.

Read more

Is that justified? Or is Arista priced for perfect execution in a market that could stumble?

The answer depends on six specific debates: Ethernet vs. InfiniBand, white-box competition, customer concentration, gross margins, supply chains, and valuation.

I scored each one against Q4 earnings evidence. But first, you need to understand why the Ethernet TAM is expanding faster than most investors realize.

Ethernet’s Opening Is Wider Than You Think

Nvidia has historically dominated AI cluster networking with a vertically integrated stack. NVLink for scale-up, InfiniBand for scale-out, CUDA libraries like NCCL to make it all programmable. The famous Mellanox acquisition! Nvidia’s networking segment alone generated $8.2B in Q3 FY2026. Up 162% year-over-year!

But two shifts underway are expanding the TAM in Ethernet’s direction. (Read GPU Networking Basics Part 3 if you need a refresher on scale-up vs. scale-out.)

First, Ethernet is making inroads in scale-out. InfiniBand was purpose-built for ultra-low-latency distributed computing, but Ethernet has been closing the gap through RoCE (RDMA over Converged Ethernet) and a new generation of lossless Ethernet fabrics designed for AI. Nvidia itself launched Spectrum-X, its own Ethernet-for-AI platform, and Meta, Microsoft, Oracle, and xAI are all building on it. But Spectrum-X isn’t the only game in town, as hyperscalers are also choosing vendor-neutral Ethernet fabrics. The Ultra Ethernet Consortium is standardizing this.

Second, scale-up is no longer Nvidia-only. Scale-up used to mean NVLink. Only game in town. But as I covered in GPU Networking Part 4: Year End Wrap, every major XPU vendor now has its own scale-up architecture, and several are building on Ethernet:

AMD Helios uses Ethernet for both scale-up and scale-out. Microsoft Maia 200 is Ethernet-native for scale-up. AWS Trainium3 uses Ethernet for scale-out. Even Nvidia now offers Spectrum-X Ethernet alongside InfiniBand. As accelerator vendors build on Ethernet and hyperscalers standardize on it, more of the AI cluster fabric shifts into domains served by Ethernet switch vendors.

Take Microsoft’s Maia 200. As Saurabh Dighe explained in our interview: “We have not invested in a scale-out network. We have invested in an inference-driven chip. We have invested in taking a scale-up approach more than a scale-out approach. That’s brought our cost down.” Microsoft deliberately chose Ethernet-native scale-up and skipped proprietary scale-out entirely, optimizing for inference cost per token. That design choice routes directly through standard Ethernet switches. Pun intended.

Agentic AI Adds Another Layer of Demand

GPUs don’t operate in isolation. Every GPU needs a host CPU for orchestration, data loading, and network management. Inside a rack, CPUs and GPUs are increasingly tightly coupled; Nvidia’s GB200/GB300 systems integrate them via NVLink-C2C into a single coherent unit. But agentic AI adds another layer of Ethernet demand on top of that.

In agentic workflows, a CPU-heavy orchestration tier sits outside the GPU racks running agent loops, tool calls, sub-agent coordination, and dispatching inference requests to GPU pools over the network. Vik Sekar’s the CPU bottleneck in agentic AI illustrates:

That’s a new class of traffic between CPU server racks and GPU clusters, all flowing over Ethernet. Add in scale-across opportunities (connecting clusters to clusters), and the networking TAM keeps expanding.

So Where Does Arista Stand?

Ethernet is gaining share in scale-out, making inroads in scale-up, and now serving a new orchestration tier driven by agentic AI.

AI clusters need more ports per rack, faster speeds (400G → 800G → 1.6T), and less oversubscription than traditional cloud networks, meaning more switches per cluster at higher ASPs.

And Arista shipped 150 million Ethernet ports in FY25, grew revenue from $5.9B to $9.0B in two years, and just guided for $11.25B in 2026.

But… Arista doesn’t design its own chips. It outsources manufacturing. Its two largest customers account for 42% of revenue. And its CEO just called memory costs “horrendous”.

So is the stock a compounding machine riding a structural Ethernet tailwind? Or is it priced for perfection in a business with real concentration risk, margin pressure, and a supply chain that can’t keep up with demand?

I identified six specific debates that drive Arista’s valuation and scored each one against Q4 FY2025 earnings evidence. Here’s what’s behind the paywall:

• What Arista actually sells: the business units, the EOS software moat, and why hyperscalers keep buying blue-box switches when white-box is cheaper

• The Six Debates: bull vs. bear on Ethernet vs. InfiniBand, white-box competition, customer concentration, gross margins, the “golden screw” supply chain, and whether ~40x P/E is justified

• Earnings Validation Scorecard: each debate scored against Q4 results, with key management quotes

Read more

Wall Street models Nvidia’s autonomy business as a ~$1-2B automotive SoC competing with Qualcomm and Mobileye for in-car socket wins. But Nvidia’s current automotive opportunity is actually $5-10B. Why? The data center infra revenue from automotive OEMs is significantly larger than the in-car silicon, but it’s hidden within the Data Center revenue line. Let’s dig into that more, and also ask:

• Why would Nvidia open-source its VLA models? Expensive to train!

• How does Nvidia’s autonomy business compare to Mobileye and Qualcomm?

• Is custom silicon from Tesla and Rivian a threat?

• Could Chinese OEMs be a tailwind?

• Does AMD have a play here?

Three Computers, Not One

At CES 2026, Jensen explained the “three computers” concept:

“Inferencing the model is essentially a robotics computer that runs in a car... But there has to be another computer that’s designed for simulation… One computer, of course, the one that we know that Nvidia builds for training the AI models.”

So three “computers” are

1. Frontier training cluster (e.g., NVL72 or Rubin)

2. Simulation node (could be on older GPUs like Hopper, or cost optimized GPUs like Blackwell RTX Pro 6000, etc)

3. Onboard compute in the car (e.g. Orin, Thor)

Nvidia’s reported automotive revenue is #3 only, which dramatically understates the real business. Jensen said that automotive is

already a multibillion-dollar business... venture to say somewhere between $5 billion to $10 billion...And by the end of 2030, by the end of the decade, it’s going to be a very large business

JP Morgan Research confirmed as much in its Oct 15th 2025 report From Chips to Cars: Deep Dive into ADAS and Robotaxis:

“Within its automotive industry vertical, Data Center compute/networking currently represents a more significant revenue stream for Nvidia than its in-car hardware/software business.”

Thus, Nvidia’s serviceable addressable market spans all three computers: training infra, simulation, and onboard compute. That’s a fundamentally larger SAM than any competitor focused only on the in-car socket.

Given Nvidia’s Ampere/Hopper/Blackwell dominance in data center GPUs, it’s very likely Nvidia gets paid for computers #1 and #2 for most OEMs regardless of which vendor wins the in-car socket. Nvidia can lose, but still win.

The Three-Computer Business Model

Let’s go one layer deeper.

Training is the largest revenue driver for Nvidia. Every OEM needs massive GPU compute to train AV models on their own sensor data. For example, per JPMorgan’s February 2026 Korea Autos note, Hyundai alone “signed a supply contract last October with Nvidia for 50,000 Blackwell GPUs to build its data center.” At roughly $30-40K per GPU plus networking, that’s an estimated $1.5-2.5B deal from a single automotive OEM, rivaling Nvidia’s entire reported annual automotive revenue! But it shows up in the Data Center segment, not Automotive. This is the perfect illustration of why Nvidia’s reported autonomy business looks undersized, as the largest revenue streams are hiding in a different line item.

Simulation in Omniverse is important for safe AV development. No one should be skipping computer #2. An example workflow: when something fails, the autonomy ML team reconstructs the scene using NeuRec (neural reconstruction) and tests a fix on exactly that failure.

That might seem crazy at first blush. No harm if your reaction is: what, we feel confident putting our lives in the hands of a model thats… tested on computer simulations….!?

But the fidelity of such a simulation is quite good these days.

Herman Ross, Nvidia’s Director of Simulation Ecosystem Development, noted that reconstructed sensor data is “pretty much impossible to tell apart from the real sensor. So the gap there is considered to be close to zero.”

Anyway, after reconstructing the scene, Cosmos Transfer can give you many “similar but different” scenarios for testing. So a single recorded driving scene can be transformed via prompt into a different location and different weather pattern in under 60 seconds.

In-car compute is the final piece. Nvidia has the AGX Orin and Thor portfolio, from entry-level (~100 TOPS Orin) through dual-Thor for L3/L4.

BYD, for example, uses the entry Orin for its Gods Eye highway L2+ product.

Rivian’s Gen2 platform showed how this business model plays out in practice, as it ran on dual Orin chips in the car and accessed “tens of thousands of GPUs” in the cloud for training, without owning the clusters. Read up on it here:

So that’s the framework: three computers, not one. Training dwarfs the in-car business. Simulations adds up too. Now, in the long run, I think the onboard compute market will grow significantly too. I’m bullish on that TAM long-term. Won’t more than half of new cars by 2030 come equipped with autonomy functionality? And that’s before counting commercial trucking (e.g. Aurora) and robotaxis.

Ok, so now you appreciate the Three Computer business model. But that still leaves questions.

• Why would Nvidia open-source its AV models?

• Nvidia has tons of OEM engagements — how do you monetize that without building a bunch of custom stacks? After all, they each have different sensors and compute configurations...

• Qualcomm leaves one of the computers (training) to Nvidia, but what about simulation?

• Which OEMs are actually software-defined versus just marketing it? And how might that impact adoption

• What does the L2-to-L4 economic shift mean for who captures value over the next five years?

• And might AMD have a play?

We’ll get into all of that below the fold.

I’m aiming for institutional-grade quality, but this is industry analysis, not investment advice.

Read more

Amazon.com Inc. shares dropped the most in six months after the company announced plans to spend $200 billion this year on data centers, chips and other equipment, worrying investors that its colossal bet on artificial intelligence may not pay off in the long run.

The market reaction is understandable. $200 billion is unprecedented. FCF will almost certainly go negative in 2026. Management didn’t communicate any guardrails when asked directly.

And ROIC is structurally harder for Amazon than Meta, Google, and even Microsoft. AWS runs at a 35% operating margin, but accounts for less than 20% of total revenue. The rest of revenue comes from retail at single-digit margins. So Amazon has to justify the biggest CapEx in corporate history through cloud margins alone.

On that front though, the market may be underappreciating the custom silicon story. Could Trainium lift AWS margins? And, should advertising revenue and margins be broken out to show how Trainium can directly impact non-retail margins?

For paid subscribers, we’ll dig into all of that and more:

• How the custom silicon narrative shifted from defensive to confident in one quarter, and what the $10B chips ARR disclosure means

• Why Amazon’s margin structure makes the ROIC story structurally harder than Meta or Google, and what Trainium has to deliver to close that gap

• The CapEx escalation from Q3 to Q4. How $125B became $200B with no explicit guardrails

• Where the sell side landed and why

• Four things Amazon’s IR team should be telling investors, from Trainium margin quantification to the Graviton-agentic AI connection

Keep reading for institutional-grade work but with an independent perspective.

Read more

But Meta articulated the same strategy the entire time, and it’s actually quite simple. GEM now, LLMs later.

GEM now: immediate, measurable ROI in the core ads business
LLMs later: incremental core gains today, substantial potential upside later

Let’s dig in. Then we’ll address potential risks.

GEM Now

Let’s start with the basics.

Facebook launched the News Feed in 2006, ads in 2007, and by 2011 the feed was algorithmically curated. From the beginning, what you see in the feed and which ads you are shown have been driven by machine learning. Meta has used AI infrastructure to drive engagement and advertising revenue for nearly two decades, evolving from CPU-based datacenters to GPU-accelerated systems.

This is not a new pivot. It is the core of the business. Meta’s business has always depended on using AI infrastructure to drive engagement and generate advertising revenue.

Today, that stack includes Andromeda for large-scale ad retrieval, Lattice as a unified prediction architecture across objectives, and GEM, Meta’s foundation model for ad ranking. They are distinct components, but together they form a single system optimized around monetization.

BTW, to learn the basics of Meta’s core business, I highly recommend this podcast episode between Eric Seufert and Matt Steiner, Meta’s Vice President of Monetization Infrastructure, Ranking & AI Foundations.

An important topic to understand is the Generative Ads Recommendation Model (GEM), Meta’s foundation model for ad ranking. Conceptually, it’s a recommendation system that makes one decision, billions of times per day: which ad should this person see right now?

Again, ML recommendation systems have always driven Meta’s business, but this latest model, GEM, introduces an important innovation. Meta figured out how to unlock scaling laws for recommenders in a similar fashion to LLMs, where more training compute yields better rankings:

“This is the first time we have found a recommendation model architecture that can scale with similar efficiency as LLMs.”

Better rankings lead to more revenue. So when Meta says they want to buy more GPUs, there’s a clear line of sight to increased revenue.

This was missed by investors, who early on punished Meta’s CapEx increase with confused statements like “well Meta isn’t a CSP, so it’ll take longer to get an ROI on that CapEx”. Ummm… have you compared cloud margins to Meta’s advertising margins? Meta’s gonna have no trouble paying off that incremental compute with incremental advertising dollars from GEM improvements, even if it takes a bit longer than renting out a GPU…

Investors need to think about the ROI of Meta’s incremental compute differently. Remember, Meta is building recommendation systems like GEM and training frontier LLMs. The ROI has to account for these separately.

So, some things to note. LLM AI labs build massive models (e.g. 1T+ params) that run in production at enormous cost (think Claude Opus). GEM doesn’t work that way. From CFO Susan Li on the recent Q425 earnings call in January,

“So we don’t use our larger model architectures like GEM for inference because their size and complexity would make it too cost-prohibitive. The way that we drive performance from those models is by using them to transfer knowledge to smaller, lightweight models that are used at runtime.”

GEM is a teacher. It absorbs massive user, content, and advertiser signals, then distills that knowledge into small, latency-sensitive models that serve ads. Inference remains cheap. Scaling GEM during training does not translate into a proportional increase in runtime model size or serving cost.

And that efficiency is improving:

“In Q3, we made improvements to GEM’s model architecture that doubled the performance benefit we get from adding a given amount of data and compute.”

Double the performance. In other words, the same incremental compute+data is now producing roughly twice the impact.

Viewed through that lens, the right question for evaluating Meta’s CapEx is not whether its models beat OpenAI or Anthropic on public benchmarks of how “smart” the AI is. Rather, GEM CapEx should be measured as incremental revenue per dollar of compute. Put differently, Meta is underwriting CapEx against an internal efficiency curve tied directly to monetization, not against leaderboard performance.

On a risk-adjusted basis, I think we can all agree GEM is a more defensible place right now to deploy capital than frontier LLMs, whose returns remain uncertain. When do OAI and Anthropic become profitable? Whereas GEM is immediately profitable… As I said before,

Wouldn’t others (cough cough OpenAI) LOVE to have 3.5B DAILY active users… and …. Wait for it… make almost $60 PER USER PER YEAR!

As we’ll see later, of course, Meta is investing in LLMs, and there’s upside there… but just wanted to point out that CapEx spent on GEMs is a no-brainer.

Now, you might ask, are we sure additional CapEx spent on GEM is truly driving revenue? Concrete results from the earnings call:

• +5% Instagram ad conversions

• +3% Facebook Feed ad conversions

• +3.5% Facebook ad clicks in Q4, alongside sequence learning

With a $150B+ ad business, each of those incremental points seriously adds up.

Again, that’s a lot more straightforward than investing in larger training clusters to unlock an extra IQ point for a GenAI LLM. I’m not saying GenAI LLMs aren’t worth it — just pointing out the clear line between CapEx and measurable results.

And while incremental revenue from improved ad recommendation is great, so is decreasing the cost of that compute. After all, a dollar saved goes straight to the bottom line. One way to decrease cost is through compute diversification. Like with custom silicon (MTIA):

“We extended our Andromeda ads retrieval engine, so it can now run on NVIDIA, AMD, and MTIA. This, along with model innovations, enabled us to nearly triple Andromeda’s compute efficiency.”

“In Q1, we will extend our MTIA program to support our core ranking and recommendation training workloads, in addition to the inference workloads it currently runs.”

This looks a lot like the Microsoft Maia conversation, where Saurabh discussed building custom silicon to optimize known workloads:

I hope to talk with Meta about MTIA in the future so we can dig into the rationale and ROIC of the CapEx invested in the MTIA roadmap.

And one interesting note. While GEM focuses on ads, Meta is explicitly unifying ads ranking with organic content:

“We’re also going to start validating the use of ad signals in organic content recommendations as we continue to work towards having a more shared platform for organic and ads recommendations over time.”

One model ranking both organic and paid content across 3.5B daily users. Sounds like a good use of CapEx.

Part II: LLMs Later

So GEM is the sure thing — CapEx with a clear, measurable payoff. But Meta isn’t stopping there. Zuck is investing heavily in Meta Superintelligence Labs, co-led by Alexandr Wang and Nat Friedman, to build frontier LLMs. And this is where most of the Street’s anxiety lives, because it pattern-matches to the metaverse spending era. But the framing of “Meta needs to beat OpenAI and Anthropic” misses the point.

The way I think about it: GEM alone can deliver serious returns on invested capital. LLMs are layered on top. And some of that LLM upside is already showing up in the core business today. Some of it is more speculative — future products and features that could expand engagement in ways we can’t fully predict, yet.

Let’s start with the straightforward stuff that LLM enables:

Better understanding of what people actually want. Traditional recommendation systems are mostly pattern-matching on past behavior. LLMs bring something different, as they can reason about what someone might be interested in, even without a ton of historical data:

“Building new model architectures from the ground up that will work on top of LLMs, leveraging the world knowledge and reasoning capabilities of an LLM to better infer people’s interests.”

Making it easier for businesses to create ads. One of the biggest constraints in digital advertising is creative (photos, videos, etc). Small businesses don’t have design teams cranking out polished video ads or the means to outsource. GenAI changes that. If anyone can quickly and cheaply generate ad variations, more businesses can run better ads. That’s good for advertisers and good for Meta’s revenue. From 2026: AI Drives Performance:

AI is also powering stronger ad creative. In Q4 2025, the combined revenue run-rate of our video generation tools hit $10 billion, with quarter-over-quarter growth nearly three times faster than overall ads revenue.

Solving the cold start problem. Recommendation systems struggle with new content because there’s no engagement data to work with yet. LLMs can fill that gap because they can infer what a piece of content is about and who might care before anyone’s even seen it:

“We will work on more deeply incorporating LLMs into our existing recommendation systems… useful for content that has been more recently posted since there’s less engagement data.”

These are all additive to GEM. But there’s also the less predictable upside of future features that expand the surface area of engagement itself. One that caught my eye is AI dubbing. Meta is auto-translating videos into nine languages, and hundreds of millions of people are watching them every day.

“One area we’re already seeing promise is with AI dubbing of videos into local languages. We are now supporting nine different languages, with hundreds of millions of people watching AI-translated videos every day. This is already driving incremental time spent on Instagram, and we plan to launch support for more languages over the course of this year.”

That’s content that simply wouldn’t have reached those users before. More content people want to watch means more time spent, which means more ad inventory.

IMO the bull case is clear. But what about the risks?

Behind the paywall, we dig into the noise surrounding Meta’s AI strategy: why Meta is spending billions on frontier models when it could just use OpenAI or Anthropic as teachers, whether this is really different from the metaverse spending era that has yet to payoff, and whether there’s a hard ceiling on engagement gains when you already reach half the planet every day. We also cover a less obvious near-term catalyst for incremental time spent on Meta’s platforms.

Read more

Investors are telling us they don’t know why to own AMD right now.

The MI450 inflection doesn’t hit until Q4 2026, and investors won’t see those numbers until early 2027. That’s a long time to wait. Server CPU is strong, but is it enough? Client faces a softening TAM. China's revenue was helpful to the beat but might not be repeatable.

Meanwhile, Nvidia is describing an expanding inference datacenter portfolio, with recent announcements including Rubin CPX, Context Memory Storage, and Groq. What’s AMD’s counterpositioning?

Here’s my breakdown of the quarter. What the numbers say, what management said on the call, and what they didn’t say. Including:

• MI450 timing: why Q4 2026 is perceived as the only 2026 quarter that matters, and the buy-side’s why now? problem

• OpenAI: the leading indicator investors should watch and AMD should talk about

• Inference strategy: AMD vs Nvidia’s portfolio

• Server CPU: a bright spot. But 2026?

• China: strip out $390M in MI308 sales and the beat looks different

• Client and Embedded: where margins are heading in 2027

Let’s dig in.

Read more

Infrastructure Economics

Lithography Economics: EUV has a cost problem. Two US-based startups, xLight and Substrate, are taking different paths to bend the cost curve.

LRO/LPO Optics: At datacenter scale, every watt consumed by networking is a watt not available for compute. LRO and LPO offload transceiver DSP to the switch, trading modularity for power efficiency. They’re a stepping stone toward co-packaged optics.

Credo’s Reliability Thesis: AECs made Credo. But Credo is more than AECs.

Post-GPT Architecture

Microsoft Maia 200: Maia 200 is inference-first by design with a large on-die SRAM, 6,144-accelerator scale-up via Ethernet, 750W TDP, and no scale-out network. These decisions unlock a 30% improvement in price-performance. Deployed through Azure services, not bare metal.

Agentic Workloads: Speed isn’t enough for coding agents. Context grows with each iteration, and the KV cache must stay hot. Can pre-GPT accelerators built for stateless inference handle memory hierarchy? What about post-GPT designs (Etched, MatX)?

Frontier AI From Cloud to Desk in 5 Years: Frontier models have been migrating from the cloud to the desk on a 4 to 5-year cadence. GPT-4-class capabilities are now reaching workstations. What’s the implication? Think minicomputers in the mainframe era: TAM expansion.

Autonomy and LiDAR

LiDAR Primer: A primer on how LiDAR works. Wavelengths (905nm silicon vs 1550nm InGaAs), sensing methods (ToF vs FMCW), and why the technology is now viable at scale.

LiDAR Market: Waymo’s momentum has triggered an L4 gold rush. L3 programs are reaccelerating now that OEMs are decoupling ADAS from stalled EV platform transitions. Behind-the-windshield is the winning form factor. The market is consolidating toward one or two Western suppliers.

Intel

Intel Q4’25: Back to Reality: The stock run-up was vibes; the correction was reality. Supply constraints were telegraphed already, though. The turnaround is execution-bound, not quarter-driven. And don’t expect anything dramatic next quarter.

Intel’s Product Marketing at CES: B-: Intel led with real value props (battery, graphics) instead of AI hype. The AI PC pitch still lacks compelling GenAI use cases. Bright spot: Panther Lake on 18A shipping to edge and consumer simultaneously.

Semi Doped Podcast

I launched Semi Doped this month with Vik Sekar. The first seven episodes have surpassed 150K cumulative views on X. Follow along on X, any podcast player, or YouTube.