2026.09.30 | TB | AI Compute | Anthropic Ex-Exec on AI Infra & Compute

Client-led • 1 Sep 2026

Companies: NVDA US

Expert Former Senior Executive, AI Accelerators & GPUs at Anthropic PBC

Agenda

• Over 20 years' experience in the semiconductors and cloud compute space, with a strong understanding of hyperscale infrastructure and compute capacity for frontier AI platforms
• Knowledgeable on hardware selection trade-offs between proprietary silicon and Nvidia architectures, tracking performance and unit economics across massive B200 and GB200 clusters
• Well-placed to evaluate gigawatt-scale data centre expansion, capital allocation models and strategic hosting dynamics across major players such as Anthropic and OpenAI
• Well-versed in AI revenue monetisation, tracking ARR per gigawatt and capacity planning across research, internal inference and external commercial workloads

Introduction

Client [00:01:56]: Could you discuss the amount of compute currently available for Anthropic to use out of a gigawatt?

Expert [00:02:12]: I obviously cannot answer anything specific to Anthropic and obviously, something that is not publicly discussed. Obviously, OpenAI has a lot more sovereign compute compared to, let's say, Anthropic, which is still heavily reliant on the hyperscalers, but it's also building its own with the whole (inaudible) Broadcom and Google deal. Maybe you can ask me another dimension that I can help you with.

Questions

00:02:54] Rumours suggest that by the end of 2026, major model developers, Anthropic and OpenAI, will each hold approximately 5GW, totalling 10GW. Does this align with your knowledge? Does the industry expect two major laboratories, Anthropic and OpenAI, to hold around 5-6GW each by the end of the year?

Expert [00:04:18]: Yes. I think that is definitely directionally aligned given first of all, how much hyperscalers are building for these frontier labs, and secondly, how much of that is typically allocated to frontier labs. That is directionally aligned, yes. I think, obviously, given all the delays that we're seeing like power, etc, that might be off by, I would say, 0.5GW plus or minus. I think that is generally around the ballpark, yes.

Client [00:04:51]: Would that figure double to 10GW each by the end of next year?

Expert [00:04:59]: Correct. That is accurate. Yes.

00:05:04] When we discuss these two figures, are we referring strictly to capacity reserved for Anthropic's first-party use cases, such as first-party training and inferencing? This does not include capacity on Amazon AWS provided as part of their Bedrock services. Is that right?

Expert [00:05:29]: That is my understanding, yes, because when, let's say, even OpenAI makes the models available through AWS Bedrock, it is not counted towards dedicated capacity for OpenAI customers, where it is counted as an offering that hyperscalers serve to their customers. The way it typically works, you're likely aware of this, that the frontier lab company didn't get a cut out of this. Fundamentally, the capacity is reserved for the hyperscalers and their customers. It's not available to direct customers of frontier labs.

00:06:04] Could you help us understand the split between training and inferencing for the 5-6GW targeted for the end of the year? How do teams internally bucket compute across different use cases?

Expert [00:06:32]: First of all, the capacity is typically very fungible. In general, across the ecosystem, across frontier labs, it's very hard to put a number down because it changes everything from during the day based on model roadmaps to also based on demand and then also available supply based on inference peak. Overall, I would say, let's say, if I were to start a new frontier lab and if I were to allocate compute, I don't think the split really makes sense in terms of training vs inference because I think a few things. Firstly, the typical split is how much, A, you allocate to research, which is just throwing things at the wall and then experimenting new things out. Secondly, how much would you typically allocate to pre-training, which is the final long horizon run that you have to kick off once you've done all your research. Then, how much would you allocate to internal inference, which is part of the external inference as well? Then, production inference, as in external inference.

Fundamentally, if you have to just take a split, I would say about, typically, if I were to start, I think, first of all, I would allocate about 30% of this to research only because that is fundamentally extremely important and this is where a lot of unlocks happen. After that, I would allocate another, I'd say, let's say, about 20% or 25-30% more to pre-training, which is like these long horizon runs that, let's say, GPT-7 or GPT-6 or something, anywhere from five days to five weeks to five months. Then, the remaining is our production inference, but then part of it is also allocated to internal inference within the company. At the same time, there are just guardrails built around it. My point is you can just consider that as general inference, both paid inference vs unpaid inference across the (talking over each other).

Client [00:08:34]: What percentage of the 5-6GW would be allocated to internal inference?

Expert [00:08:39]: That depends a lot and it changes. I say it changes because it's very cyclical based on, again, the model roadmap where you need to do synthetic data generation, etc. I think that would be, I would say, anywhere like 5-10% of the total inference.

Client [00:08:57]: Is the remaining amount, approximately 40%, dedicated to external inference?

Expert [00:09:05]: Typically, yes. Again, all of these are swings of plus or minus 10%, sometimes even worst case, 20%, but not minus so much, but across the industry, even OpenAI has these swings. Again, this is very, very dynamic and organic.

00:09:24] What is the difference between research and pre-training and what is done differently in each? For the 30% allocated to research, what is it actually used for?

Expert [00:09:39]: Typically, across the industry, the research part is all these AI researchers who are just trying out completely new experiments, things like new attention mechanism, new model compression methods, new quantisation, new fine-tuning, reward modelling, mechanistic interpretability, safeguards, safety classifiers, red teaming, just everything that gives you the recipe to build your final pre-training recipe. Everything that gives you all the micro steps. Then, obviously, a lot of research typically is sequenced in the sense like it's not like you can try everything. It's more like just digging for gold. You have to keep trying somewhere or the other. Then, based on what you find, then you do the next step. My point is you need a lot more on-demand and typically available compute rather than just constant runs, which are typically what pre-training runs are, which is you just fire off and you give it all the data, you give it all the compute and you just keep monitoring the loss function. That is typically the pre-training runs. The research is very ad hoc. It's very exploratory.

Client [00:10:57]: You mentioned that research work is typically interruptible, whereas pre-training work is not interruptible, correct?

Expert [00:11:07]: Interruptible, correct, both from like, A, how typically spiky the workloads are across the industry. Then secondly, there are also short course runs. Typically, it is interruptible, it is preemptible. There's a lot of prioritisation that happens across research only because compute is limited and then you have to keep balancing it out.

Client [00:11:32]: Pre-training is on-demand, uninterruptible, and requires parallel execution.

Expert [00:11:39]: Exactly. Exactly. Because that has a significant penalty on any interruptions because you have to go back to checkpoints and you typically touch it. Then, secondly, it is also a little bit disconnected from all the other clusters. It has no reason being connected to mission-critical big scale across networks or anything.

00:12:07] How much compute is required for pre-training? Are fine-tuning, post-training and mid-training classified under training, pre-training or a separate category altogether?

Expert [00:12:32]: Across the industry, different companies do it differently. I don't think there's a straight-up definition. Technically, you can consider, yes, reinforcement learning also as part of pre-training because it has very similar characteristics of workloads. At the same time, it's not the same run.

Client [00:12:55]: The reinforcement learning category falls into the second bucket rather than the first research bucket.

Expert [00:13:01]: Looks like.

00:13:07] Given that parameter counts are increasing for new models, could you provide an illustrative example? For instance, for a 10-trillion-parameter parent model, how long does a training run take and how much compute is required for what duration to complete it?

Expert [00:13:43]: I do, but it's a little hard to answer because a few things. First of all, it really depends on how much data corpus is it actually trained on? What are the different attention methods involved, what are the hacks, etc? Then, also what kind of compute is available? Are we looking at just H100 equivalents? Are we looking at GB200 or 300, etc? I think it is hard to do the math. Let's assume, typically, based on how many are active parameters are there and by the way, it also depends on… Is it dense and is it sparse, MoE, etc. Zooming out, it's typically 6ND is the formula, which is the flops needed. 6x the number of parameters times the data format. I'm sure you know this is like the typical Chinchilla compute paper. Fundamentally, I don't know, I'm going to have to do the math, but I think a few things. This would give you the training compute. I'm going to have to do the math. This gets into really large numbers. 10 trillion times 200 trillion, let's assume it's 200 trillion of training tokens times six.

I think this would be about 12 zettaflops. I'm just putting up the calculator on my computer right now. If you assume a 40% model flops utilisation or MFU, which is chip utilisation that's typically used for training, sometimes you can go as high as 50%, but oftentimes it's also low as 30%. I think 40% is a good average. I'm going to have to look at the H100 equivalent. My point is, let's say, if it's H100, this can be anywhere from 120-something days. I'm going to have to do the math. I wasn't prepared for this level. It could be anywhere, I would say, from four months if you have let's assume 1 million H100 GPUs vs if you have 1 million B200, I think it is likely going to be half of that. 120 or 150 days and half of that would be 60-80 days. This assumes that almost 1 million GPUs because obviously, 10 trillion is a massively big model. I don't think the industry has really done this. Even Kimi K3 was 2.8 trillion, that was already too big to train, but more importantly, also extremely big to serve. I can do the math. It's just that I'm going to have to just do it in a spreadsheet.

00:16:57] How much compute, how many gigawatts and how much time would be required for a 3-trillion-parameter model using B200 GPUs?

Expert [00:17:16]: You said B200 or GB200?

Client [00:17:19]: Yes, using B200 or GB200 GPUs.

Expert [00:17:20]: B200. I think B200 at FP8, with sparsity, gives you about 9,000 teraflops. I think this would be about 50-ish days if you have 1 million of B200 and 1 million B200, each B200 consumes about… It's about 1,000 watts. I think that would be a gigawatt because 1,000 watts for 1 million B200, it would be about 50-60 days. Gigawatt for about two months.

Client [00:18:02]: Is that for two months using 1GW of B200 GPUs?

Expert [00:18:13]: Right.

Client [00:18:14]: Is this for a 3 trillion-size parameter model?

Expert [00:18:19]: This is for a 10 trillion parameter model. I typically would do all of this with a spreadsheet, so which is why I was doing approximate calculations now.

00:18:38] Regarding external inference, you would have 2GW, or 2-2.5GW, by the end of the year. Would that essentially support first-party inferencing hosted by Anthropic to serve its customers?

Expert [00:19:08]: Yes. Again, for OpenAI and Anthropic, just to make sure that I'm not speaking specifically to Anthropic, typically, all of that compute is for first-party. In OpenAI's case, we have a good chunk of compute directly available to them as well with all the Stargate builds that they have. Also this Cerebra capacity coming online for fast inference, etc. I would say, yes, I think about 3-4GW or maybe 5GW-ish by the end of the year.

Client [00:19:41]: From an ROI perspective, how much revenue supports the planned external inference capacity? Assuming Anthropic reaches USD 120bn ARR by the end of this year, with roughly 30% hosted first-party and the rest supported by hyperscaler clouds, do these assumptions seem reasonable?

Expert [00:20:31]: No, they're not too off. I think in general, that is aligned.

Client [00:20:36]: Does that leave USD 40bn ARR as external revenue requiring your external inference capacity?

Expert [00:20:52]: Yes and no. I think, yes, because typically, that is only a part. Then, I would say no because there are other revenue streams as well, like licensing costs and in general, outcome-based pricing, also subscription models and even for enterprise use cases beyond just the token usage. Typically, again, based on public reports, the ARR for Anthropic is already much more than that so far in the year. I think it's typically much more than that.

Client [00:21:33]: Are you saying the USD 120bn ARR by end of the year is too small?

Expert [00:21:42]: No, no. No, I think if you are projecting USD 120bn, I think that is directionally completely in line.

00:28:40] Looking ahead to 2027, will the distribution across the four categories, which are research, training runs, internal inference, and external inference, remain similar or change significantly?

Expert [00:29:04]: I think it will likely continue being the same, more importantly because they are extremely fungible, and secondly, I think that is where the industry is heading towards directionally.

Client [00:29:29]: If that ratio holds, external inference would account for 4-5GW, correct?

Expert [00:29:47]: Right away or how are you thinking about it?

Client [00:29:51]: That would be by the end of 2027, with 10-11GW in total.

Expert [00:29:53]: By '27, yes.

Client [00:30:03]: At 40%, that allocation would provide roughly 4GW for external inference.

Expert [00:30:13]: Yes.

00:30:14] Is there sufficient monetisation in place to support that capacity? That would essentially require USD 200bn in first-party ARR, according to the calculation.

Expert [00:30:37]: Since end of 2027?

Client [00:30:38]: Yes.

Expert [00:30:39]: Obviously, I cannot comment on forward-looking statements. For typically about 5GW of compute, I think that is very likely. Then, obviously, this is just token cost. Then, you have to add on top of it licensing costs. For example, all the healthcare and life sciences deals, they often have a lot of additional premiums as well. Secondly, OpenAI and Anthropic, etc, they also have these Forward Deployed Engineering companies or FDEs that work with the consulting agencies. A lot of this can help with integration and then digital transformation. My point is there are a lot of other revenue streams as well. Then, if you consider OpenAI, like ads, it might not be a massive number now, but I think by end of next year, it could bring in a lot more revenue as well. Then, in the future, also web hosting or app hosting, etc, I think all of that by 2027 could also be additional revenue streams. My point is, I think beyond just tokens, I think these would be significant multipliers as well. I don't know if you're accounting for all of these, but it's really hard to quantify them.

00:32:04] Could you restate your point regarding the factors we should consider and incorporate?

Expert [00:32:14]: Absolutely. Firstly, it is just outcome-based pricing that these frontier labs are also working with for very specific industries like life sciences, legal, etc. I think that is one part. Obviously, this goes beyond just token usage. The second one after that is also the whole FDE or Forward Deployed Engineering notion that these companies are working with, for example, Deloitte, or McKinsey, and all of this. Then, they all have their own FDE arms as well. That's the second one. Thirdly, hopefully, by 2027, these companies are also able to do web hosting and drive other revenue for enterprise users beyond just token usage. Then fourthly, I'm forgetting what was the fourth point I had in mind. I just mentioned. I'm blanking out. I think there was a fourth one, I can't remember right away. My point is, yes, you can have these four additional streams as well that are very, very likely, especially when you're looking at end of 2027.

00:33:57] Why was the agreement made with SpaceX, given the seemingly high premium?

Expert [00:34:06]: Great point. A few things. Firstly, I think frontier labs in general, they are very focused on just allocating more and more compute to research and pre-training rather than just purely increasing ARR by allocating to inference. The first part is just because more training can do everything. It's like buying time. If you could buy time, you would. Firstly, it is just to allocate more and more to training and research, and this is where GPUs are great at. Secondly, it's because you fundamentally just can't get so much capacity instantly available. You're paying for the short time horizon or the time to value, which is why like, for example, when Anthropic they made this deal, they publicly announced that they were increasing all Claude Code limits, etc. All of that is possible with this added capacity. The third point is just that getting contiguous capacity is extremely, extremely challenging. If you want so many GPUs, it's very hard to get because neoclouds don't have these massive allocations either. All the neoclouds are tightly controlled by Nvidia and Nvidia ensures that no one neocloud gets a massive amount.

Let's say, if you're OpenAI and you really want about 300MW of compute, you're not going to get it from hyperscalers because Nvidia also doesn't give so many GPUs to hyperscalers. You're not going to get it from neoclouds. This was the only one that was available. Obviously, on the other hand, SpaceX or xAI wasn't using it either. My point is zooming out, both Anthropic and Google, they really opportunistically use that excess compute. At the same time, xAI is happy to generate revenue and be like this quote, unquote, the neocloud provider for the time being for this. I think it's definitely like a win-win for all parties over here, especially in this extremely compute-constrained world.

Client [00:36:22]: Margin opportunities appear minimal when leasing compute at such expensive rates.

Expert [00:36:33]: Right. That's the whole point. It's not just margin in terms of generating ARR. This is fundamentally to be able to train models faster. Hypothetically, let's say, that if you are able to allocate more and more to training, let's say, that you do a bunch of design experiments, you can train a whole new model, which behaves, let's say, hypothetically just exactly like GPT-5.6 Sol. It is extremely efficient and let's say, it only consumes 20% of the tokens. My point is, in this case, you can easily just swap it out, let's say, OpenAI really trains a new model. From a capability perspective, it is just exactly like 5.6, but it's extremely efficient, let's say, only 20% or 30% of the compute footprint, you can swap it out and you can have the rest of the capacity available for either more users or more training, etc. My point is it's not so much about how much premium are you paying based on the ARR or the inference cost. It's more about what can training unlock for you. Then this is where I mentioned the research part is fundamentally more critical because, let's say, during research, if you already have enough unlocks either algorithmic unlocks or safety unlocks or even interpretability unlocks, etc, you would really benefit as a company from just going all into training and finding out just extremely innovative way because zooming out a lot of problems that exist right now, they are fundamentally because the world doesn't have enough compute.

Client [00:38:04]: Is ROI evaluated differently rather than focusing strictly on monetising above expensive compute costs?

Expert [00:38:17]: Exactly. Right. Again, these companies are obsessed with AGI, recursive self-improvement, more importantly, safety, security. More importantly, even the impact to humanity and accelerating AI researchers so that that can help with everything from drug discovery to so many other elements. My point is, all these are much more important for these frontier labs, all of them, rather than just purely ARR or inference. Which is why this is, I would say, it's a premium to pay, but I think it's a premium to pay for the right reason. At the same time, these companies are held in very confident perspective by investors and by customers. My point is you can take this cost and you can either pass it on to customers because customers are willing to pay higher for more efficiency and more intelligence. For example, fast inference or just more intelligent models, etc, or you can pass it on to investors because investors also fundamentally believe in the mission of these companies. My point is, yes, it seems like an overspend. When you look at compared to how much money has gone in and how much they can continue to raise even in these rounds, etc, it is not that big.

Let's say, even if we rent these two clusters out at USD 15bn per year, even for two years, let's say, if you rent it out at even USD 30bn, as an example. The previous math that we did was, let's say, about 50-ish days at B200 one million, divided by three for 300,000 chips, which is multiplied. It's like, let's say, 180 days. My point is that every year, you can train two completely new models that are 10 trillion parameters. In two years, you can come out with four models, which are 10 trillion parameters, hypothetically, again, using your numbers, which is a massive unlock. Let's say, imagine in two years, you have models which are insanely more capable than Mythos, GPT-5.6 or anything. My point is USD 30bn seems nothing if you can churn out four of these 10 trillion parameter models in two years. This is why I don't think it's a massive investment. It's a massive investment in absolute numbers, but fundamentally from relative to what the gains are, it is much lower because those 10 trillion parameter models, we calculated, they can generate much more, much more in the future.

00:41:16] Considering the current compute shortage, where spot market prices for B-series and H-series GPUs are inflating, how long is this crunch sustainable? Will the release of Vera Rubin and expanding neocloud leasing efforts restore market equilibrium?

Expert [00:42:03]: I would anticipate, yes, given a few things. When you look at the demand as well, it is significant and it will continue to be so, I would say, at least for the next 4-5 years. Even on the supply side, I think we will be in a compute crunch for at least about, say, 4-5 years. That is my thesis. I don't see this whole supply-demand elasticity smoothening out or reducing any time until the next 4-5 years. My point is, yes, even Nvidia is extremely, extremely positive and confident about this demand. Then this is also evident, even Nvidia, they charges 80% margins. They can easily lower their margins given the threats that they're seeing from hyperscaler AI ASICs and OpenAI and Anthropic building their own silicon, but they're not. Anthropic is constantly maintaining its margin. They're still growing past their quarterly both revenue and margins. On the other hand, they're also improving the financial economics by the whole USD 500bn backstop deal for GPU as an investable asset.

All of these are directional signals that, yes, the compute crunch is going to be here for the longest time. I don't really see any headwind immediately that would cause any such questionable slowdown. Obviously, there are likely or probable headwinds that can happen, which is like, A, if we don't see model intelligence improving like scaling laws, etc. Secondly, if there is obviously a geopolitical challenge with logic dies with Taiwan and then memory dies with Korea. Thirdly, obviously, if power and regulations become a big challenge in the US, I think that would be one. Then fourthly, I think, again, if safety and security become massive blockers or the government really… My third part was around data centre build-outs and power and utility, etc. The fourth one is just software and model restrictions where, let's say, the US government really imposes these. I would say other than these four, which are extremely less probable headwinds, there are no structural blockers for any of the compute demand and supply.

00:44:57] Given the proliferation of data centre and neocloud projects, is the industry overbuilding relative to future revenue? Additionally, how much is recursive self-improvement driving these large capacity agreements from Anthropic and OpenAI?

Expert [00:45:52]: Quite a few things. Fundamentally, the first part is compute is a bottleneck and then compute just solves a lot of the missions and the goals that these frontier labs have. Fundamentally, everything that is not done yet is because of not enough compute to do research, not enough compute to pre-train new models, not enough compute to serve inference at a lower cost, even existing inference. First of all, it is just more and more compute. Secondly, it's to have more sovereign compute rather than just relying on somebody else's roadmap. Even if you're relying on Nvidia's roadmap, this is how these labs started. Eventually, they started relying on hyperscalers' roadmap because for production inference, you can fundamentally use hyperscalers, which are way better than using Nvidia GPUs. Then eventually, instead of even reduced margins that they're paying to hyperscalers compared to the margins that they were paying for Nvidia, the next step is to pay even lower margins by in-house compute. The second part is just sovereign compute, which is, again, OpenAI's Jalapeño chip demonstrated that once you have completely hardware, software co-design, you can just get so much more efficiency. The third part is that a lot of these investments are long term. These investments are in securing the LPS, the Land, Power, Shell parts of the data centre.

You can keep swapping chips in and out. You will always have a demand. These are long term. When you go everything from just the model shelf life to the chip shelf life or the models to chips to racks to build-outs to energy utility, the models have a very short shelf life, but the power structure that you build, especially behind-the-meter, etc, they will likely be good for 10-12 years before you really need to tear them down because at that point, your spatial density in terms of power requirements will be so much. Vera Rubin already requires 300kW racks, which is already a big upgrade. My point is all of this build-out will amortise over at least the next 10 years. My point is that is still a good enough investment. Chips will amortise anywhere from 4-5 years. My point is, yes, in absolute terms, the build-out makes or looks or appears extremely big. When you amortise it across, first of all, how much compute is needed, it is not that much. Secondly, how long you can amortise and run this? Because a lot of spend that is happening right now in terms of compute is not immediate. A lot of these will continue to give returns over the next few decades, obviously, with different depreciation curves as well.

When you look at all of this in totality, and then lastly, when you just look at the penetration as well. Right now, AI, even when you look at enterprise, the amount of users to use AI in enterprise use cases is very low. Then secondly, even for them within enterprises and their workflows, the amount of penetration of workflow is also very low. I really imagine AI to be as ubiquitous as software is across an enterprise. We haven't even there yet. Then lastly, we haven't even unlocked completely different multi-modalities like image, audio, video, all of these or even spatial intelligence, which is physical AI. My point is that we haven't even scratched the surface. This is like we're in early days of internet where everything was text-based and it was bulletin boards. This is the current scenario we're living in. For example, we haven't even invented the Uber or the Netflix of AI, just how a dial-up internet was a thing. People were still counting minutes and megabytes used, just how we're counting tokens right now. We haven't even unlocked all these higher order products and revenue streams yet, which is why I think all this build-out, it definitely feels a lot, but I think it is quite justified in the perspective of the demand growth and then secondly, also the multiplier in terms of the revenue I think.

00:50:43] Could you explain the strategic dynamic between neoclouds and AI laboratories? As laboratories shift towards self-hosting for cost efficiency, will the long-term value proposition of neoclouds deteriorate?

Expert [00:51:29]: Good point. Honestly, I don't think any major frontier lab right now serves production inference on neoclouds. Neoclouds just don't have enough contiguous capacity. They don't have the backhaul like fibre network that is needed to serve inference at scale. Then thirdly, they don't even have enough security, reliability, uptime, availability, service personnel, backup power, etc, to run production inference. Just to clarify, I think neoclouds are great when you're running isolated, let's say, training clusters and run when you can do check-pointing and all of that. Fundamentally, if you're looking at just pure production inference, it's extremely challenging to run them on neoclouds. Now coming to your second point on how does the sovereign or the self-build-out look like. First of all, I think even in the next few years, I think frontier labs will continue to use or piggyback on hyperscalers for most of their capacity. I think hyperscalers will still continue to have a good edge and a good chunk. I don't see frontier labs just completely replacing neoclouds or hyperscalers. For neocloud specifically, I think the value proposition has always been around two things. One is open source models from China. Then secondly, it has been around enterprises using fine-tuning, etc, to build their own models.

Now on the second part around like fine-tuning their own models, that has proven to be not so effective. Fine-tuning, not many people do that because of many reasons, and I can get into those details, but TL;DR. Fine-tuning isn't really a thing that is materially yielding different or better results compared to just using a very intelligent model and giving it a very large context window like one million tokens because that gives you better results and accuracy. Taking an intelligent model with a long context window and then using RAG, compared to taking a small model and fine-tuning it, and then deploying it and serving it. I can go deep into that space. For example, you cannot really take a Kimi K3, which is a 2.8 trillion parameter model. Let's say, if you have Pepsi or Mercedes or something and you want an enterprise model, you can't really just go and fine-tune that model or do reward modelling on it to make it whatever, like a Pepsi enterprise model or a Mercedes enterprise model. This was the first one. The second part was open source models. Which is why there's always this worry that the best open source model comes from China and then ironically, the US is suffocating China for chips, whereas China is helping the world with open source model.

What if that stops or that pauses? Which is why Nvidia has been pushing on so aggressively on their open source models like the Nemotrons and also in the US, there's Thinking Machines that shared their Inkling model. Google has Gemma 4 or Gemma 3, which is not great, but a lot of enterprises are using it only because they are a little sceptical of using Chinese open source models. First of all, the entire neocloud industry was created by Nvidia because they didn't want to just be a vendor and keep selling to hyperscalers only. They wanted a direct access for people to be able to use their GPUs. They created the entire hyperscaler ecosystem. They financed them. They gave them GPUs at a lower price, which is why when you ask anybody their GPU pricing, everyone gives you different numbers because it really depends on many, many other factors. Even right now, Nvidia has reportedly also building the fibre network to connect all the neoclouds together just so that they are not bottleneck. My point is the neocloud ecosystem will definitely be spun up by Nvidia and Nvidia is obviously fuelling it to keep that entire ecosystem alive by everything like connectivity, to financing, to open source models, etc. Quite a few reasons.

I think to address your question like going in the future about how the value proposition for neocloud, I think it will continue to be so because the value proposition for neoclouds is around a few different things. Firstly, it is access to the latest and greatest GPUs because this is what Nvidia does. This is why when the GB300 came along or some of these other ones came along, like CoreWeave and Nebius, they are the ones who typically get first access to them. If you want access to the latest and greatest GB300 or even in the future, Vera Rubin, etc, you will get them first on neoclouds before hyperscalers. This is how Nvidia very strategically controls their allocations. I think that's one part. Secondly, you get them for cheap. If you look at the GPU per hour pricing on neoclouds vs the same GPUs of hyperscalers, neoclouds are much cheaper. This is where Nvidia subsidises this. I think for these reasons, I think neoclouds will still have a value proposition because Nvidia will make sure that it keeps fuelling. This is why Nvidia is doing this whole backstop financing deal because a few things.

First of all, like I mentioned, they are not decreasing their margins. That's a clever move. At the same time, they know that neoclouds need a lot of investment in the sense it's not just buying GPU chips, but like I mentioned, Vera Rubin requires 200kW per rack, which means the power consumption to upgrades, to electrical, to cooling, to connectivity, etc. This is why Nvidia is helping GPUs be this investable asset so that they can get loans and they're also doing this backstop guarantee. My point is that, yes, I think neoclouds will continue to flourish as long as Nvidia exists because Nvidia will continue to do this. They're supplying everything.

Client [00:58:25]: From a laboratory perspective, is it definitive that major players will construct their own cloud infrastructure going forward?

Expert [00:58:42]: I think so, yes, because when you think of Google, it is the perfect example of being extremely vertically integrated. There are tremendous benefits. What frontier labs also want to do is prevent what happened in the traditional cloud ecosystem. For the longest time, there were many cloud providers, but over time, it got consolidated to just three, Amazon, Microsoft, Google. If you have to go and fundamentally rent a computer in the sky, you have to go to one of these three. The other challenge is with these three cloud providers, it is actually extremely sticky. Let's say, Google runs its entire business on AWS infrastructure and then Snapchat runs everything on GCP, it is almost extremely impossible for them to move their entire workloads from one to the other. This is a big, big challenge. This is also what the frontier labs are trying to prevent because, first of all, if you lock in customers much early on and you're able to give them significant value proposition, it just makes a lot of sense for these labs to be vertically integrated end-to-end.

In the future, let's imagine, if every website is an AI website natively, in the sense that it's not like the traditional design for humans, it's either for agents or it's for LLMs or even for humans, it's very dynamic. You would rather have ChatGPT or Anthropic, or OpenAI or Anthropic hosted because you get direct access, you get cheaper. Then rather than hosting it on some random server and then pointing it to these APIs. Zooming out, yes, it totally makes sense to be vertically integrated because eventually, you can imagine all the businesses will be hosted out here, all the websites, a good chunk of web traffic would also go to some of these. To me, it makes total sense to be completely vertically integrated.

Client [01:00:51]: Once self-built data centres are operational, will neocloud services no longer be required?

Expert [01:01:02]: Not necessarily. I think neoclouds, from a frontier lab perspective, they will continue to be just spot or on-demand capacity because instead of reserve capacity, you can get a lot more specific cluster. Let's say, you're a researcher, you have this great idea, you want to try it out instantly on the latest and greatest Vera Rubin architecture in the future because you can get to know within two days whether it works or it doesn't work. First of all, this is a great case for neocloud for frontier labs as well instead of waiting for hyperscalers to get GB or Vera Rubin or sign them, etc. Even if they get an older one, you're going to have to wait 20 days vs let's say, here, you can try it out in two days. You know the result much earlier. I think, first of all, for spot capacity is extremely helpful. Then secondly, even for just, let's say, international neoclouds would also be quite helpful for doing these long horizon training runs because, let's say, randomly, if you spin up a neocloud in Indonesia or something where they have cheap power, you can fundamentally just outsource all your training runs. You pay for data transfer cost, but you just pay so much less for power and you can use neoclouds wherever you want.

Then thirdly, I think neoclouds will continue to make a good hosting bed for enterprises as well. Obviously, enterprises don't really have neither the power nor the infrastructure nor the talent to manage a lot of these compute racks or AI compute racks in-house. They will continue to lean on neocloud. I think, yes, I don't imagine the business going away. Given the fact that you can only rent GPUs from neoclouds, that is also a bit of a limiting factor. There are two neoclouds, one is called Hot Aisle and the other one is called TensorWave, I think, that also allow for AMD chips to be rented out. I think such heterogeneous neoclouds will likely sustain. My point is if somebody is only using GPUs in the future, I think we'll see. My point is, I think it is not as lucrative to keep running these neoclouds.

Client [01:03:40]: Is the argument that self-built infrastructure will handle inference, while on-demand training and research will still rely on neoclouds?

Expert [01:04:01]: Not entirely. Even for pre-training, what I'm thinking is that for spot capacity for research runs, etc, this is where neoclouds are helpful or even for part of pre-training runs. At the same time, neoclouds are very opportunistic. My point is what if a neocloud randomly decides that they don't want to give you capacity. Does that mean OpenAI stops training their models? No. My point is it's not completely pre-training, but just a part of it. How do I put it? I think you can just think of neocloud as incremental capacity, etc, but not full-time capacity.

Client [01:04:42]: How are CPUs, GPUs, and Trainium chips deployed internally and what factors guide those decisions?

Expert [01:05:01]: Can you help me understand in which dimension because I can address this in quite a few dimensions?

01:05:11] Are the majority of training runs currently conducted on TPUs [tensor processing units]? Given that most new capacity relies on GPUs, is this due to TPU supply limits or GPU architectural and economic advantages? Setting aside investment bundles, which chip represents the most cost-effective solution across training and inference?

Expert [01:06:12]: Typically, across the frontier AI landscape, researchers and pre-training algorithms, they always prefer GPUs because it is known and tested. It is much more resilient and you don't have to worry about going and figuring out all the different challenges at the lower layers. Which is why typically for both of these, GPUs just make a ton of sense also because if you can get NVLink, which is all-to-all communications, etc, and that just makes overall training just so much more efficient. Let's say, in this case, if you were to use Ethernet or RoCE, etc, instead of InfiniBand, for scale-out domain, scale up is NVLink, but scale out, it would just slow down the overall training run so much more. My point is for all of these, you just prefer GPUs. For production inference, this is where TPUs and Trainiums are excellent because, first of all, a typical inference job doesn't require one million-plus GPUs or a few hundred thousand GPUs just like our training does, but it is more around, whatever, let's say, if it's Kimi K3 or something that sits on eight chips or 16 chips. You typically have a localised job.

For that, what we need is the ability to duplicate these across different racks and clusters. Then more importantly, what you need is end-to-end networking. If I'm using inference, the round trip or the connectivity from my laptop to that tray that is running my model and then that tray processes my prompt to convert them into output tokens and then send it back. What matters is not so much like cluster-to-cluster or rack-to-rack communication, but it's more around end-to-end latency to make inference a better experience. For all these reasons, TPUs and Trainiums, even architecturally, they are systolic arrays, so they are a lot more efficient in running inference rather than just training. Then secondly, the entire networking end-to-end chain for hyperscalers is just a lot more suitable for inference than it is just purely for training.

The training is more around just internal networking and then inference is more around end-to-end, which is what data centres are great at. There's also when neoclouds are better at training because when you just look at the back-end network between GPUs, it is great because you can just buy the standard NVLink, config with Spectrum-X switches with Nvidia DPUs, you can be a full Nvidia shop and you would get amazing performance for training. For inference, you have to be like this data centre that has everything from fibre optic connectivity from data centre to data centre, you need integration with public points of presence, the edge or the CDN network for distribution. You also need a lot of redundancy to reduce latency, routing, etc. For all these reasons, TPUs and Trainiums are just a lot more efficient for inference.

Client [01:09:25]: Are TPUs and Trainium used primarily for inference and GPUs for training?

Expert [01:09:30]: Typically, yes.

Client [01:09:30]: Was Opus 4.8 entirely pre-trained on TPU v5e?

Expert [01:09:42]: Again, unfortunately, I cannot comment on that. Net-net, yes, GPUs have gotten a lot better for training as well. This is what I was referring to. First of all, when you have all the research experiments, they are best done on GPUs because it's turnkey solutions. When you want to do a full-on pre-training run, this is when you can typically use GPUs or even TPUs are good enough. It really depends on what's the model size, how many you have available, how much time you want to spend on it, etc, all of these. Fundamentally, this is what your typical decision-making approach is.

Client [01:10:20]: Are GPUs best suited for experimental research, while modern TPUs perform effectively for large-scale training runs?

Expert [01:10:37]: Correct, correct.

01:10:46] What impact does RSI [recursive self-improvement] have on training and compute resource requirements?

Expert [01:10:54]: RSI, I think, it's just a way to get to AGI. Obviously, training is a process for RSI. Training is just one of the ingredients for recursive self-improvement. At the end, it's really about constantly running, not only training, the reinforcement learning, synthetic data generation, reward modelling, safety alignment, all of these in tandem. They're constantly running all of this together. My point is obviously, training is fundamentally mission-critical for RSI, but it's also all these other elements together.

Client [01:11:41]: Does RSI research demand greater GPU utilisation or can TPUs fulfill those workload requirements?

Expert [01:11:56]: Good point. I think it's a combination. I don't think it's just one because, like I mentioned, we have to orchestrate the entire model life cycle end-to-end, and you have to keep accelerating it. Inference is actually a big part of this as well. It's not just training, but it's also inference. Which is why if you can run inference on TPUs and Trainiums and you can run training and fine-tuning, etc, on GPUs, you fundamentally get to use both elements.

01:12:30] As RSI becomes more viable, is it altering how organisations invest in advance for training and research compute?

Expert [01:12:54]: I think so. That is why, like I mentioned, if I were to start my own frontier lab, I would definitely allocate more and more. I don't think traditionally, this was the case where so much was allocated to training only because there was a lot of uncertainty around all of this. Then more importantly, inference is just a way for these companies to monetise their model so that they can generate enough revenue, so that they can dedicate more to training. My point is that the inference ARR or revenue wasn't even so much where they could literally go so deep into going all into training.