The economics of a Neolab.
A neolab is loosely defined as a startup of AI researchers who raises a lot of money pre-production to be able to finance GPU compute to take on a large AI problem.
To buy 1000 GB300s or ~14 NVL72 racks will set you back $125-150M for 3yrs with 15-30% upfront. That’s about ~2-2.5MW.
Thats about enough to do 10^25 flops a quarter and get to a GPT-4 level model which is 1-2 OOMs off frontier for pretraining.
If you post-train on a great open source model, you have a better chance of getting to frontier. The risks are a) you need to spend millions on RL environments too and b) being lapped by another model release while being tied to a base model.
For this to payback, you need to give your customers a better and ideally cheaper inference service than a base model and serve them for long enough to recoup your large investment. Even at 50% margin on inference, to recoup $10M in training means serving ~10T tokens (!) if you price like Fable / Astra given a standard cache read / input / output split ($2/M blended). And you have to justify being better than a release like Opus 5.5 which is even cheaper. Often, you end up charging your customers a huge premium in terms of platform fees and compute fees on top of pure inference.
Meanwhile, every hour you’re not utilizing your GPUs you are burning money so you typically resell this compute back to a broker or run inference for open models / resell spot instances. At below a ~60% utilization on spot, you will still lose money.
Add to that insane cost of talent.
So what can you do with the compute?
- Not play the model game at all.
- Play an entirely different model game (Jev, World Labs) that if big labs played, would either a) cannibalize their business or b) be incrementally not significant revenue c) would cause too much distraction from the main main thing
- Acquire a proprietary data set (Peridodic Labs) in enough volume in a domain of usefulness to eclipse frontier quality. Often happens in robotics, biology, chemistry.
If you do overcome the challenge of building a model that is useful and well priced beyond big labs models, given the huge price of compute, you still need to play in an area where the revenue / compute ratio is signficant and market demand is large enough to payback your compute spend.
It is a difficult game.
Conversation
Do you think that as big labs become capture more and more compute, we’ll see less neolabs?
Not bc of lack of finance but lack of available compute
Very interesting but you still don’t say how they can make money in the long run. The AI business is money-based business; you pay to win and that’s it. Neolabs will struggle when AI winter comes or when Meta and Google sell AI for a fraction of the price.
Love this content. Somewhat naive to the space - if you buy the chips, I’m assuming you’re generally responsible for standing up the power those chips need (2MW in this case)? Isn’t that quite expensive in itself?
Your base assumption is incorrect.
A Neolab is not going to buy compute, it's going to lease it.
The biggest financier of neolabs is going to remain Semi companies. Today majority of the that comes from Nvidia, but others will fund more of it as they increase their revenue.
very difficult game that is only getting more difficult. the other trend we're seeing is neolab margins are getting eaten from both directions - OEMs charging more for equipment (BOMs +80% in last 12 months), customers deploying their own compute
You ignore that the price of those GPUs will 5x in 2 years. That startup made money already without even doing anything.
considering all the difficulties, if a team still goes for it and has the experience, i would expect a significant belief and confidence in what they are doing.
What about working on a specific niche? Let's say you don't want to train a large language model, but you want to make it specific for a task. For example, we see that in cybersecurity, when you want to train a model that is good on several tasks only. Would that make sense?
AI research must be government funded. It's as significant as the Internet if not more. It's insane that we're subjecting AI research to capitalist math.