Anthropic CEO Dario Amodei stated within the In Good Company podcast that AI models in improvement at this time can cost up to $1 billion to train. Current models like ChatGPT-4o solely cost about $100 million, however he expects the cost of coaching these models to go up to $10 and even $100 billion in as little as three years from now.
“Proper now, 100 million. There are models in coaching at this time that are extra like a billion.” Amodei additionally added, “I believe if we go to ten or 100 billion, and I believe that will occur in 2025, 2026, possibly 2027, and the algorithmic enhancements proceed a tempo, and the chip enhancements proceed a tempo, then I believe there’s in my thoughts probability that by that time we’ll have the opportunity to get models that are higher than most people at most issues.”
The Anthropic CEO talked about these numbers when he mentioned the event of AI from generative synthetic intelligence (like ChatGPT) to synthetic normal intelligence (AGI). He stated that there would not be a single level the place we immediately attain AGI. As an alternative, it might be a gradual improvement the place models construct upon the developments of previous models, very similar to how a human youngster learns.
So, if AI models develop ten occasions extra highly effective annually, we are able to rationally anticipate the {hardware} required to train them to be at the very least ten occasions extra highly effective, too. As such, {hardware} may very well be the most important cost driver in AI coaching. Again in 2023, it was reported that ChatGPT would require more than 30,000 GPUs, with Sam Altman confirming that ChatGPT-4 cost $100 million to train.
Final yr, over 3.8 million GPUs had been delivered to information facilities. With Nvidia’s newest B200 AI chip costing round $30,000 to $40,000, we are able to surmise that Dario’s billion-dollar estimate is on monitor for 2024. If developments in mannequin/quantization analysis develop on the current exponential charge, then we anticipate {hardware} necessities to maintain tempo until extra environment friendly applied sciences just like the Sohu AI chip turn into extra prevalent.
We are able to already see this exponential development taking place. Elon Musk wants to purchase 300,000 B200 AI chips, whereas OpenAI and Microsoft are reportedly planning a $100 billion AI data center. With all this demand, we might see GPU information middle deliveries subsequent yr balloon to 38 million if Nvidia and different suppliers can sustain with the market.
Nevertheless, except for the availability of the particular chip {hardware}, these AI corporations want to be involved with energy provide and associated infrastructure, too. The full estimated power consumption of all data center GPUs bought simply final yr might energy 1.3 million properties. If the information middle energy necessities proceed to develop exponentially, then it is attainable that we might run out of sufficient economically-priced electrical energy. Moreover, whereas these information facilities want energy vegetation, additionally they want a wholly upgraded grid that can deal with all of the electrons the power-hungry AI chips want to run. Because of this, many tech corporations, together with Microsoft, are now considering modular nuclear power for their data centers.
Synthetic intelligence is shortly gathering steam, and {hardware} improvements appear to be maintaining. So, Anthropic’s $100 billion estimate appears to be on monitor, particularly if producers like Nvidia, AMD, and Intel can ship. Nevertheless, as our AI applied sciences carry out exponentially higher each new era, one massive query nonetheless stays: how will it have an effect on the way forward for our society?