Dylan Patel, the chief analyst at SemiAnalysis, a semiconductor research firm, has reportedly stated that utilizing ChatGPT for tasks such as writing cover letters, generating lesson plans, or revising dating profiles can lead to a cost of up to $700,000 per day for OpenAI due to the high-priced tech infrastructure necessary for the AI's operation. ChatGPT's reliance on substantial computing power to formulate responses based on user inputs is the primary reason for this high cost. "The majority of the expenses incurred relate to the costly servers they need," Patel informed the tech publication.
According to Dylan Patel, in a phone conversation with Insider, the current cost of operating ChatGPT is presumably higher than his initial calculation. The original estimate was based on OpenAI's GPT-3 model, and the latest model, GPT-4, would likely result in even greater expenses, Patel informed Insider. Insider reached out to OpenAI for a comment before publication, but they did not respond immediately.
According to SemiAnalysis analysts Dylan Patel and Afzal Ahmad, while the training of ChatGPT's expansive language models may cost tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale." "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they stated to Forbes.
For years, businesses using OpenAI's language models have been paying exorbitant prices. Nick Walton, CEO of Latitude, a start-up that created an AI dungeon game utilizing prompts to produce storylines, disclosed to CNBC that in 2021, the model's operation, as well as payments for Amazon Web Services servers, cost the company $200,000 monthly to answer millions of user queries.
Walton disclosed that he switched to a language software provider supported by AI21 Labs due to the high expense, which resulted in his firm's AI expenditures decreasing by 50%, to $100,000 each month. "We used to joke that we had human staff and AI staff, and we spent nearly the same amount on each of them," Walton informed CNBC. "We spent hundreds of thousands of dollars monthly on AI, and we are not a large startup, so the cost was enormous."
Microsoft allegedly working on a clandestine chip
Microsoft is reportedly working on a new AI chip named Athena, with the aim of lowering the operational cost of generative AI models, as initially disclosed by The Information. This project was launched in 2019, several years after Microsoft and OpenAI struck a $1 billion deal, requiring OpenAI to operate its models solely on Microsoft's Azure cloud servers.
According to The Information, Microsoft had two objectives in mind when developing the Athena chip. Firstly, Microsoft executives became aware that they were lagging behind Amazon and Google in their efforts to create their own in-house chips, a source knowledgeable about the situation revealed. Secondly, Microsoft was seeking less expensive alternatives to Nvidia's graphics processing units (GPUs), which were used to run their AI models, and thus decided to create a more affordable chip.
As per the report, over 300 Microsoft workers are now working on the chip, almost four years after the project began. The Athena chip could be available for Microsoft and OpenAI's internal usage as soon as next year, according to two sources familiar with the situation.
Insider contacted Microsoft for a response, but the company declined to comment.