The Large Language Models (LLM) have a well-kept secret: their development and functioning require large amounts of energy. Furthermore, the true extent of these models’ carbon footprint remains a mystery. The start-up Hugging Face believes it has found a way to calculate this footprint more accurately, estimating the emissions produced during the entire lifecycle of the model and not just during its development.
The attempt could be a step towards getting more realistic data from tech companies about the carbon footprint of their artificial intelligence (AI) products at a time when experts are urging the industry to better assess the AI impact environment. Hugging Face’s work is published in a paper that has not yet been peer-reviewed.
To test its new approach, Hugging Face evaluated the global emissions of its own language model called BLOOM. The latter was released at the beginning of the year. This process involved adding many numbers together: the amount of energy used to train the model on a supercomputer, the energy required to manufacture the supercomputer’s hardware and maintain its computing infrastructure, and the energy used to make BLOOM work once it was deployed. The researchers calculated this last part using a software tool called CodeCarbon. This monitored the carbon emissions produced by BLOOM in real-time over an 18-day period.
Hugging Face estimated that the development of BLOOM caused the emission of 25 tons of carbon. But the researchers found that number doubled when they took into account the emissions produced by manufacturing the computer hardware used for the development of this LLM, the broader computing infrastructure, as well as the energy required to run BLOOM once the phase of development is complete.
Pressure, exhaustion… The search for a responsible AI is done at the expense of experts
Helping the AI community get a better idea of its impact on the environment
While this number may seem high for a single model – 50 metric tons of carbon emissions or the equivalent of approximately 60 flights from London to New York – it is significantly lower than the emissions associated with other LLMs of the same size. That’s because BLOOM was developed by a French supercomputer powered primarily by nuclear energy that produces no carbon emissions. Models formed in China, Australia or parts of the United States, whose power grids rely more on fossil fuels, are likely to be more polluting.
After the BLOOM was released, Hugging Face estimated that using the model emits around 40 pounds of carbon dioxide every day. A number similar to the emissions produced by an average new car with just over 85 kilometers.
For comparison, OpenAI’s GPT-3 and Meta’s OPT are estimated to emit over 500 and 75 metric tons of carbon dioxide, respectively, during development. The extent of GPT-3 emissions can be partially explained by the fact that it was trained on older, less efficient hardware. But it’s hard to say for sure about these numbers. There is no standard way of measuring carbon emissions and these numbers are based on external estimates or, in the case of Meta, limited data published by the company.
“Our goal was to go beyond the carbon emissions of electricity consumed during development and consider more of the lifecycle to help the AI community get a better idea of its impact on the environment and how we can start to reduce that impact. ,” says Sasha Luccioni, a researcher at Hugging Face and lead author of the article.
The carbon footprint of language models
The Hugging Face paper sets a new standard for organizations developing AI models, says Emma Strubell, an assistant professor at Carnegie Mellon University’s school of computer science, who in 2019 wrote a seminal paper on AI’s impact on climate. Emma Strubell did not participate in this new survey.
This document “represents the most in-depth, honest, and well-researched analysis of the carbon footprint of a large machine learning (ML) model to date, as far as I know, going into much more detail… than any other article [ou] report that I know”, emphasizes Emma Strubell.
According to Lynn Kaack, assistant professor of computer science and public policy at the Hertie School in Berlin, who was not involved in the work on Hugging Face, the paper also sheds much-needed light on the extent of the carbon footprint of large computer models. language . She says she was surprised to see the magnitude of the lifecycle emissions numbers, but feels there is still much to be done to understand the environmental impact of large language models in the real world.
“We need to better understand the much more complex downstream effects of AI use and abuse… It’s much harder to estimate. That’s why this part is often overlooked,” details Lynn Kaack. The latter co-authored a paper published last summer in the journal Nature that proposed a way to measure chain emissions caused by AI systems.
The technology sector is responsible for around 2% to 4% of global greenhouse gas emissions
For example, recommendation and advertising algorithms are often used in advertising, which in turn induce people to consume and buy more stuff, resulting in more carbon emissions. According to Lynn Kaack, it is also important to understand how artificial intelligence models are employed. Many companies, such as Google and Meta, use AI models to rank user comments or recommend content to users. Taken individually, these actions consume very little energy, but because they are performed a billion times a day, they add up and emissions increase.
It is estimated that the technology sector as a whole is responsible for 1.8% to 3.9% of global greenhouse gas emissions. While AI and machine learning only account for a fraction of these emissions, AI’s carbon footprint is still too high for a single technology sector.
Climate change: Bill Gates’ venture capital fund is still investing in green technologies
Thanks to a better understanding of the amount of energy consumed by AI systems, companies and developers can make choices about the trade-offs they are willing to make between pollution and generated costs, analyzes Sasha Luccioni.
A “Warning Signal” for Big Tech Groups
The authors of the Hugging Face paper hope companies and researchers can think about how they can develop great language models while limiting their carbon footprint, says Sylvain Viguier, co-author of the paper in question and director of applications at Graphcore, a semiconductor company.
It could also encourage people to adopt more efficient AI research methods, for example by refining existing models rather than forcing even bigger models to be created, adds Sasha Luccioni.
The article’s findings constitute “a wake-up call for people who use this type of model, that is, most often people who work for large technology companies,” says David Rolnick, assistant professor at McGill University’s School of Informatics and the Institute of Artificial Intelligence of Quebec (MILA). With Lynn Kaack, he is one of the co-authors of the article published in Nature, but he was not involved in the work on Hugging Face.
“AI impacts are not inevitable. They result from our choices about how we use these algorithms and the choice of algorithms used,” concludes David Rolnick.
Article by Melissa Heikkilä, translated from English by Kozi Pastakia.
Meet Alex Hanna, the researcher who left Google to save AI
Receive our latest news
Every day, the selection of main information of the day.