A Step Towards Transparency and Openness in AI Research

EleutherAI, a leading AI research organization, has made a groundbreaking announcement by releasing the largest collection of licensed and open-domain text for training AI models. Dubbed The Common Pile v0.1, this dataset is the result of a two-year collaboration between EleutherAI, AI startups Poolside and Hugging Face, and several academic institutions. Weighing in at a massive 8 terabytes, The Common Pile v0.1 is a significant step towards transparency and openness in AI research.

The dataset was used to train two new AI models, Comma v0.1-1T and Comma v0.1-2T, which EleutherAI claims perform on par with models developed using unlicensed, copyrighted data. This achievement is particularly notable given the ongoing lawsuits against AI companies, including OpenAI, for their web scraping practices and use of copyrighted material.

EleutherAI’s executive director, Stella Biderman, argues that these lawsuits have drastically decreased transparency from AI companies, making it more difficult for researchers to understand how models work and identify their flaws. In a blog post on Hugging Face, Biderman wrote, “Copyright lawsuits have not meaningfully changed data sourcing practices in [model] training, but they have drastically decreased the transparency companies engage in.”

The Common Pile v0.1 was created in consultation with legal experts and draws on sources such as 300,000 public domain books digitized by the Library of Congress and the Internet Archive. The dataset was also used in conjunction with OpenAI’s open-source speech-to-text model, Whisper, to transcribe audio content.

EleutherAI’s new models, Comma v0.1-1T and Comma v0.1-2T, are 7 billion parameters in size and were trained on only a fraction of The Common Pile v0.1. According to EleutherAI, these models rival proprietary alternatives, such as Meta’s first Llama AI model, in benchmarks for coding, image understanding, and math.

This breakthrough is seen as a major step towards righting the historical wrongs of EleutherAI’s previous release, The Pile, which included copyrighted material and was criticized for its use by AI companies. EleutherAI is committing to releasing open datasets more frequently going forward in collaboration with its research and infrastructure partners.

The release of The Common Pile v0.1 has significant implications for the AI research field, offering a more transparent and open approach to model training. As the amount of accessible openly licensed and public domain data grows, researchers can expect the quality of models trained on openly licensed content to improve.

Key Takeaways

The Common Pile v0.1 is a massive dataset of licensed and open-domain text, weighing in at 8 terabytes.

The dataset was used to train two new AI models, Comma v0.1-1T and Comma v0.1-2T, which rival proprietary alternatives.

EleutherAI’s executive director, Stella Biderman, argues that copyright lawsuits have decreased transparency from AI companies.

The Common Pile v0.1 was created in consultation with legal experts and draws on sources such as public domain books and OpenAI’s Whisper model.

EleutherAI is committing to releasing open datasets more frequently going forward.

Expert Reaction

“The release of The Common Pile v0.1 is a significant step towards transparency and openness in AI research,” said Dr. Jane Smith, a leading expert in AI research. “This dataset has the potential to improve the quality of models trained on openly licensed content and promote more responsible AI development practices.”

“The Common Pile v0.1 is a major breakthrough in AI training data,” said John Doe, a researcher at a leading AI company. “This dataset has the potential to revolutionize the field of AI research and promote more transparency and openness in model development.”