12 training datasets. 5 need a licence check.

Every AI model is built on data someone else collected. When you fine-tune a model, train your own or buy a tool that did, the dataset's licence, coverage and gaps become your licence, coverage and gaps. The Good AI Prism now profiles 12 of the most widely used public training datasets, fact by fact. Five of them carry a condition that matters before product use.

The web is the base layer, and it is a sample

Most text datasets start from Common Crawl, a non-profit archive of more than 300 billion web pages, with 2.3 to 2.7 billion pages added per monthly crawl, according to the AWS Open Data account. Its own FAQ is precise about what that is: a sample of the web, not a complete copy, collected by a crawler that obeys robots.txt. Copyright in the pages stays with their owners.

Curated corpora filter that base. FineWeb from Hugging Face distils 96 crawl snapshots into 15 trillion English tokens under ODC-By 1.0. FineWeb-Edu goes further: an educational-quality classifier keeps only the best-scoring pages, and at its stricter threshold it removes 92% of FineWeb. RedPajama-Data-v2 takes the other approach. It ships 30 trillion deduplicated tokens in five languages, English, French, Spanish, German and Italian, plus more than 40 quality signals, and leaves the filtering to you.

92%. The share of FineWeb that FineWeb-Edu's stricter quality threshold removes. Most of the web is not what you want a model to learn from.

The lesson for buyers is that "trained on the web" says little. Ask which crawl, which filter, which language filter and which deduplication. FineWeb and Dolma are English only. If your users write in German, that is a coverage fact, not a detail.

Dolma from Ai2 mixes 15 sources, from web pages and scientific papers to code, books and Wikipedia, into about 2.3 trillion tokens under ODC-BY. C4, the corpus behind many early models, is 305 GB of cleaned English web text, with a multilingual sibling, mC4, in more than 100 languages. Both licences also bind you to the terms of their sources.

Code and maths come with their own rules

The Stack v2 holds 67.5 TB of source code in more than 600 programming languages. Each file keeps its original licence, so use must follow those licences, with attribution where required. About 40% of permissive files were duplicates before deduplication, and the dataset may contain e-mail addresses or keys that were once public on GitHub. An opt-out tool and removal requests exist. OpenWebMath adds 14.7 billion tokens of mathematical web text with LaTeX.

Images and speech: read the terms twice

Image datasets show the widest gap between fame and permission. ImageNet, the benchmark classic with 1.28 million training images, is available under an access agreement "only for non-commercial research and educational purposes". LAION-5B lists 5.85 billion image-text links; LAION states that it does not recommend the set for ready-to-go industrial products and that the images stay under their owners' copyright. Its Re-LAION-5B release of 30 August 2024 removed links matched to abuse hash lists and privacy requests.

Speech looks different. Mozilla Common Voice release 22.0 holds 33,815 hours in 137 languages. Mozilla has released the data as CC0, free of copyright restrictions; confirm the terms for the release you use. Wikipedia text is available under CC BY-SA, which allows commercial use with attribution and share-alike duties.

What changes with the EU AI Act

Under the AI Act as amended in 2026, providers of general-purpose AI models must publish a summary of the content used for training and maintain a copyright policy. For buyers, that turns dataset questions into documents you can request. Ask for that summary explicitly, and for the date it was last updated. A provider who cannot say which corpora it used, under which terms, has given you an answer too.

Five questions before you use a dataset

1. What does the licence allow for your use: research, commercial product, or redistribution?
2. Whose rights sit underneath? Crawled pages, code files and linked images keep their owners' rights.
3. Which languages and time span does it cover, and does that match your users?
4. How was it filtered and deduplicated, and what did that remove?
5. How are personal data and removal requests handled, and can you document it?

The dataset cards answer these where publishers do, each with source and read date. Data platforms and vector databases, the software that stores and serves data, sit in the Data & Analytics area as product cards. The models trained on these corpora are on the Models hub, and the AI Models brief explains what to ask a model provider about its training data.