Text pretraining corpora
8 entriesLarge text corpora for pretraining language models.
01Common CrawlA nonprofit that publishes a large open repository of web crawl data, the raw source for many pretraining corpora.
02C4 (Colossal Clean Crawled Corpus)A cleaned English web-text corpus derived from Common Crawl, originally built to train T5.
03DolmaAi2's open pretraining corpus, built to train the OLMo models and released with its tooling.
04FineWebHugging Face's large, cleaned and deduplicated English web dataset drawn from Common Crawl.
05FineWeb-2A multilingual extension of FineWeb covering many languages.
06RedPajama-Data-V2Together's web dataset released with quality signals so you can filter it yourself.
07DCLM-BaselineFiltered web data from the DataComp-LM project, used as a baseline for studying data curation.
08SmolLM-CorpusThe pretraining corpus for Hugging Face's SmolLM models, including synthetic textbooks (Cosmopedia) and educational web text.
Instruction, chat and preference data
11 entriesData for instruction tuning, chat and preference alignment.
09OpenAssistant Conversations (OASST1)Human-written, multilingual assistant conversations with quality rankings, crowdsourced by volunteers.
10Databricks Dolly 15kAbout 15,000 human-written instruction-following records released by Databricks employees under an open licence.
11Stanford Alpaca52K instruction-following examples generated with OpenAI models, used to fine-tune Alpaca. Check the terms of use before reuse.
12FLAN collectionGoogle's collection of instruction-tuning tasks and templates.
13OpenOrcaAn open reproduction of the Orca instruction data, built by augmenting FLAN tasks with model-written explanations.
14Tülu 3 SFT mixtureAi2's mixture of supervised fine-tuning data used to train the Tülu 3 models.
15UltraChatLarge-scale multi-turn dialogue data generated with language models.
16UltraFeedbackModel responses with ratings and feedback, used for RLHF and DPO-style preference training.
17Anthropic HH-RLHFHuman preference data on helpfulness and harmlessness, plus red-teaming examples.
18WildChat-1MAi2's collection of real user conversations with ChatGPT, collected with consent and anonymised.
19LMSYS-Chat-1MOne million real conversations with many LLMs, collected from Chatbot Arena and the Vicuna demo.
Code, maths and reasoning data
5 entriesCode, mathematics and reasoning datasets.
20The Stack v2BigCode's large collection of permissively licensed source code from Software Heritage, for training code models.
21StarCoderDataThe pretraining data behind StarCoder: code plus GitHub issues, commits and notebooks.
22GSM8KAbout 8,500 grade-school maths word problems with step-by-step solutions.
23MATH12,500 competition mathematics problems with full solutions.
24OpenThoughts-114kAn open reasoning dataset with long chain-of-thought traces across maths, code and science.
Image-text and video-text data
10 entriesPaired image, video and text data for multimodal models.
25InternVidA large-scale video-text dataset for multimodal understanding and generation, part of the InternVideo foundation model project.
26Flickr30k Image DatasetA standard benchmark dataset for sentence-based image description, widely used for training and evaluating image-captioning models.
27VQA DatasetA Visual Question Answering dataset pairing images with natural-language questions and answers, for training multimodal reasoning models.
28LLM Vision Datasets CollectionA curated collection of image and video datasets for generative AI and multimodal models, requiring at least images and corresponding captions.
29COCOCommon Objects in Context: a large image dataset with object annotations and captions.
30Conceptual CaptionsGoogle's image-caption pairs collected from the web.
31Re-LAION-2B-en-research-safeLAION's revised image-text dataset, re-released for research after safety checks.
32DataComp-1BImage-text pairs curated for the DataComp benchmark on data filtering for CLIP-style training.
33LLaVA-Instruct-150KVisual instruction-following data used to train the LLaVA models.
34The CauldronHugging Face's collection of dozens of vision-language datasets formatted for instruction tuning.
Speech, emotion and 3D data
5 entriesSpeech, emotion and 3D datasets.
35LibriSpeechAbout 1,000 hours of read English speech, a standard benchmark for speech recognition.
36Mozilla Common VoiceA crowdsourced, multilingual speech dataset released under an open licence.
37MuSE: Multimodal Stressed Emotion DatasetStudies the multimodal interplay between the presence of stress and expressions of affect, with annotated recordings and baseline models.
38Social-IQA question-answering benchmark for artificial social intelligence, testing a model's ability to reason about human social behavior.
39RGB-D Object DatasetA large-scale, multi-view RGB-D object dataset from the University of Washington, widely cited for 3D object recognition research.
Evaluation benchmarks and harnesses
4 entriesBenchmarks and tools for evaluating models.
40MMLUMultiple-choice questions across 57 subjects, a widely used test of broad knowledge.
41GPQAGraduate-level, expert-written science questions designed to be hard to look up. Access requires accepting terms.
42HELMStanford CRFM's holistic evaluation framework and leaderboards for language models.
43lm-evaluation-harnessEleutherAI's framework for running language models against many benchmarks.
Dataset hubs and discovery
4 entriesHubs and search engines for finding datasets.
44Hugging Face DatasetsThe largest hub of open datasets for machine learning, with a library for loading them.
45Kaggle DatasetsCommunity-shared datasets with notebooks and discussion.
46Google Dataset SearchA search engine for datasets published across the web.
47Data Provenance InitiativeAudits the licences and provenance of datasets used to train language models.
Related Pages
6 entriesRelated directories on this site.
Frequently Asked Questions
What datasets are used to pretrain language models?
Large cleaned web corpora such as Common Crawl derivatives, C4, FineWeb, Dolma and RedPajama, often mixed with code and curated text.
What is instruction tuning data?
Prompt and response pairs, often human-written or model-generated, used to teach a pretrained model to follow instructions; preference data additionally ranks responses for alignment.
How do I check a dataset's licence?
Read the dataset card and terms; the Data Provenance Initiative audits licences and provenance of many collections, and some data is restricted for commercial use.
Which benchmarks evaluate language models?
MMLU, GPQA and HELM cover knowledge and reasoning, and EleutherAI's lm-evaluation-harness runs many benchmarks consistently.
Where do I find more datasets?
Hugging Face Datasets, Kaggle and Google Dataset Search, listed in the hubs section, plus the Data Sets for ML/DL page.