dclm-baseline-1.0 mlfoundations
DCLM-baseline DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks. Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime. Model Params Tokens Open dataset? CORE MMLU EXTENDED Open weights, closed datasets Llama2 7B 2T ✗ 49.2 45.8 34.1 DeepSeek 7B 2T ✗ 50.7 48.5 35.3 Mistral-0.3 7B ? ✗ 57.0 62.7 45.1 QWEN-2 7B ? ✗ 57.5 71.9 50.5 Llama3 8B 15T ✗… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0.
- 类型
- dataset
- 许可
- cc-by-4.0
- 下载量
- 292,863
- 点赞
- 304
- 访问
- public
- 文件
- 0
标签
- 预训练语料
- 文本数据集
- 大语言模型
- 数据处理
- 机器学习
摘要
DCLM-baseline 是由 DataComp-LM 团队构建的约4万亿 token、30亿文档的英文预训练数据集，从 Common Crawl 中通过启发式清洗、Bloom 过滤器去重和基于 fastText 分类器的模型过滤等步骤精选而成。该数据集用于展示数据策展对训练高性能语言模型的重要性，经其训练出的 7B 模型在多个基准上表现优异。该数据集专注于语言理解和通用任务，主要面向研究用途，不适用于代码、数学等特定领域。
README
--- license: cc-by-4.0 dataset_info: features: - name: bff_contained_ngram_count_before_dedupe dtype: int64 - name: language_id_whole_page_fasttext struct: - name: en dtype: float64 - name: metadata struct: - name: Content-Length dtype: string - name: Content-Type dtype: string - name: WARC-Block-D…