ettin-pretraining-data jhu-clsp
Ettin Pre-training Data Phase 1 of 3: Diverse pre-training data mixture (1.7T tokens) used to train the Ettin model suite. This dataset contains the pre-training phase data used to train all Ettin encoder and decoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository. 📊 Data Composition Data Source Tokens (B) Percentage Description DCLM 837.2 49.1% High-quality web crawl data CC Head 356.6… See the full description on the dataset page: https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data.
- Type
- dataset
- License
- mit
- Language
- en
- Downloads
- 189,173
- Likes
- 9
- Access
- public
- Files
- 0
Tags
- 预训练语料
- 文本数据集
- 文本生成
- 多语言语料
- 代码数据
- 大语言模型
- 预训练
README
--- language: - en license: mit task_categories: - text-generation - fill-mask - text-classification - retrieval tags: - pretraining - language-modeling - encoder - decoder - foundation-model - transformer --- # Ettin Pre-training Data [![License: MIT](https://img.shields.io/badge/License-MIT-yello…