CulturaX uonlp
CulturaX Cleaned, Enormous, and Public: The Multilingual Fuel to Democratize Large Language Models for 167 Languages Dataset Summary We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages, tailored for large language model (LLM) development. Our dataset undergoes meticulous cleaning and deduplication through a rigorous pipeline of multiple stages to accomplish the best quality for model training, including language… See the full description on the dataset page: https://huggingface.co/datasets/uonlp/CulturaX.
- 类型
- dataset
- 语言
- af
- 下载量
- 12,768
- 点赞
- 665
- 访问
- gated
- 文件
- 0
标签
- 多语言语料
- 预训练语料
- 大语言模型
- 文本数据集
- NLP
- 数据处理
摘要
CulturaX是一个面向大语言模型训练的超大规模多语言数据集，包含167种语言、6.3万亿token。该数据集通过多阶段严格清洗和去重流程确保训练数据质量，为LLM训练提供高质量的多语言语料。主要适用于多语言大语言模型的预训练，是民主化多语言LLM发展的重要数据资源。