falcon-refinedweb tiiuae
📀 Falcon RefinedWeb Falcon RefinedWeb is a massive English web dataset built by TII and released under an ODC-By 1.0 license. See the 📓 paper on arXiv for more details. RefinedWeb is built through stringent filtering and large-scale deduplication of CommonCrawl; we found models trained on RefinedWeb to achieve performance in-line or better than models trained on curated datasets, while only relying on web data. RefinedWeb is also "multimodal-friendly": it contains links and… See the full description on the dataset page: https://huggingface.co/datasets/tiiuae/falcon-refinedweb.
- 类型
- dataset
- 许可
- odc-by
- 语言
- en
- 下载量
- 93,564
- 点赞
- 947
- 访问
- public
- 文件
- 0
标签
- 预训练语料
- 文本数据集
- 大语言模型
- NLP
- 数据处理
- 多模态数据集
摘要
Falcon RefinedWeb 是阿布扎比技术创新研究院（TII）构建的大规模英文网络语料库，基于 CommonCrawl 经过严格过滤与大规�模去重处理而成。它提供了约 10 亿条网页文本，总规模达 2.8TB，专为大语言模型预训练设计，也是 Falcon 系列模型的训练数据。该数据集还包含图片链接与替代文本，具有多模态友好的特性。
README
--- dataset_info: features: - name: content dtype: string - name: url dtype: string - name: timestamp dtype: timestamp[s] - name: dump dtype: string - name: segment dtype: string - name: image_urls sequence: sequence: string splits: - name: train num_bytes: 2766953721769 num_examples: 968000015 dow…