fineweb HuggingFaceFW
🍷 FineWeb 15 trillion tokens of the finest data the 🌐 web has to offer What is it? The 🍷 FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove library, our large scale data processing library. 🍷 FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb, with a… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb.
- Tipo
- dataset
- Licencia
- odc-by
- Lenguaje
- en
- Descargas
- 418,229
- Me gusta
- 3,211
- Acceso
- public
- Archivos
- 500
README
--- license: odc-by task_categories: - text-generation language: - en pretty_name: FineWeb size_categories: - n>1T configs: - config_name: default data_files: - split: train path: data/*/* - config_name: sample-10BT data_files: - split: train path: sample/10BT/* - config_name: sample-100BT data_fil…