fineweb-tokenized anisoleai
FineWeb Tokenized > 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer What is it? This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards. By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.
- 类型
- dataset
- 许可
- odc-by
- 语言
- en
- 下载量
- 1,717,378
- 点赞
- 31
- 访问
- public
- 文件
- 0
标签
- 预训练语料
- 文本数据集
- 大语言模型
- 文本生成
- NLP
- 深度学习
摘要
该数据集是FineWeb的预分词版本，使用AnisoleAI BPE分词器（52,022词表）将约15万亿token的语料先行分词并打包成紧凑的uint16 Parquet分片。通过预分词，用户可直接将token ID流式送入训练管线，省去CPU和分词开销，显著加速大语言模型预训练。适用于需要高效加载大规模预训练语料的训练场景。
README
--- license: odc-by task_categories: - text-generation language: - en pretty_name: FineWeb Tokenized (AnisoleAI) size_categories: - n>1T tags: - tabular - text - pre-training configs: - config_name: default data_files: - split: train path: data_*/*.parquet --- # <img src="https://i.postimg.cc/SRBB1…