Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 below:

Website Navigation

• TOS
• Privacy
• About
• Jobs
• Models
• Datasets
• Spaces
• Pricing
• Docs

HuggingFaceFW/fineweb-2 · Datasets at Hugging Face

A sparkling update with 1000s of languages

This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages.

The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments.

In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂 FineWeb2 outperforms other popular pretraining datasets covering multiple languages (such as CC-100, mC4, CulturaX or HPLT, while being substantially larger) and, in some cases, even performs better than some datasets specifically curated for a single one of these languages, in our diverse set of carefully selected evaluation tasks: FineTasks.

The data was sourced from 96 CommonCrawl snapshots, spanning the summer of 2013 to April 2024, and processed using 🏭 datatrove, our large scale data processing library. This carefully deduplicated and filtered dataset comprises roughly 20 terabytes, across 5 billion documents, with over 3 trillion words (see How many tokens? for more details). For PII and opt-out see Personal and Sensitive Information and opt-out.

You will find our ablation and evaluation setup in this github repo. We will soon upload model checkpoints from our ablation experiments.

Read our 📝 research paper for details on the dataset creation!

For English data, please refer to the original 🍷 FineWeb.

Each language is identified by its ISO 639-3 code, and the data is grouped by language-script pairs, since some languages have content in multiple scripts.

In total, we provide filtered data for 1,868 language-script pairs. Of these, 474 have more than 1 thousand documents, and 203 have more than 10 thousand documents of filtered data. Most languages also include a small test split which should not be trained on.

While we tried our best to not overfilter, we know that our filtering isn't perfect, and wanted to allow the community to easily re-filter the data with their own filtering criteria. We have therefore also uploaded the data that was removed by our filtering pipeline for each language (it is suffixed by _removed). The filtered + the removed subsets of each language represent the entire data for a given language following global deduplication, which means that you do not have to re-deduplicate it yourself. You can find and adapt our filtering code here. The removed data is available through direct download (using hub_download for example) but not through load_dataset, as there would otherwise be an excessive number of subsets.

Additionally, we also uploaded data for scripts that the language classifier does not support or in a supported script but unknown language, without any deduplication or filtering. These are prefixed by und_.

The following table shows the size of the filtering subset for the biggest 80 languages. The full list is available on Github.

The number of tokens obtained when tokenizing data in a specific language heavily depends on whether the tokenizer was trained with that language, and its script, in mind. For instance, while employing the gpt2 tokenizer to tokenize Thai data might result in a very large number of tokens, using a tokenizer explicitly trained for south-east asian languages would considerably bring down this number.

As such, we chose to only report total number of documents, disk size and words for each language, as reported by the word tokenizer (we don't mean gpt2 here, but a tool that will only split words) that we assigned to each language.

Previous versions remain available in the branch version name. You can access them using for example revision="v2.0.0".

• v2.1.0 (27-06-2025): Filtering was slightly changed to match the version from our paper. The dataset size has increased. We have also added additional filtering to lower-resource languages to increase precision.
• v2.0.1 (08-01-2025): We reran the "fixes" step with most fixes from FTFY disabled except encoding correction. These fixes were, for example, changing all full-width punctuation in Chinese to half-width (which is not commonly used), as well as applying other normalizations that could make models not recognize certain types of characters or formatting. See here.
• v2.0.0 (08-12-2024): Initial version

See the tables above for the subset of the language and version (filtered or removed) of the data you want to download.

We currently do not provide smaller sample versions, but by setting limit or using streaming=True you can easily fetch a sample of the data. If there is interest from the community we might upload smaller sampled versions later on.

from datatrove.pipeline.readers import ParquetReader



data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-2/data/por_Latn/train", limit=1000) 
for document in data_reader():
    
    print(document)





from datatrove.executor import LocalPipelineExecutor
from datatrove.pipeline.readers import ParquetReader
from datatrove.pipeline.filters import LambdaFilter
from datatrove.pipeline.writers import JsonlWriter

pipeline_exec = LocalPipelineExecutor(
    pipeline=[
        ParquetReader("hf://datasets/HuggingFaceFW/fineweb-2/data/por_Latn/train", limit=1000),
        LambdaFilter(lambda doc: "hugging" in doc.text),
        JsonlWriter("some-output-path")
    ],
    tasks=10
)
pipeline_exec.run()

from huggingface_hub import snapshot_download
folder = snapshot_download(
                "HuggingFaceFW/fineweb-2", 
                repo_type="dataset",
                local_dir="./fineweb2/",
                
                allow_patterns=["data/ces_Latn/train/*", "data/ces_Latn_removed/train/*"])

For faster downloads, make sure to install pip install huggingface_hub[hf_transfer] and set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1.

As mentioned above, load_dataset will not work for und_ or _removed splits.

from datasets import load_dataset

fw = load_dataset("HuggingFaceFW/fineweb-2", name="hrv_Latn", split="train", streaming=True)

We used the 🏭 datatrove library to process the data. You can find a working script that launches the entire processing pipeline here.

The processing pipeline had to be heavily adapted for a multilingual setting. As each language has its own peculiarities, we individually tuned each filter, defining different thresholds and stopwords for each language. 📊 These thresholds and stopwords are available in /configs/{iso3_lang}_{script}.yml in our github repo.

The starting point for our dataset was the non-English data (< 0.65 score in English) we obtained when processing the original FineWeb. This data was text extracted using trafilatura and went through our URL filters (for more info see 🍷 FineWeb. To this data, we applied the following processing steps:

1. Additional Language Identification and filtering 🔍
2. Deduplication per language 🔄
3. Filtering per language 🧹
4. PII Anonymization and fixes 🎭

Performed using GlotLID, which not only covers a wider variety of languages (2000+ available labels) compared to fasttext176 (used in the original FineWeb), as it also identifies the script used in each document. 📜

For each language, we defined different minimum language classifier confidence scores to keep a document.

Unlike in 🍷 FineWeb, where data was deduplicated per CommonCrawl snapshot, in 🥂 FineWeb2, data is deduplicated per language, globally. However, following our deduplication findings in the original 🍷 FineWeb, while we remove all except one document from each duplicate cluster, we save the size of this cluster in the kept document's metadata, saved in minhash_cluster_size. This allows us to "re-hydrate" the dataset: by upsampling documents based on their cluster size, we see clear performance improvements for some languages, particularly high resource ones. 📈

We think upsampling weights should be dataset specific, and have therefore used the filtering rates of each duplicate cluster to compute different weights per language. They are available on our Github repo, along with sample code to Rehydrate the dataset.

WARNING: If you do not upsample based on these weights, dataset performance may be lower than the one obtained on our results.

We mostly kept the original 🍷 FineWeb set of filters, and do not create new filters targeting individual languages. As such, we had to extensively ablate on different processes of adapting the English filters to all the languages we supported. 🔍

Based on the results of our experiments, we also disabled/changed global values of some specific filters:

• For FineWebQuality filters, we removed short_line_thr and changed char_dup_ratio from 0.01 to 0.1.
• Gopher Repetition filter: disabled paragraph related filters as trafilatura does not keep them ❌
• C4 filters: we did not include the C4 filters as they seemed to degrade performance in this multilingual setting 📉

• PII Removal: Kept unchanged, emails and ip addresses are anonymized. ✉️
• We applied FTFY to fix encoding issues. 🔧
• Added some code to fix trafilatura created artifacts related to tables 🛠️

We will soon release more details regarding the reasoning behind each of these decisions in our upcoming blogpost.

We chose 9 diverse (in script, language family and resource availability) languages for our ablation setup: Chinese, French, Arabic, Russian, Thai, Hindi, Turkish, Swahili, and Telugu. We then selected high signal tasks for these languages out of almost 200 benchmarks. We wrote an entire blogpost about this process: FineTasks, where you will find the full list of tasks we evaluated on, as well as how they were selected. As for metrics, we use normalized probability mass (not accuracies!) for discriminative tasks and f1 for generative tasks, as these metrics have proven to be far more stable than their alternatives.

We conducted our dataset performance ablations and evaluations by training a series of 1.45B parameters models on ~30 billion tokens, tokenized using the gemma tokenizer. To compare 🥂 FineWeb2 with other datasets, we also trained one of these 1.45B models per target dataset, on 30 billion tokens sampled from it (or the entire dataset when its size was < 30 billion tokens). We chose 30B as some of the comparison datasets were relatively small for some languages, but we will soon release some longer ablation runs.

The detailed configurations for training the models can be found here.

Note: the results below use an older version of the dataset. Please check our paper for updated results. You will find all the evaluation results in the repo files. The 🥂 FineWeb2 runs were trained on the final data (dedup+filtering) with re-hydration (see the section on deduplication above), unless explicitly stated (e.g. Swahili).

We compared 🥂 FineWeb2 with the following multilingual datasets:

• mC4
• CC-100
• HPLT v1.2
• CulturaX
• HPLT V2.0

And with language specific monolingual datasets:

• ArabicWeb24 (arabic)
• Arabic-101B (arabic)
• Croissant (french)
• Sangraha (hindi & telugu)
• Odaigen(hindi)
• Omnia Russica (russian)
• Sea CommonCrawl (thai)
• VNGRS-Web-Corpus (turkish)
• MNBVC (chinese)
• TigerBot (chinese)
• MAP-CC (chinese)

Expand each individual language to see the corresponding plot. The error bars correspond to one standard deviation of the scores of 4 models trained on different randomly sampled 30B tokens of unfiltered CommonCrawl data.

This dataset was created by processing 96 CommonCrawl dumps comprising web data crawled from the summer of 2013 to April 2024. 🥂 FineWeb2 includes a variety of domains and topics in a variety of languages and is primarily intended to be used as a research artifact on public data in the context of pretraining datasets for large language models. The CommonCrawl data was carefully processed, deduplicated and filtered with the 🏭 datatrove library, resulting in the largest publicly available multilingual clean LLM pretraining dataset.

The following is an example sample from the dataset. It is part of the French (fra_Latn) data, originally belonged to the CC-MAIN-2013-20 CommonCrawl snapshot and was crawled on 2013-05-19T07:12:36Z.

{
   "text": "Il y a 61 ans le match le plus long de l'histoire\nLe 6 janvier 1951 les Rochester Royals recevaient les Indianapolis Olympians pour ce qui allait être le match le plus long de l'histoire. Rochester qui sortait d'une victoire face aux Knicks de New York en prolongation étaient sur une série de 7 victoires avant la réception d'Indianapolis. Au final un match remporté au bout de la nuit par les Olympians en 6 prolongations et un tout petit score de 75 à 73. les équipes n'avaient shooté que 23 fois au total des 6 prolongations! (l'horloge de tir n'était pas encore utilisée)\nCe match reste à ce jour le plus long de l'histoire avec 78 minutes de jeu.",
   "id": "<urn:uuid:5013b1b9-5092-40f8-8d79-c517970dd814>",
   "dump": "CC-MAIN-2013-20",
   "url": "http://basket-infos.com/2012/01/06/il-y-a-61-ans-le-match-le-plus-long-de-lhistoire/",
   "date": "2013-05-19T07:12:36Z",
   "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696384213/warc/CC-MAIN-20130516092624-00033-ip-10-60-113-184.ec2.internal.warc.gz",
   "language": "fra",
   "language_script": "Latn",
   "language_score": 0.9994362592697144,
   "minhash_cluster_size": 1,
   "top_langs": "{\"fra_Latn_score\": 0.9994362592697144}"
}

• text (string): the main text content
• id (string): original unique identifier for this sample from CommonCrawl
• dump (string): the CommonCrawl dump this sample was a part of
• url (string): url to the original page where text was present
• date (string): crawl date (from CommonCrawl)
• file_path (string): s3 path for the individual CommonCrawl warc file containing this sample
• language (string): ISO 639-3 code for the language of this sample
• language_script (string): script of the text, for example Latn
• language_score (float): language prediction score as reported by the GlotLID classifier
• top_langs: language-script pairs for which the language classifier
• minhash_cluster_size: number of samples in the minhash cluster of this sample. See the deduplication section to learn why this might be useful

See "Languages and available subsets" above.

While multiple open-weights models have regularly been released in recent months, these releases often do not include the model's training data. With 🥂 FineWeb2 we aim to provide the open source community with a very large clean pretraining dataset that can be used to push the envelope on truly open source models (open source models where data is also released). We also seek to improve the representation of lower resource (and often ignored) languages, and deliberately chose a language classifier that supported a large number of language labels.

The source data consists of webpages crawled by the CommonCrawl foundation over the 2013-2024 time period.

We then extracted the main page text from the html of each webpage, identified its language, deduplicated the data per language and then filtered with specific thresholds adapted to each language.

See "Dataset processing steps" above.

We augment the original samples with the language, language_script, language_score, top_langs and minhash_cluster_size annotations. The language related annotations are automatically generated by our language filter. minhash_cluster_size is computed during the deduplication process, by saving the size of each duplicate cluster before removing all of its documents except one.

We anonymize email addresses and public IP addresses.

For emails, we apply a regex pattern and replace any occurrence of an email address with either email@example.com or firstname.lastname@example.org. For IP addresses, we also employ a regex pattern and then further filter to only anonymize IP addresses allocated for public networks. Matched IP addresses are then replaced with one of the following randomly generated IP addresses, which at the time of dataset creation were not responding to ping requests: 22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 220.127.116.11, and 18.104.22.168. We decided against applying regex patterns for phone numbers due to the high false positive rate.

Despite our efforts, given that 🥂 FineWeb2 is sourced from the internet at large, it is very likely that some personable identifiable information (PII) will be present. If you find your own PII in 🥂 FineWeb2 and would like it removed, please fill out our PII removal/opt out form.

CommonCrawl respects robots.txt at crawl time, but if you are a webmaster and find your website in 🥂 FineWeb2 and would like to have it removed, you may also use the PII removal/opt out form.

With the release of this dataset we aim to make model training more accessible to the machine learning community at large.

While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🥂 FineWeb2 we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.

While LLM advancements have primarily focused on English, Chinese, and other Western languages, this release prioritizes broader language support. We consulted with practitioners who develop LLMs for diverse languages to address their specific requirements, such as proper word segmentation (particularly for scripts that don't use whitespace separation) and handling language-specific punctuation, ensuring that medium and lower resource languages were not an afterthought.

Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🥂 FineWeb2 was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.

Some filters might disproportionately target specific domains. One such example is poetry: we noticed that the punctuation filter removes a lot of poems.

We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to disproportionately remove content in specific dialects and overclassify as toxic text related to specific social identities, respectively.

While the language classifier we used, GlotLID supports over 2000 language labels, its performance is not ideal for all of them. The training data for many languages is hard to obtain and, additionally, the classifier is prone to sometimes mistaking closely related languages (for instance, Standard Arabic and Arabic dialects or Croatian and Bosnian). We tried to mitigate this by curating stopwords for each language, but these might also not be effective in all cases.

Due to resource constraints and limited access to native speakers, we couldn't test each language individually. We encourage users to review our filtering approach for their languages of interest and modify the processing if needed. To support this, we've made available all data removed by our filtering pipeline (see "Languages and available subsets" above for more info).

You should also probably consider complementing 🥂 FineWeb2 with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🥂 FineWeb2 (we did not tailor the processing to individual websites).

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.

@misc{penedo2025fineweb2pipelinescale,
  title={FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language}, 
  author={Guilherme Penedo and Hynek Kydlíček and Vinko Sabolčec and Bettina Messmer and Negar Foroutan and Amir Hossein Kargaran and Colin Raffel and Martin Jaggi and Leandro Von Werra and Thomas Wolf},
  year={2025},
  eprint={2506.20920},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2506.20920}, 
}

Downloads last month

38,340

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4