Model2Vec as a web content classifier
Introduction
LLMs are typically pre-trained using massive volumes of text, a large percentage of which is low quality. Besides steps like (near-)deduplication and basic content filtering based on word lists, teams typically employ small classifiers.
Here are a few examples: the FineWeb-Edu corpus (Penedo et al., 2024) was filtered from FineWeb by using an educational content classifier. Ultra-FineWeb (Wang et al., 2025) was curated using several fastText (Joulin et al., 2017) classifiers, which preprocess text with a lightweight subword tokenizer instead of a more expensive one. The Kolibri tech report (Aleph Alpha, 2026) describes using a frozen Luxical-one (Merrick et al., 2025) with an MLP and achieving much better scores than a fastText model, and using it in lieu of a much larger BERT-based model.
In this short report, we contrast Model2Vec (Tulkens & Van Dongen, 2024) with fastText and Luxical on the FineWeb-Edu classification dataset. We will cover a wider range of classification datasets in a follow-up post.
Data
The FineWeb-Edu dataset consists of ~467k annotated FineWeb samples with a score from 0 to 5 inclusive, where higher scores mean that the article is more likely to be educational. The scores were initially generated by Llama-3-70B-Instruct, after which the authors trained a frozen Arctic Embed M model with a regression head on the data. While this data does not have a test split, the original code does have the snippet that generated the splits, so we can faithfully reproduce our scores using our own models.
We train in two settings: a binary setting, in which scores >= 3 become “edu” and others “not edu”, and the original setting, in which we train on all six classes. Although the original Arctic Embed model was trained as a regression model, we found this to be suboptimal, with worse scores across all models we trained.
Models
We contrast the following models:
- The original FineWeb-Edu classifier, a frozen Arctic Embed M model (109M parameters) with a regression MLP.
- fastText: a fastText model (347M parameters), trained from scratch on the data.
- fastText (pretrained): the same as above, but starting from English pretrained vectors.
- Luxical-one + MLP: a frozen Luxical-one with a 1-layer MLP with 512 units on top.
- Model2Vec + MLP: a frozen potion-base-32M with the same MLP head as Luxical-one + MLP.
- Model2Vec (fine-tuned): potion-base-32M with the same MLP head, trained using Model2Vec’s classification framework, which fine-tunes the static embeddings together with the head.
Note that potion-base-32M is much smaller than Luxical-one, coming in at 32M parameters, while Luxical-one has ~200M vocabulary parameters, with a ~10M internal MLP. Luxical-one is used as a frozen encoder, as in the Kolibri report, because it has no built-in support for fine-tuning the encoder. We compare each method as it is available out of the box, and include Model2Vec + MLP as a direct comparison with Luxical-one + MLP.
Results
We present the results of both conditions separately, followed by the inference speed of each method. Note that in real pipelines, these models would be part of a larger set of classifiers and heuristics, so you would not necessarily use the binary cutoff to actually throw away real docs. Nevertheless, binary performance is indicative of the discriminative power of the classifier. Because we are mostly interested in filtering, we use the Binary F1, the F1 score of the positive class, as our main metric.
Score prediction
| Method | Accuracy | Macro F1 | Weighted F1 | Binary F1 |
|---|---|---|---|---|
| FineWeb-Edu classifier | 0.718 | 0.497 | 0.712 | 0.676 |
| Model2Vec (fine-tuned) | 0.686 | 0.448 | 0.669 | 0.636 |
| Luxical-one + MLP | 0.691 | 0.460 | 0.679 | 0.634 |
| Model2Vec + MLP | 0.685 | 0.442 | 0.672 | 0.603 |
| fastText (pretrained) | 0.686 | 0.437 | 0.671 | 0.560 |
| fastText | 0.683 | 0.427 | 0.666 | 0.549 |
Binary prediction
| Method | Accuracy | Macro F1 | Weighted F1 | Binary F1 |
|---|---|---|---|---|
| FineWeb-Edu classifier (score ≥ 3) | 0.947 | 0.823 | 0.944 | 0.676 |
| Model2Vec (fine-tuned) | 0.935 | 0.808 | 0.936 | 0.651 |
| Luxical-one + MLP | 0.940 | 0.795 | 0.937 | 0.622 |
| Model2Vec + MLP | 0.938 | 0.784 | 0.933 | 0.603 |
| fastText (pretrained) | 0.935 | 0.767 | 0.929 | 0.568 |
| fastText | 0.934 | 0.760 | 0.928 | 0.556 |
Inference speed
Because the score and binary models share the same architecture, we time each method once, on the full test set of 46,743 documents. We report the mean and standard deviation over 5 runs, after 1 warmup run, using 6 threads.
| Method | Time (s) | Documents / second |
|---|---|---|
| FineWeb-Edu classifier | 3262.8 ± 5.6 | 14 |
| Model2Vec (fine-tuned) | 4.18 ± 0.06 | 11,184 |
| Model2Vec + MLP | 5.46 ± 1.04 | 8,558 |
| fastText | 6.29 ± 0.13 | 7,427 |
| fastText (pretrained) | 6.30 ± 0.08 | 7,415 |
| Luxical-one + MLP | 10.06 ± 0.07 | 4,648 |
Discussion
On both evaluations, the pretrained classifier outperforms all other methods, showing that strong pretrained transformers, even when frozen, outperform static models, even static models with up to three times as many parameters. Static methods, however, are between 300 and 800 times faster.
Within the static models, fastText clearly underperforms, regardless of whether we initialize it from pretrained vectors: every Model2Vec and Luxical-one variant beats it by between 3.5 and 9.5 points of binary F1. This shows that, even with hundreds of thousands of training examples, fastText likely does not have a strong enough inductive bias to solve the task correctly. This is in line with the results reported in the Kolibri tech report (Aleph Alpha, 2026), which found that fastText underperforms compared to Luxical-one.
When both are used as frozen encoders with the same MLP head, Luxical-one outperforms Model2Vec on all metrics, with a binary F1 of 0.634 versus 0.603 in the score setting, and 0.622 versus 0.603 in the binary setting. Given that Luxical-one has roughly six times as many parameters, this is not surprising. Unfreezing the Model2Vec encoder, which is a built-in capability of Model2Vec, narrows this gap. In the score setting, fine-tuned Model2Vec and Luxical-one + MLP reach a similar binary F1 (0.636 versus 0.634), while Luxical-one + MLP scores higher on accuracy, macro F1, and weighted F1. In the binary setting, fine-tuned Model2Vec has a higher binary F1 (0.651 versus 0.622) and macro F1 (0.808 versus 0.795), while Luxical-one + MLP has a slightly higher accuracy (0.940 versus 0.935) and weighted F1 (0.937 versus 0.936). Fine-tuned Model2Vec is also 2.4 times faster at inference than Luxical-one + MLP, and even 1.5 times faster than fastText, although whether this actually matters in practice is debatable.
Conclusion
Model2Vec is a strong choice for web content classification. It outperforms fastText by a wide margin, and, when fine-tuned, reaches a similar or higher binary F1 than Luxical-one while being six times smaller and 2.4 times faster. If you currently use fastText to filter pre-training data, switching to Model2Vec should give you a better classifier that is also faster. Pretrained transformers remain the most accurate option, but at several hundred times the inference cost, static models are still a practical choice for filtering at web scale.
References
Aleph Alpha (2026, October). Kolibri: A Sovereign European Model on the Pareto Frontier. https://aleph-alpha.com/downloads/tech-report.pdf
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017, April). Bag of tricks for efficient text classification. In Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers (pp. 427-431).
Merrick, L., Fang, A., Carranza, A., Deng, A., Abbas, A., Larsen, B., … & Leavitt, M. (2025). Luxical: High-speed lexical-dense text embeddings. arXiv preprint arXiv:2512.09015.
Penedo, G., Kydlíček, H., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., & Wolf, T. (2024). The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37, 30811-30849.
Tulkens, S., & van Dongen, T. (2024). Model2Vec: Fast State-of-the-Art Static Embeddings. https://doi.org/10.5281/zenodo.17270888
Wang, Y., Fu, Z., Cai, J., Tang, P., Lyu, H., Fang, Y., … & Liu, Z. (2025). Ultra-fineweb: Efficient data filtering and verification for high-quality llm training data. arXiv preprint arXiv:2505.05427.