Results

The results of Model2Vec are presented in the following sections:

• MTEB Results
• Training Results
• Ablations

​

MTEB Results (English)

Model2Vec is evaluated on MTEB, as well as two additional tasks: PEARL (a phrase representation task) and WordSim (a collection of word similarity tasks). The results are shown in the table below.

Note: The potion and M2V models are our static models.

The results show that potion-base-32M is the most performant static embedding model. It reaches 92.11% of the performance of all-MiniLM-L6-v2 with an average MTEB score of 51.66 while being orders of magnitude faster.

Note: the potion-retrieval-32M, static-retrieval-mrl-en-v1, and static-similarity-mrl-multilingual-v1 models are task-specific models. We’ve included them for completeness, but they should not be compared directly to the other models for tasks that they are not designed for.

The figure below shows the relationship between the number of sentences per second and the average MTEB score. The circle sizes correspond to the number of parameters in the models (larger = more parameters). This plot shows that the potion and M2V models are much faster than the other models, while still being competitive in terms of performance with the all-MiniLM-L6-v2 model. NOTE: for fairness of comparison, we disabled multiprocessing for Model2Vec for this benchmark. All sentence-transformers models are run with the sentence-transformers library’s default settings for encode.

​

MMTEB Results (Multilingual)

The results for the multilingual models are shown in the table below. We compare against the LaBSE model, as well as other multilingual static embedding models.

As can be seen, potion-multilingual-128M is the most performant static multilingual model, reaching 90.86% of the performance of LaBSE. There are differences per task. The static-similarity-mrl-multilingual-v1 model is better for retrieval and STS tasks (which can be explained by the fact that it’s trained for STS), while the potion-multilingual-128M model is better for classification and clustering tasks. It is important to note that the potion-multilingual-128M model supports a total of 101 languages, while static-similarity-mrl-multilingual-v1 supports only 50 languages. It is also important to note that MMTEB does not include tasks for every language, and there may be a bias towards larger languages.

​

Retrieval Results

A subset of models we created and compare against are specifically designed for retrieval tasks. The results are shown in the table below, including two general-purpose models for comparison and a transformer.

As can be seen, potion-retrieval-32M model is the most performant static retrieval model, reaching 86.65%% of the performance of all-MiniLM-L6-v2 with a retrieval score of 36.35.

​

Training Results

The main results for Model2Vec training are outlined in this section.

We compare five different architectures for our main results:

• model2vec + logreg: A model2vec model with a scikit-learn LogisticRegressionCV on top.
• model2vec full finetune: A model2vec classifier with the full model finetuned. This uses our StaticModelForClassification.
• tfidf: A TF-IDF model with a scikit-learn LogisticRegressionCV on top.
• setfit: A SetFit model trained using all-minilm-l6-v2 as a base model.
• bge-base + logreg: A BGE-base encoder model with a scikit-learn LogisticRegressionCV on top.

We use 14 classification datasets, using 1000 examples from the train set, and the full test set. No parameters were tuned on any validation set. All datasets were taken from the Setfit organization on Hugging Face.

As can be seen see, full fine-tuning brings modest performance improvements in some cases, but very large ones in other cases, leading to a pretty large increase in average score. Our advice is to test both if you can use potion-base-32m, and to use full fine-tuning if you are starting from another base model.

The speed difference between model2vec and the other models is immeense, with the full finetune being 35x faster than a setfit based on all-minilm-l6-v2 on CPU and 200x faster than thebge-base transformer model.

The figure below shows the relationship between the number of sentences per second and the average training score, where we’ve included more transformer-based models for comparison.

​

Ablations

To better understand the factors contributing to the performance of Model2Vec, we conducted a comprehensive set of ablation studies, covering various aspects of the model’s architecture and preprocessing methods. In these studies, we examined the impact of key elements such as PCA, Zipf weighting, and the use of Sentence Transformers versus regular transformer models. We also compared the performance of input embeddings versus output embeddings, since it would seem plausible that these should also work well. The results are shown in the table below.

There’s four main findings in these results:

1. Non-Sentence Transformers do not work well. This can be seen by comparing M2V_bert_output_nozipf_nopca (which uses BERT, a non-Sentence Transformer) and M2V_base_output_nozipf_nopca (which uses BGE-base, a Sentence Transformer). Using a Sentence Transformer gives a ~5.2% increase in performance.
2. PCA is crucial for performance. This can be seen by comparing M2V_base_output_nozipf_nopca and M2V_base_output_nozipf which gives a ~2.8% increase in performance. Furthermore, PCA improves performance on all tasks.
3. Zipf weighting is crucial for performance. This can be seen by comparing M2V_base_output_nozipf_nopca and M2V_base_output_nopca which gives a ~3.1% increase in performance.
4. Output embeddings outperform input embeddings. This can be seen by comparing M2V_base_input and M2V_base_output which gives a ~6.1% increase in performance. Note that input embeddings do work well for some tasks. We hypothesize that this is because input embeddings are inherently normalized.