Deduplicate datasets with SemHash
SemHash
instance with your dataset.
This will build an index, which can then be used for fast deduplication.
You can set several parameters here, such as the model to use.
The default model is minishlab/potion-base-8M
, which is a lightweight model that works well for most English text datasets.
For multilingual datasets, you can use minishlab/potion-multilingual-128M
, which is optimized for multilingual data.
Note that you can also use your own custom model, or any SentenceTransformer model.
Parameters
None
, uses the default (minishlab/potion-base-8M).self_deduplicate
method.
This will remove semantic duplicates from the dataset.
Parameters
deduplicate
method.
This allows you to remove duplicates from one dataset against another dataset, which is useful for ensuring that your test set does not overlap with your training set.
Parameters
question
and context
columns, you can deduplicate based on both columns.
This will filter out records that have similar questions and contexts, ensuring that you do not have redundant entries in your dataset.
This is useful for datasets like SQuAD, where you can have the same question asked with different contexts, and you want to ensure that each question-context pair is unique.
DeduplicationResult
object returned by the deduplication methods provides several useful attributes that help you understand the deduplication process:
selected
: The deduplicated records.filtered
: The records that were removed as duplicates. This is returned as a list of DeduplicationResult
, each consisting of:
record
: The original record that was removed.exact
: Whether the record was an exact duplicate.duplicates
: A list of records that were considered duplicates of the removed record, along with their similarity score.duplicate_ratio
: The ratio of duplicates found in the dataset.exact_duplicate_ratio
: The ratio of exact duplicates found in the dataset.get_least_similar_from_duplicates(n)
: Returns the least similar record from the duplicates. This allows you to easily find the right deduplcation threhsold.rethreshold(threshold)
: Re-applies the deduplication with a new threshold, allowing you to adjust the sensitivity of the deduplication process without rebuilding the SemHas index.