Text similarity/comparison

Native pacakage - SequenceMatcher

https://docs.python.org/3/library/difflib.html

There are are many are different string metrics like Levenshtein , Damerau-Levenshtein , Hamming distance , Jaro-Winkler  and Strike a match .

Levenshtein

• much faster than sequenceMatcher

Locality-sensitive hashing

• https://towardsdatascience.com/understanding-locality-sensitive-hashing-49f6d1f6134

Elasticsearch

• It supports fuzzy search - which calculates Levenshtein distance
  • https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html
• Text similarity search with vectors
  • https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch
• Advanced usage - Text similarity with TF models and Elastic search
  • https://www.ulam.io/blog/text-similarity-search-using-elasticsearch-and-python
    • Not training a model, but use a pre-trained model to get text embeddings

Comments