What is semantic search vs lexical search?

Lexical search is based on exact matching of words or phrases in the text. It searches for documents containing exactly the words specified in the query, without considering meaning or context. For example, keyword search in a database or simple string filtering.

Semantic search tries to understand the meaning of the query and documents using models that consider context, synonyms, and relationships between words. It allows finding relevant results even if there are no exact word matches. For example, searching for "car" might return documents with "automobile" or "transport".

Thus, lexical search is a search by literal matching, while semantic search is based on semantic relevance.

An example using Python and a library for semantic search:

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

corpus = ["I love driving a car", "The weather is sunny today", "Cars come in different brands"]
corpus_embeddings = model.encode(corpus, convert_to_tensor=True)

query = "automobile"
query_embedding = model.encode(query, convert_to_tensor=True)

hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=2)

for hit in hits[0]:
    print(corpus[hit['corpus_id']], f"(score: {hit['score']:.4f})")

This code finds sentences close in meaning to the word "automobile".