What is the BERT-as-a-reranker approach?

BERT-as-a-reranker is an approach in information retrieval where a pre-trained BERT model is used to re-evaluate (rerank) a list of documents obtained in the first stage of search. Initially, a traditional search engine (e.g., BM25) returns a set of relevant documents for a query. Then, BERT analyzes query-document pairs, assessing their relevance considering context and semantics, and rearranges the documents in the list to improve ranking quality.

This approach allows for increased search accuracy, especially for complex queries where deep context is important. However, it requires significant computational resources, so it is often applied only to a limited number of top results from the first stage.

Example of usage:

from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

query = "machine learning applications"
documents = ["Intro to ML", "Deep learning tutorial", "Cooking recipes"]

# Preliminary search returns documents
# Now BERT evaluates the relevance of each document
scores = []
for doc in documents:
    inputs = tokenizer.encode_plus(query, doc, return_tensors='pt')
    outputs = model(**inputs)
    score = outputs.logits.softmax(dim=1)[0][1].item()  # relevance probability
    scores.append((doc, score))

# Sort documents by descending score
reranked = sorted(scores, key=lambda x: x[1], reverse=True)
print(reranked)