How to use ready-made language models to evaluate answer quality?

Pre-trained language models (like GPT, BERT, and their derivatives) can be used to evaluate answer quality by applying them as evaluators in NLP tasks.

Main approaches:

1. Semantic similarity — the model computes the similarity between the generated answer and a reference answer, for example, using cosine similarity of embeddings.

2. Criteria-based evaluation — the model classifies or regress the quality of the answer based on given parameters (relevance, completeness, grammaticality).

3. Zero-shot or few-shot learning — by providing instructions or examples, the model can give an assessment without additional training.

Example with Hugging Face Transformers for embeddings:

from transformers import AutoTokenizer, AutoModel
import torch
import numpy as np

# Load model and tokenizer
model_name = 'sentence-transformers/all-MiniLM-L6-v2'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

def embed(text):
    inputs = tokenizer(text, return_tensors='pt', truncation=True, padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)
    return embeddings[0].numpy()

# Calculate cosine similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

reference = "Reference answer"
generated = "Generated answer"

ref_emb = embed(reference)
gen_emb = embed(generated)

score = cosine_similarity(ref_emb, gen_emb)
print(f'Semantic similarity: {score:.3f}')

This way, you can automate answer quality assessment using pretrained language models without manual annotation of large datasets.