How to evaluate the quality of a retriever? Where to get a collection of documents for metric evaluation?

The quality of a retriever (search retrieval) is evaluated using metrics that reflect the relevance of the found documents:

• Precision: the proportion of relevant documents among the retrieved ones.
• Recall: the proportion of relevant documents found out of all relevant documents.
• F1-score: the harmonic mean of precision and recall.
• MAP (Mean Average Precision), NDCG (Normalized Discounted Cumulative Gain): consider the ranking of results.

For evaluation, a collection of documents with relevance annotations (ground truth) is needed. Such collections can be obtained from:

• Public datasets: for example, TREC, Cranfield, MS MARCO.
• Internal data: if access to historical queries and user assessments is available.
• Manually annotated: experts evaluate the relevance of documents for queries.

Example: to evaluate a news search system, a set of queries and, for each, a list of relevant articles can be used, after which precision and recall for the system's results can be calculated.