How was the quality of the RAG system evaluated?

The quality of the RAG (Retrieval-Augmented Generation) system is usually evaluated based on several key metrics:

• Retrieval Accuracy: how relevant the documents or fragments retrieved by the system are for generating the answer.
• Generated Text Quality: assessed using metrics like BLEU, ROUGE, METEOR, as well as human evaluation (e.g., coherence, completeness, and reliability of the answer).
• End-to-End Efficiency: how well the final answer matches the user's query, considering both search quality and generation.

An example of an evaluation approach:

1. First, check how relevant the found documents are (e.g., using Precision@k, Recall@k).
2. Then, evaluate the quality of the generated answer based on these documents.
3. Often, A/B testing with users is conducted to assess satisfaction with the answers.

Thus, the evaluation combines the retrieval component and the generative component for a comprehensive assessment of the RAG system.