What metrics do you know for evaluating an LLM system as a whole?

To evaluate an LLM (Large Language Model) system, several types of metrics are usually used, which can be divided into technical and application-specific:

1. Text Generation Quality:

   • Perplexity — a measure of how well the model predicts test data (the lower, the better).
   • BLEU, ROUGE, METEOR — metrics for assessing the quality of generated text compared to reference answers.
   • Accuracy / F1-score — for classification or information extraction tasks.

2. Performance and Efficiency:

   • Response Time (latency) — how quickly the model responds.
   • Throughput — the number of requests processed per unit time.
   • Resource Usage — CPU, GPU, memory.

3. Usability and User Experience:

   • Human Evaluation — assessment of answer quality by people based on relevance, coherence, creativity.
   • User Satisfaction — surveys and metrics of user satisfaction.

4. Safety and Ethics:

   • Sensitivity to Toxicity — how much the model generates undesirable or harmful content.
   • Bias Metrics — assessment of model bias.

The choice of metrics depends on the specific task and system requirements. For example, speed and dialogue quality are important for chatbots, ROUGE for automatic summarization, and accuracy and recall for classification.