Rääkige meile oma kogemusest logide analüüsis.

Log analysis was used for debugging, monitoring, and root cause analysis of incidents.

Main usage scenarios:

• Error detection: Identifying errors, warnings, and exceptions generated in application, server, or database logs.
• Performance diagnostics: Analyzing response times, CPU load, and memory usage from monitoring systems or application logs.
• Incident investigation: Reconstructing the timeline of events leading to a failure based on logs from various system components.
• System status monitoring: Setting up alerts for key events or errors in logs.
• User behavior analysis: Tracking sequences of user actions based on API or web server request logs.

Tools and approaches:

• Text editors and command-line utilities: Used grep, awk, sed for small projects or local analysis in Linux/Unix systems.
• Centralized logging systems: Worked with ELK stack (Elasticsearch, Logstash, Kibana), Grafana Loki. These systems allow collecting logs from different sources, centrally storing, indexing, and visualizing them.
• Structured logging: Encouraged development teams to use log formats that facilitate parsing (e.g., JSON), adding a Correlation ID to link requests across services.
• Monitoring and alerts: Configured rules in monitoring systems (Prometheus) and logging systems (ELK, Grafana) to send notifications upon critical errors or exceeding a certain number of warnings.

Example of using ELK for error search:

1. Identify the incident (e.g., user complaint).
2. Go to Kibana, select the relevant log index.
3. Enter keywords or filters (e.g., user ID, request time, microservice name).
4. Search by level: ERROR or level: WARNING.
5. Analyze the stacktrace and surrounding logs to determine the context of the error and identify the module or function that caused the failure.

{
  "@timestamp": "2023-10-27T10:30:00.123Z",
  "level": "ERROR",
  "message": "Database connection failed",
  "service": "user-service",
  "correlation_id": "abc-123",
  "error": {
    "type": "SQLException",
    "message": "Connection refused",
    "stacktrace": "..." // Full stacktrace here
  },
  "user_id": "user-xyz"
}

Log analysis is a critical part of product testing and support processes, enabling quick identification and localization of issues.