Sobes.tech
Middle

What does the storage time of your metrics depend on?

sobes.tech AI

Answer from AI

The retention period for metrics depends on several key factors, which are determined by business needs, storage costs, and technical capabilities of the monitoring system:

  1. Business requirements and SLA: Critical systems may require longer storage (years) for long-term trend analysis, audits, and incident post-mortems. Less critical or temporary data can be stored for shorter periods (weeks, months).

  2. Storage costs: The main factor. Storing large volumes of metrics on disk or in the cloud is expensive. Increasing the retention period results in proportional cost growth. Tiered storage (hot/cold storage) with different costs is often used.

  3. Metric granularity: Initially, metrics may be collected with high granularity (e.g., every second). For long-term storage, data is often aggregated (e.g., averaged to a minute, hour, day), significantly reducing volume. Storage policies can be multi-tiered: high granularity for short periods, lower for long periods. Example policy in Prometheus:

    // Prometheus storage configuration
    storage:
      tsdb:
        # Retention period
        retention: 365d # store data for 365 days
    
  4. Technical capabilities of the monitoring system: The maximum volume of data that the monitoring system (Prometheus, VictoriaMetrics, Graphite, Elasticsearch, etc.) can process and store, disk space limitations, and query performance when working with large data volumes.

  5. Analysis goals: For operational monitoring and alerting, a short period and high granularity are sufficient. For performance analysis over time, resource planning, or incident analysis of past periods, longer storage may be required, possibly with reduced granularity.

Typically, we set up multi-tiered policies:

  • Low granularity (seconds/minutes) - several days/weeks.
  • Medium granularity (minutes/hours) - several months.
  • High granularity (hours/days) - years (for historical reports and trends).

Specific values are determined within Capacity Planning and Cost Management for each monitoring system.