How would you set up monitoring for 200 servers? What approaches and tools would you use, and how would you design the monitoring system?

To monitor 200 servers, I would develop a scalable and reliable system that includes collection, storage, analysis, and visualization of metrics and logs.

Approaches:

1. Agent-based vs Agentless: I would use a combination. Agent-based for deep collection of system metrics (CPU, RAM, disk, network), agentless for checking service and port availability (ping, curl).
2. Centralized Monitoring: All data is collected and processed in a centralized system.
3. Automation: I would use automation tools for deploying agents, configuring settings, and creating dashboards.
4. Alerting Strategy: I would set up an alert system with clear rules, escalation procedures, and integration with notification tools (Slack, PagerDuty).
5. Logging: Centralized log collection for analysis and troubleshooting.
6. Visualization: Informative dashboards for quick system status overview.

Tools:

• Metrics collection: Prometheus (with exporters: node_exporter for hosts, blackbox_exporter for availability checks).
• Metrics storage: Prometheus (locally) and Thanos or VictoriaMetrics for long-term storage and scaling.
• Log collection: Fluentd or Filebeat for collection, Kafka or RabbitMQ (optional) for buffering.
• Log storage: Elasticsearch.
• Visualization: Grafana for metrics and logs.
• Alerts: Alertmanager (integrated with Prometheus), connected with Slack, PagerDuty.
• Automation: Ansible or Puppet for deployment and configuration.
• Orchestration (optional but recommended): Kubernetes for deploying the monitoring stack.

Monitoring System Design:

1. Architecture:

   • Multiple instances of node_exporter on servers.
   • One or more replicated Prometheus servers (using Thanos Receiver or VictoriaMetrics VMAgent) for data collection.
   • Thanos / VictoriaMetrics for aggregation, long-term storage, and queries over Prometheus.
   • A single Elasticsearch cluster for log storage.
   • A Grafana cluster for visualization.
   • One or more Alertmanager instances.
   • Fluentd / Filebeat agents on servers for log collection.

   graph TD
       A[200 Servers] -- node_exporter --> B(Prometheus Server)
       A -- Fluentd/Filebeat --> C(Kafka/RabbitMQ - Optional)
       C -- Fluentd/Logstash --> D(Elasticsearch Cluster)
       B -- Remote Write --> E(Thanos/VictoriaMetrics)
       E -- Query --> F(Grafana)
       D -- Query --> F
       B -- Alert --> G(Alertmanager)
       G -- Notify --> H(Slack/PagerDuty)
   

2. Metrics collection configuration (Prometheus):

   • Dynamic target discovery via service discovery (if infrastructure like Consul or Kubernetes is available) or static configurations (managed via Ansible).
   • Setting scrape intervals (scrape_interval).
   • Applying recording rules for aggregating frequently queried metrics.

   # Example scrape config for Prometheus
   scrape_configs:
     - job_name: 'node_exporter'
       # Dynamic discovery or static_configs
       static_configs:
         - targets: ['server1:9100', 'server2:9100', ...]
       # metrics_path: /metrics # Default value
   

3. Log collection configuration:

   • Setting up agents (Fluentd/Filebeat) to monitor log files and send them to the central node.
   • Parsing logs at the agent level or before sending to Elasticsearch for structuring data.

   # Example Filebeat configuration (filebeat.yml)
   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /var/log/*.log
       - /var/log/syslog
   #-------------------------- Elasticsearch output ---------------------------
   output.elasticsearch:
     hosts: ["elasticsearch:9200"]
   

4. Grafana setup:

   • Adding data sources (Prometheus, Elasticsearch).
   • Creating dashboards:
     • Overview dashboards (Health Overview).
     • Category dashboards (CPU, Memory, Disk, Network).
     • Dashboards for specific services/applications.
     • Log dashboards.
   • Using variable templates for dynamic server/service selection.

5. Alerting setup:

   • Defining threshold values based on metrics.
   • Creating alert rules in Prometheus (or Alertmanager).
   • Configuring recipients in Alertmanager (integrations).
   • Defining escalation policies (e.g., notify Slack first, then call via PagerDuty).

   # Example recording and alert rules in Prometheus (rules.yml)
   # Recording rule: CPU user load percentage over 5 minutes
   - record: instance:node_cpu_user_percent:rate5m
     expr: 100 - avg without (cpu)(rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100)
   
   # Alert rule: High CPU load (> 80%)
   - alert: HighCPULoad
     expr: instance:node_cpu_user_percent:rate5m > 80
     for: 5m # Wait 5 minutes before firing
     labels:
       severity: warning
     annotations:
       summary: "High CPU load on {{ $labels.instance }}"
       description: "CPU load on {{ $labels.instance }} is at {{ $value }}%."
   

6. Automation:

   • Using Ansible playbooks to:
     • Install and configure node_exporter, Fluentd/Filebeat.
     • Deploy the Prometheus/Alertmanager/Grafana/Elasticsearch stack.
     • Manage configuration files (scrape configs, rules, dashboards).

This would be a comprehensive, scalable, and flexible approach to monitoring a large number of servers.