Senior
How would you design a table for storing high-load time series with tags (up to several million records per minute)? What key decisions about the table structure would you make?
sobes.tech AI
Answer from AI
To store high-load time series with tags (up to several million records per minute), I would choose a specialized time series database, such as TimescaleDB, InfluxDB, or ClickHouse, as they are optimized for such loads.
Main table structure solutions:
- Sharding and partitioning by time — to quickly delete old data and speed up queries by time ranges.
- Indexing by tags and time — for fast search and aggregation.
- Columnar storage — for efficient compression and quick reading of necessary columns.
- Using data compression — to reduce the volume of stored data.
- Write optimization — batching inserts, asynchronous writing.
Example of a table structure in TimescaleDB:
CREATE TABLE metrics (
time TIMESTAMPTZ NOT NULL,
tag1 TEXT,
tag2 TEXT,
value DOUBLE PRECISION,
PRIMARY KEY (time, tag1, tag2)
);
SELECT create_hypertable('metrics', 'time', chunk_time_interval => INTERVAL '1 day');
This approach allows for efficient storage and quick processing of large volumes of time data with many tags.