Why do we need to store raw data in S3 storage? Is it correct to use TTL of 5 minutes on Redis, or is it better to store longer — 24 hours, and use ScyllaDB as a data verification source?

Raw data is stored in S3 for long-term and cost-effective storage of large volumes of information in its original form. This allows:

• To maintain a complete historical archive for auditing and re-analysis.
• To ensure data recovery in case of errors or loss in other systems.
• To use data for model training or analytics without risking damage to the original.

Regarding TTL in Redis:

• Redis is an in-memory store, optimal for quick access to frequently used data.
• A TTL of 5 minutes is suitable for caching fresh data that quickly becomes outdated.
• If data is needed for longer (e.g., 24 hours), it is better to increase the TTL or use a more appropriate storage.

ScyllaDB is a distributed database with high performance and durability, well-suited for storing and verifying data over long periods.

Recommendation:

• Use Redis with a short TTL for cache and quick access.
• Store raw data in S3 for long-term storage.
• Use ScyllaDB as a source of truth and for analytics, where data needs to be stored longer and be accessible for queries.

This approach provides a balance between access speed and storage reliability.