Golang

Using ScyllaDB before the producer is an anti-pattern. How to organize deduplication correctly?

Why do we need to store raw data in S3 storage? Is it correct to use TTL of 5 minutes on Redis, or is it better to store longer — 24 hours, and use ScyllaDB as a data verification source?

Suppose we are a large advertising network. We place banners on partner sites around the world. We need to design a system for collecting and processing events in real-time. This data is critically important for two purposes: Billing: Deducting money from advertisers for clicks. Analytics: Displaying the current campaign effectiveness (CTR, impressions) in the user dashboard. Initial data (for load calculation) You need to independently estimate the required capacities (RPS, traffic, storage) based on the following metrics: Partner network: 500,000 active sites. Traffic: On average, each site has 2 page views per second. Ad blocks: Each page displays 3 banners simultaneously. Conversion: The average CTR (Click-Through Rate) is 1%. Unevenness: Peak load (evening hours) is 4 times higher than the average. Event size: The event object (banner ID, site ID, UserID, Timestamp, event type) weighs approximately 500 bytes. Technical requirements Near Real-Time: Data in the analytics interface should appear with a delay of no more than 10 seconds. Reliability: Click loss is unacceptable (this is a direct loss of money). Impressions loss is permissible within 0.01%. Scalability: The system should easily scale as the number of platforms grows.