Suppose we are a large advertising network. We place banners on partner sites around the world. We need to design a system for collecting and processing events in real-time. This data is critically important for two purposes: Billing: Deducting money from advertisers for clicks. Analytics: Displaying the current campaign effectiveness (CTR, impressions) in the user dashboard. Initial data (for load calculation) You need to independently estimate the required capacities (RPS, traffic, storage) based on the following metrics: Partner network: 500,000 active sites. Traffic: On average, each site has 2 page views per second. Ad blocks: Each page displays 3 banners simultaneously. Conversion: The average CTR (Click-Through Rate) is 1%. Unevenness: Peak load (evening hours) is 4 times higher than the average. Event size: The event object (banner ID, site ID, UserID, Timestamp, event type) weighs approximately 500 bytes. Technical requirements Near Real-Time: Data in the analytics interface should appear with a delay of no more than 10 seconds. Reliability: Click loss is unacceptable (this is a direct loss of money). Impressions loss is permissible within 0.01%. Scalability: The system should easily scale as the number of platforms grows.

For designing a real-time event collection and processing system for a large advertising network with the specified parameters, let's consider key aspects:

Load estimation

• Active sites: 500,000
• Page views per second per site: 2
• Banners per page: 3
• CTR: 1%
• Peak load: ×4
• Event size: 500 bytes

Events per second (RPS) calculation:

• Page views: 500,000 × 2 = 1,000,000 per second
• Banner impressions: 1,000,000 × 3 = 3,000,000 impressions per second
• Clicks: 3,000,000 × 1% = 30,000 clicks per second

Peak load: 3,000,000 × 4 = 12,000,000 impressions and 30,000 × 4 = 120,000 clicks per second.

Data traffic

• Impressions: 500 bytes × 12,000,000 = ~6 GB/sec
• Clicks: 500 bytes × 120,000 = ~60 MB/sec

System architecture

1. Data collection:

   • Use lightweight agents or SDKs on sites to send events.
   • Protocols: HTTP/2, gRPC, or Kafka Producer for high throughput.

2. Reception and buffering:

   • Use a distributed message broker (e.g., Apache Kafka) for reliable reception and buffering of events.
   • Configure partitioning by site or banner ID for scalability.

3. Processing:

   • Stream processing with systems like Apache Flink, Kafka Streams, or a custom Go service.
   • Process clicks with delivery guarantees (exactly-once or at-least-once).
   • Impressions can tolerate slight loss (0.01%), use processing trade-offs.

4. Storage:

   • For billing — a transactional database (e.g., PostgreSQL, CockroachDB).
   • For analytics — OLAP storage or columnar DB (ClickHouse, Druid) for fast aggregations.

5. Data display:

   • Cache aggregated data for quick access in the personal account.
   • Update data with a delay of no more than 10 seconds.

Scalability and reliability

• Horizontal scaling of all components.
• Replication and backup of brokers and databases.
• Monitoring and alerts for data loss and delays.

Summary

• The system should handle up to 12 million events per second at peak.
• Use distributed technologies for reception and processing.
• Ensure delivery guarantees for clicks and minimal loss for impressions.
• Provide near real-time analytics updates with a delay of up to 10 seconds.