Is 5000 write requests per second a lot? How do you plan to handle write-flow through all downstream services (Inventory, Payment, Delivery)?

5000 write requests per second is a fairly high load, but not extreme for modern systems if the architecture and infrastructure are designed correctly.

To handle such write-flow through downstream services (Inventory, Payment, Delivery), the following approaches can be used:

1. Asynchronous processing via message queues:

   • Instead of synchronous writes directly to all services, publish events to a queue (e.g., Kafka, RabbitMQ).
   • Each downstream service subscribes to relevant events and processes them at its own pace.

2. Event-Driven Architecture principle:

   • The service receiving write requests generates events (e.g., UserOrderCreated).
   • Inventory, Payment, and Delivery respond to these events, updating their data.

3. Sharding and scaling:

   • Distribute the load across multiple service instances.
   • Use horizontal scaling to handle a large number of requests.

4. Idempotency and error handling:

   • Ensure idempotency of operations to avoid duplication on retries.
   • Implement retry mechanisms and compensation in case of failures.

5. Monitoring and Backpressure:

   • Monitor the state of queues and services.
   • Apply backpressure or throttling if necessary to prevent downstream overload.

An example of a simplified scheme in Go using channels to simulate a queue:

package main

import (
	"fmt"
	"time"
)

type WriteRequest struct {
	ID int
}

func main() {
	writeCh := make(chan WriteRequest, 10000) // buffered channel

	// Downstream services
	go func() {
		for req := range writeCh {
			// Processing Inventory
			fmt.Printf("Inventory processed request %d\n", req.ID)
		}
	}()

	// Simulating 5000 requests per second
	for i := 0; i < 5000; i++ {
		writeCh <- WriteRequest{ID: i}
	}

	// Give time to process
	time.Sleep(2 * time.Second)
}

In a real system, distributed queues and separate scalable services would be used instead of channels.