Sobes.tech
Middle — Senior

Improving the speed of transmission and processing of massive data streams

livecode

Task condition

The service accepts a network stream of 2 GB, splits it into chunks of 1 MB, breaking only at newline boundaries, and places the resulting blocks into a queue. Several replicas subscribe to this queue and process the data, ensuring the "at most once" delivery guarantee. What strategies can be applied to increase the speed of processing and transmitting this data?