How is the data aggregated after being processed by workers?

In Go, when using worker pools for parallel data processing, a common task is to aggregate results.

Main methods of data aggregation after worker processing include:

1. Using channels

   • Each worker sends its result to a shared channel.
   • The main goroutine reads from the channel and aggregates data (e.g., sums, collects into a slice).

2. Using sync.WaitGroup to wait for all workers to finish

   • After all workers complete, aggregation can occur in the main goroutine.

3. Using mutexes (sync.Mutex) or atomic operations

   • If the resource being aggregated is shared (e.g., a counter), workers can update it with race condition protection.

Example with channels and WaitGroup:

var wg sync.WaitGroup
results := make(chan int, 10) // buffered channel

worker := func(id int, jobs <-chan int) {
    defer wg.Done()
    for job := range jobs {
        // processing
        result := job * 2
        results <- result
    }
}

jobs := make(chan int, 10)

// Launch workers
for w := 0; w < 3; w++ {
    wg.Add(1)
    go worker(w, jobs)
}

// Send jobs
for j := 1; j <= 5; j++ {
    jobs <- j
}
close(jobs)

// Close results channel after all workers are done
go func() {
    wg.Wait()
    close(results)
}()

// Aggregate results
sum := 0
for r := range results {
    sum += r
}
fmt.Println("Sum of results:", sum)

Thus, data is aggregated via channels, and synchronization is ensured by WaitGroup.