How to parallelize log processing using a worker pool? How to get results from goroutines?

To parallelize log processing with a worker pool in Go, a pool of goroutine workers is created, which receive tasks from a channel. The main goroutine sends logs to the task channel, and the workers read, process them, and send the results to another channel.

Example:

package main

import (
	"fmt"
	"sync"
)

func worker(id int, jobs <-chan string, results chan<- string, wg *sync.WaitGroup) {
	defer wg.Done()
	for log := range jobs {
		// Log processing
		result := fmt.Sprintf("worker %d processed: %s", id, log)
		results <- result
	}
}

func main() {
	jobs := make(chan string, 100)
	results := make(chan string, 100)
	var wg sync.WaitGroup

	numWorkers := 5
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1)
		go worker(w, jobs, results, &wg)
	}

	logs := []string{"log1", "log2", "log3", "log4"}
	for _, log := range logs {
		jobs <- log
	}
	close(jobs)

	go func() {
		wg.Wait()
		close(results)
	}()

	for res := range results {
		fmt.Println(res)
	}
}

Here, the jobs channel passes tasks to the workers, and the results channel collects the results. The WaitGroup waits for all workers to finish, after which it closes the results channel for reading.