Are you familiar with fan-in/fan-out and pipeline patterns? What are they?

Fan-in/fan-out and pipeline patterns are often used to organize parallel and pipeline data processing in Go.

• Fan-out is when one data source is distributed to multiple goroutines for parallel processing. For example, several workers read from a single channel.

• Fan-in is the reverse process, where results from multiple goroutines are collected into one channel, merging data streams.

• Pipeline is a sequence of processing stages, where the output of one stage is passed to the input of the next through channels. It allows building processing chains with separation of concerns and parallelism.

An example of a simple pipeline with fan-out and fan-in:

func generator(nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for _, n := range nums {
            out <- n
        }
    }()
    return out
}

func worker(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for n := range in {
            out <- n * n // for example, square the number
        }
    }()
    return out
}

func fanIn(channels ...<-chan int) <-chan int {
    var wg sync.WaitGroup
    out := make(chan int)
    output := func(c <-chan int) {
        defer wg.Done()
        for n := range c {
            out <- n
        }
    }
    wg.Add(len(channels))
    for _, c := range channels {
        go output(c)
    }
    go func() {
        wg.Wait()
        close(out)
    }()
    return out
}

func main() {
    nums := generator(1, 2, 3, 4, 5)

    // Fan-out: start multiple workers
    w1 := worker(nums)
    w2 := worker(nums)

    // Fan-in: merge results
    for n := range fanIn(w1, w2) {
        fmt.Println(n)
    }
}

Thus, these patterns help efficiently utilize parallelism and build scalable processing pipelines.