What pipeline?

A pipeline is a sequence of data processing steps or tasks, where the output of one step serves as the input for the next. In programming and data processing systems, pipelines help structure complex processes and ensure data flow.

In Go, the pipeline pattern is often used to organize processing pipelines with goroutines and channels. For example, data passes through several processing stages in parallel, which increases performance and code readability.

A simple pipeline example in Go:

func gen(nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        for _, n := range nums {
            out <- n
        }
        close(out)
    }()
    return out
}

func sq(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            out <- n * n
        }
        close(out)
    }()
    return out
}

func main() {
    c := gen(2, 3, 4)
    out := sq(c)
    for n := range out {
        fmt.Println(n)
    }
}

Here, data passes through two stages: number generation and squaring.