Sobes.tech
Junior

Optimization of bulk HTTP requests

livecode

Task condition

It is necessary to adapt the implementation of parallel HTTP requests to a large number of URLs to avoid excessive resource consumption. It is required to limit the number of requests executed simultaneously.

func printCodes(urls []string) {
    var wg sync.WaitGroup
    wg.Add(len(urls))

    get := func(url string) {
        resp, err := http.Get(url)
        if err != nil {
            log.Fatal(err)
        }
        defer resp.Body.Close()
        fmt.Println(url, resp.Status)
        defer wg.Done()
    }

    for _, url := range urls {
        url := url
        go get(url)
    }

    wg.Wait()
}