Sobes.tech
Middle

How to limit parallelism when making 10 API requests, where the maximum is 3 simultaneous requests?

sobes.tech AI

Answer from AI

To limit parallelism to 3 concurrent requests when making 10 API requests, you can use a semaphore from the asyncio module in Python. A semaphore allows you to restrict the number of coroutines running simultaneously.

Example using asyncio and a semaphore:

import asyncio
import aiohttp

async def fetch(session, url, semaphore):
    async with semaphore:  # limit parallelism
        async with session.get(url) as response:
            return await response.text()

async def main():
    urls = ["http://example.com/api/data" for _ in range(10)]
    semaphore = asyncio.Semaphore(3)  # maximum 3 concurrent requests

    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url, semaphore) for url in urls]
        results = await asyncio.gather(*tasks)
        print(results)

asyncio.run(main())

This way, no more than 3 requests will be executed simultaneously, and the others will wait for the semaphore to be released.