If a local ML model (for example, a reranker) is used in an asynchronous service and is called synchronously, what problem arises and how can it be solved?

If a local ML model is called synchronously in an asynchronous service, it causes blocking of the main request processing thread. This leads to reduced service throughput and increased latency, as while the model processes the current request, other requests wait in line.

To solve this problem, you can:

• Offload the model call to a separate thread or process to avoid blocking the main event loop.
• Use asynchronous wrappers or libraries that support asynchronous model calls.
• Organize a task queue and a pool of workers that will handle model requests in parallel.

An example using concurrent.futures.ThreadPoolExecutor for asynchronous model calls in Python:

import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

def reranker_sync(input_data):
    # call to local ML model (synchronous)
    return model.predict(input_data)

async def reranker_async(input_data):
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(executor, reranker_sync, input_data)
    return result

This approach allows the main asynchronous service to remain unblocked and handle multiple requests concurrently.