Junior — Middle
How to implement data collection from three different services, each using its own database?
sobes.tech AI
Answer from AI
To collect data from three different services, each using its own separate database, the following approach can be implemented:
- Create separate modules or classes for working with each database, encapsulating connection and query logic.
- Use a common data aggregation layer that calls each service's methods and combines the results.
- If necessary, implement asynchronous calls for parallel data retrieval to improve performance.
Example in Python using SQLAlchemy and asyncio:
import asyncio
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
# Initialize engines for three databases
engine1 = create_async_engine('postgresql+asyncpg://user:pass@host/db1')
engine2 = create_async_engine('postgresql+asyncpg://user:pass@host/db2')
engine3 = create_async_engine('postgresql+asyncpg://user:pass@host/db3')
async def fetch_data_from_db(engine):
async_session = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
async with async_session() as session:
result = await session.execute("SELECT * FROM some_table")
return result.fetchall()
async def gather_all_data():
results = await asyncio.gather(
fetch_data_from_db(engine1),
fetch_data_from_db(engine2),
fetch_data_from_db(engine3)
)
# Combine data as needed
combined = []
for data in results:
combined.extend(data)
return combined
# Run
all_data = asyncio.run(gather_all_data())
This approach allows isolating work with each database and efficiently collecting data in one place.