How to validate an incoming CSV/Parquet file in FastAPI (presence of required columns, value ranges)?

In FastAPI, to validate an incoming CSV or Parquet file, you can:

1. Obtain the file via UploadFile.
2. Read the content into memory (for example, using pandas).
3. Check for the presence of required columns.
4. Verify value ranges in columns.

Example for CSV:

from fastapi import FastAPI, File, UploadFile, HTTPException
import pandas as pd

app = FastAPI()

REQUIRED_COLUMNS = {'id', 'value', 'date'}

@app.post("/upload-csv/")
async def upload_csv(file: UploadFile = File(...)):
    if not file.filename.endswith('.csv'):
        raise HTTPException(status_code=400, detail="Only CSV files are accepted")
    content = await file.read()
    try:
        df = pd.read_csv(pd.compat.StringIO(content.decode('utf-8')))
    except Exception as e:
        raise HTTPException(status_code=400, detail=f"Invalid CSV file: {e}")

    missing_cols = REQUIRED_COLUMNS - set(df.columns)
    if missing_cols:
        raise HTTPException(status_code=400, detail=f"Missing columns: {missing_cols}")

    if not df['value'].between(0, 100).all():
        raise HTTPException(status_code=400, detail="Column 'value' must be between 0 and 100")

    # Additional checks...

    return {"message": "File is valid"}

For Parquet, you can use pd.read_parquet() instead of pd.read_csv(). The main thing is to handle exceptions and check the data structure.