Validate rows at ingest

APIs get schemas. Batch files often get hope. A partner CSV lands in an S3 bucket, a job loads it with pandas or csv.DictReader, and three transforms later a null quantity becomes a wrong total — or a string age becomes a silent NaN. The failure is not where the bad cell entered. It is wherever someone finally assumed the column was clean.

Pipelines need the same rule as HTTP: validate at the trust boundary. For files, that boundary is ingest.

A row is a payload

Treat each record like a tiny request body. Declare the shape once, then parse every row through it before it joins joins, aggregations, or feature stores:

from pydantic import BaseModel, Field, ValidationError

class OrderRow(BaseModel):
    order_id: str
    sku: str
    quantity: int = Field(gt=0)
    unit_price: float = Field(ge=0)

def load_orders(rows: list[dict]) -> list[OrderRow]:
    good: list[OrderRow] = []
    errors: list[str] = []
    for i, row in enumerate(rows, start=2):  # 1 = header
        try:
            good.append(OrderRow.model_validate(row))
        except ValidationError as exc:
            errors.append(f"line {i}: {exc.error_count()} issue(s)")
    if errors:
        raise ValueError("\\n".join(errors[:20]))
    return good

Now a bad quantity fails with a line number at load time — not as a mysterious dip in a weekly report. Coercion still helps when producers send "3" as a string; constraints still reject zero and negatives when those are illegal for your domain.

Fail the batch or quarantine rows

Strict jobs should abort on the first invalid file: better a failed run than a half-loaded warehouse. Softer ingest can quarantine bad rows to an error path and continue — but only if that is an explicit product decision, with metrics and alerts. “Skip and hope” is not a strategy.

Either way, keep the schema at the edge. Do not validate “later in the notebook” after ten cleanups have already erased provenance.

Same tool, different boundary

Pydantic here is not about FastAPI. It is about naming the shape of untrusted tabular input and refusing to pretend dict rows are already safe. Downstream code receives OrderRow instances — or typed dumps you chose — instead of a soup of optional keys.

If the file format drifts, the model is the changelog. Update the contract, redeploy the loader, and stop rediscovering broken columns in production queries.

Bad rows break good pipelines. Catch them when they enter.