---
title: Building a Parquet-backed data API, end to end
date: 2026-06-19
description: "An HTTP API over a single Parquet file: polars scan_parquet per request, tests that never touch the real data, and a Railway deploy. Plus a follow-up to my last post, where the page cache that makes local reads fast turns out to be billed as memory."
tags: ["railway", "polars", "fastapi", "parquet"]
author: Rian Dolphin
---

# Building a Parquet-backed data API, end to end

This is a write-up of building a small read-only HTTP API over a single Parquet file and deploying it on [Railway](https://railway.com?referralCode=3pXBGQ). The data is a table of categorized disclosure tags extracted from SEC Form 8-K filings: one row per disclosure, with a filing date, the filing company's identifiers, a three-level category, and a supporting text excerpt. The file is about 44 MB. The API serves filtered, sorted, paginated slices of it with [FastAPI](https://fastapi.tiangolo.com/) and [Polars](https://pola.rs/).

In a [previous post](https://riandolphin.com/blog/dev/s3-parquet-latency-railway) I worked out why I download Parquet files to a container's ephemeral disk at startup rather than reading them from S3 on every request: local reads are about two orders of magnitude faster, and the ephemeral disk is free. This is a different project, but it reuses that idea, and the post below covers the whole build. Most of it was routine. Two parts were not. The first is a testing approach that keeps the large data file out of the test path entirely. The second is a follow-up to that earlier post: the operating system page cache that makes local reads fast is itself billed as memory, which changes the calculation.

## Reading the data on each request

The service reads the Parquet file on every request rather than loading it into memory at startup. polars makes this cheap through `scan_parquet`, which returns a lazy frame: a description of the query that is only executed when you call `collect`. Filters, column selection, and the row limit are pushed down into the scan, so a selective request reads a fraction of the file rather than all of it.

The endpoint builds a list of filter expressions from the query parameters, then hands the lazy frame to a shared runner that applies the filters, sorts, takes one page, and collects:

```python
lf = pl.scan_parquet(parquet_path)

filters: list[pl.Expr] = []
if cik is not None:
    filters.append(pl.col("cik") == cik)
if tickers is not None:
    filters.append(pl.col("tickers").list.contains(tickers))
# ... one block per supported filter

rows, next_url = run_query(
    lf, filters=filters, columns=RESULT_COLUMNS,
    sortable=SORTABLE_COLUMNS, sort=sort, limit=limit,
    cursor=cursor, request=request, default_sort_column="filing_date",
)
```

The response shape is defined with Pydantic models, which gives validation and an accurate OpenAPI schema for free. The envelope carries a status field, a request id, the results, and a `next_url` for the following page when one exists. Pagination uses an opaque base64 cursor that encodes an offset; the runner fetches one row beyond the requested limit to decide whether to emit a `next_url`.

The query parameters follow a dotted convention: `cik.any_of`, `filing_date.gte`, `tickers.all_of`, and so on. These are not valid Python identifiers, so each maps to a function argument through an alias:

```python
filing_date_gte: Annotated[str | None, Query(alias="filing_date.gte")] = None
```

The `limit` parameter defaults to 100 and is capped at 10000 through `Query(ge=1, le=10000)`, so an out-of-range value returns a 422 before any data is read.

## Package structure, and explicit over abstract

The package is organized to mirror the URL hierarchy. A `core` package holds the parts that every endpoint shares: configuration, cursor pagination, the query runner, the response envelope, and later the storage and cache helpers. Each endpoint lives in its own domain package (`stocks/filings_8k/`) with just two files, a router and its Pydantic models. Adding an endpoint is one `include_router` line plus that package.

The filters were the one place I considered an abstraction and decided against it. The operators repeat in a regular way: an exact match, an `any_of` list, a range with four bounds, an array `contains`. A declarative layer could describe each column once and generate both the parameters and the polars expressions. I kept the filters written out instead, one block per operator, for a concrete reason rather than a stylistic one. The query parameters declared in the function signature are what produce the OpenAPI schema and the request validation. A declarative layer that read the raw query string would have to reconstruct both by hand, and would lose the automatic documentation and validation that come from declaring the parameters directly. The repetition is the price of keeping those two properties, and at the current number of endpoints it is a low price.

## Testing without the 44 MB file

The data file is large and not committed to version control. Tests that depend on it would be slow, would not run in continuous integration without extra setup, and would couple assertions to whatever happens to be in the production data. The approach that avoids all three is to give the tests their own data.

The fixture is a small JSON file with a handful of rows, chosen to cover the cases that matter: a company with more than one filing, a filing with two ticker symbols, a row with no ticker, and a spread of dates and categories. A test fixture materializes that JSON into a temporary Parquet file and injects it through a dependency override:

```python
@pytest.fixture(scope="session")
def parquet_file(tmp_path_factory, fixture_rows) -> Path:
    path = tmp_path_factory.mktemp("data") / "disclosures.parquet"
    pl.DataFrame(fixture_rows, schema=_SCHEMA).write_parquet(path)
    return path

@pytest.fixture(scope="session")
def client(parquet_file) -> Iterator[TestClient]:
    app.dependency_overrides[disclosures_parquet_path] = lambda: parquet_file
    with TestClient(app) as test_client:
        yield test_client
    app.dependency_overrides.clear()
```

JSON is the source of truth that a person reads and edits; the Parquet file is what the code actually queries. Building the Parquet from the JSON means the real `scan_parquet` path runs, including predicate pushdown, list-column handling, and null semantics, without depending on the production file being present.

The tests split into two kinds. Some derive their filter values from a sample row and assert invariants: every returned row matches the requested filter, dates come back in descending order by default. These do not break when the data changes. Others pin known facts about the fixture: the total row count, the company that appears twice, the row whose tickers come back as an empty list.

That last case is the one the tests caught. Some rows store the tickers field as a SQL null rather than an empty list. Pydantic's `default_factory` only supplies a value when the field is absent, not when it is present and null, so those rows failed validation. The fix is a validator that runs before parsing and maps null to an empty list:

```python
@field_validator("tickers", mode="before")
@classmethod
def _null_tickers_to_empty(cls, value):
    return [] if value is None else value
```

This is the kind of data-shape difference that the fixture made visible. A row with a null array is easy to include in a six-row JSON file and easy to forget in a schema written from the documented response.

## Cheap pre-commit gates: prek, ruff, ty

The project runs four checks before each commit: lint and format with ruff, type checking with ty, and the test suite. They run through prek, a pre-commit runner, configured as local hooks that call `uv run`. Using local hooks rather than pinned remote ones means the tool versions are exactly those in the project's development dependencies, so there is one place to update them.

Type checking earned its place immediately. ty found a test fixture annotated as returning a `TestClient` when it actually yields one, which should be `Iterator[TestClient]`, and it found a nullable bucket key passed to a function that required a non-null string. ruff's bugbear rules flagged a re-raise inside an `except` block that should chain explicitly with `raise ... from None`. None of these were caught by running the program; all three were caught before commit.

A justfile sits alongside as the task runner for the same commands run by hand: `just fmt`, `just check`, `just test`. prek owns the commit-time trigger; the justfile is the manual entry point.

One behavior of prek is worth knowing before it surprises you. When you commit a subset of files, prek stashes unstaged changes to tracked files so the hooks see only what is staged, but it leaves untracked files in place. If your unstaged work includes a new module that an untracked test imports, the stashed-away state is inconsistent: the test imports a symbol that the reverted module no longer defines, and the hook fails. The fix is to stash the in-progress work yourself, by path, so the working tree is self-consistent when the hooks run.

## Deploying on Railway: an S3 sync gated by health

Railway builds the repository with Railpack, its current build system; Nixpacks, the previous one, is in maintenance mode. Railpack detects the uv project from the lock file and runs `uv sync --locked --no-dev`, so the lock file must be current or the build fails, and runtime dependencies must not sit in a development group.

The data file is stored in an S3-compatible bucket. On startup the service downloads it to local disk, so that `scan_parquet` reads from local storage rather than over the network on every request. The download takes a few seconds, and during those seconds the file is not yet present. The `/health` endpoint reports this:

```python
@asynccontextmanager
async def lifespan(app):
    app.state.ready = False
    if settings.s3_sync_enabled:
        asyncio.create_task(_sync_data(app, settings))
    else:
        app.state.ready = True
    yield

@app.get("/health")
def health(request):
    if getattr(request.app.state, "ready", False):
        return {"status": "ok"}
    raise HTTPException(status_code=503, detail="Warming up.")
```

The sync runs as a background task so the server can answer health checks (with 503) while the download is in progress. Railway's deployment healthcheck holds back the new deployment until `/health` returns 200, and keeps the previous deployment serving until then. The result is that a deployment never takes over until its data is in place, even though the copy takes time. The download writes to a temporary file and renames it into place, so a partially downloaded file is never read, and it retries a few times before giving up; if it never succeeds, readiness stays false and the previous deployment continues to serve.

## Ephemeral storage, and the cost of memory

The next decision was where the downloaded file should live. Railway offers a persistent volume and an ephemeral container disk. The volume survives across deployments; the ephemeral disk is wiped on each one.

The service re-downloads the file from the bucket on every deployment, so persistence across deployments is not needed. Downloads from a Railway bucket to a service are free, so re-fetching costs nothing in transfer. A volume costs 0.15 USD per GB per month, and it binds the service to a single replica: a service with a volume cannot be scaled horizontally. The ephemeral disk has none of these constraints, and each replica downloads its own copy at startup. The ephemeral disk was the better fit on every axis that applied.

Then the part I did not account for in the earlier post. Reading a file from local disk populates the operating system's page cache, the in-memory copy of recently read file pages. That cache is what makes the local reads fast, and last time I left it there. Reading more carefully into how Railway bills memory complicates the picture: the billed memory metric includes this cache. This is confirmed by Railway staff on their support forum, though it is not stated in the formal pricing documentation, and there are reports of services billed for several gigabytes of cache built up by reading large files. The consequence is that the file's footprint in the page cache is billed as memory, even though page cache is reclaimable and the kernel will drop it under pressure without the service running out of memory.

The amount matters more than the mechanism. At 44 MB and a memory rate of 10 USD per GB per month, the file's cache footprint is about 0.43 USD per month. That is small. The reason to understand it is that it scales linearly with the file: a 5 GB file fully cached is about 50 USD per month, at which point both the cost and the latency of scanning a multi-gigabyte file per request would justify a different design.

There is a useful way to think about the trade-off. You pay for the file's memory footprint however you make reads fast. The page cache is the cheapest version of that footprint, because it is reclaimable: under memory pressure the kernel drops it and the service does not crash. Loading the whole file into an in-memory frame at startup would cost the same memory or more, would not be reclaimable, and would risk running out of memory. Reading from the bucket on every request avoids the cache entirely but pays network latency on each call. For a service that is read often, the page cache is the right default and the small recurring cost is the price of fast reads.

## A tangent I'm not sure about: evicting the cache when idle

This part is exploratory, and I want to be upfront that I have not convinced myself it is worth doing. It came out of going back and forth with Claude on the page-cache question, and it was interesting enough to write down even though I have not run it in production.

If a service is read rarely, the page cache buys nothing between requests, and it can be reclaimed. The targeted tool is `posix_fadvise` with `POSIX_FADV_DONTNEED`, which drops one file's pages from the cache. It is scoped to that file's inode: it does not touch any other file's cache, the process heap, or shared library pages. (The blunt alternative, dropping the entire system cache, is the thing to avoid.)

Doing it without hurting the request path took some thought. The endpoint is synchronous, which Starlette runs in a thread pool, so several scans can run at once, and evicting the cache while a scan is reading the file would just send that scan back to disk. The version I landed on is a debounced evictor: each request stamps an activity time, and a background task drops the cache only after the service has been idle for a set interval with nothing in flight. The decision to evict is claimed under the same lock as the in-flight counter, so the cache is not dropped between the check and the drop while a request is starting.

Where I am unsure is whether any of this earns its place. At 44 MB the cache costs about 0.43 USD per month, so eviction saves cents while adding a background task, a lock, and a new failure mode to reason about. That is a poor trade at this size. It only starts to look worthwhile if the file grows large enough that the cache is a real line item, and even then, reading from the bucket directly or letting the platform stop the service when idle might be simpler. So it is off by default, and it is here because it sounded interesting, not because I am recommending it.

## What I took away

The storage and memory model should match the traffic. A frequently read service keeps the file local and warm and pays a small, predictable amount for the cache. A rarely read service has cheaper options: read from the bucket directly and pay latency instead of memory, stop running between requests on a platform that bills idle compute, or drop the cache when idle (the tangent above, which I am still unsure about). The file size is the variable that decides when the cheap default stops being cheap.