A Million Rows, 30 Milliseconds a Frame

Most analytics platforms are fast at loading and slow at thinking. You wait for the dashboard, then every filter you touch makes you wait again.

This is a million taxi trips. Drag a box on either map. The other charts re-aggregate before your hand stops moving — no loading state, no spinner, no backend round trip per frame.

Every frame of that drag re-aggregated a million rows into three charts in 28–77 ms. The architecture underneath is three pieces you already trust, arranged in an order most teams haven't tried: Parquet for storage, DuckDB for the engine, Mosaic as the coordinator.

This piece is about that arrangement — what it costs, where it stops working, and why a platform team can build it in a week instead of buying it.

The problem: the interaction budget

Take the obvious approach. The brush moves, every chart re-runs its query, each query scans the table. Priced honestly on first load, before any index exists:

About 300 ms for one frame. Your hand generates pointer events every 8–16 ms. You fall behind on frame one and never catch up.

Worse, it's O(N). Ten million rows means ten times the wait, forever. Every "make the dashboard faster" project that ends in more indexes, more caching layers, or a bigger warehouse cluster is fighting this exact curve — and paying per query to lose.

The fix is not a faster scan. It is not scanning during the interaction at all.

Three pieces

Parquet: the storage format, not just an export target

Parquet is columnar, compressed, and carries statistics per row group. Two properties matter here:

• Projection pushdown — a query touching 4 of 30 columns reads 4 columns off disk. Row-oriented storage reads all 30.
• Predicate pushdown — row-group min/max lets the engine skip entire blocks without decompressing them.

A million trips with a dozen columns is 46 MB as Parquet. The same data as CSV is several hundred. That difference is the whole reason this page loads.

DuckDB: an engine that reads Parquet natively

DuckDB queries Parquet directly — no load step, no ingest job, no copy into a proprietary format:

SELECT count(*), avg(fare)
FROM 'trips.parquet'
WHERE hour BETWEEN 17 AND 19

It's vectorized and columnar end to end, so aggregating a million rows is milliseconds on one core. It runs as a library inside your server process, in a notebook, or — as on this page — compiled to WebAssembly inside the browser tab.

Same SQL, same engine, three deployment targets. Hold that thought.

Mosaic: the coordinator

This is the piece most data teams haven't met, and it's the one doing the interesting work.

Mosaic sits between your views and DuckDB. Every chart registers as a client. Every filter is a selection. Nothing queries the database directly — the coordinator does, on everyone's behalf.

That indirection buys four things no individual chart could arrange for itself:

• It resolves one selection into one predicate, shared by every view.
• It deduplicates and queues concurrent queries.
• It caches identical statements.
• And — the reason we're here — it decides when to build a cube.

The whole system in one picture

Two phases, and the split between them is the entire trick.

Phase 1: the cube is built on hover

When your pointer enters a chart — before you press anything — Mosaic materializes index tables. One per view that can use one. Here that's three CREATE TABLE statements, 62 / 92 / 146 ms.

You never feel it, because you are still moving your hand toward the thing you intend to click.

Here is what one of those tables actually is:

CREATE TABLE mosaic.preagg_5e8541db AS
SELECT floor(time)     AS x1,               -- the histogram's own buckets
       floor(time) + 1 AS x2,
       count(*)        AS pre_e87fc4ab,     -- the partial answer
       floor(0.01305 * (px - 970000))::INT AS active0,  -- brush position, in pixels
       floor(0.01306 * (py - 188000))::INT AS active1
FROM trips
GROUP BY x1, x2, active0, active1

Read it as a grid: the chart's own buckets crossed with every position the brush could possibly take, with the answer already computed for each cell.

If you've built OLAP cubes, this will feel familiar — because it is one. The difference is that nobody designed it. Mosaic derived the cube's shape from the chart's encoding and the interactor's geometry, then built it during a gesture the user doesn't perceive as waiting.

Dragging then becomes arithmetic, not aggregation — sum the cells whose brush position falls inside the current box:

SELECT x1, x2, sum(pre_e87fc4ab) AS y
FROM mosaic.preagg_5e8541db
WHERE active0 BETWEEN 214 AND 287
  AND active1 BETWEEN 122 AND 190
GROUP BY x1, x2

No coordinates recomputed. No million rows scanned. A range-sum over a table built for this exact gesture.

Phase 2: three queries per frame

The figure has four views: the pickups map, the dropoffs map, the hour histogram, and the KPI tiles. While you drag the pickups map, three queries run per frame:

Three things here surprise people, and each is a design lesson.

The map you're dragging isn't in the list. A view is filtered by every selection except its own — otherwise your brush would erase the context you are selecting against. Dragging the pickups map costs nothing on the pickups map.

Four tiles is one query, not four. The unit of work is the component, not the number on screen:

SELECT count(*) AS trips, avg(distance) AS distance,
       avg(fare) AS fare,  avg(riders)   AS riders
FROM trips

The three charts stay three queries, though, because they ask genuinely different questions — different GROUP BY, different measures, 24 buckets vs 59,740 pixels vs 1 row. No single result set answers all three without shipping raw rows to the client and aggregating there, which is the design this exists to avoid. Three queries is correct. What changed is which table they read.

Cost barely tracks result size. The histogram returns 24 rows and costs more than the map returning 59,740. What matters is the size of the cube, not the size of the answer.

Only the filtered data crosses the wire

Look at the bytes column again: 32 B, 891 KB, 576 B.

That is the enterprise headline. The engine sits next to the data — 46 MB of Parquet — and what leaves it is the aggregate, sized by the chart's resolution, not the table's row count. The million rows never move.

This inverts how most BI tools behave, where the client pulls a result set and does the last mile of work itself, and payload grows with your data.

Notice too that the figure narrates itself: a status line reporting whether data came from cache or network, per-query timings, the SQL behind every box. Hint text is not decoration. When a number moves and the user can see why, they trust it; when it moves silently, they open a ticket. It's the cheapest trust feature you will ever ship, and the first one cut from most internal tools.

Same architecture, two deployments

This page fetches Parquet from a CDN, caches it in the browser, and runs DuckDB-WASM in the tab. Zero backend. That's one deployment of the pattern, and it's the right one for a surprising amount of internal tooling.

The architecture doesn't change when the data outgrows a tab. Mosaic ships three connectors:

Your charts, selections and cubes don't know which one you picked. Views talk to the coordinator; the coordinator talks to a connector. Swapping transports is a line of configuration, not a rewrite.

So a team can prototype entirely in-browser against a sampled Parquet extract, then move the same code to a server-side DuckDB reading from object storage when governance or volume demands it. The cube strategy, the three-queries-per-frame shape, and the payload sizes are identical. Only the hop changes.

The enterprise shape: snapshot, then serve

Most enterprise analytics is not asking questions of live transactional state. It's asking questions of a period that has closed — yesterday, last week, last quarter. That distinction is where this architecture earns its keep.

Postgres / Oracle / Snowflake        ← system of record, stays authoritative
        │
        │  scheduled snapshot  (nightly · weekly · quarterly)
        ▼
    Parquet on object storage        ← immutable, partitioned, cheap
        │
        ▼
  DuckDB + Mosaic coordinator        ← every interaction served from here

What that buys:

• The OLTP database stops absorbing analytical load. Every drag on a dashboard is a query your transactional system never sees.
• Snapshots are immutable, so results are reproducible. "The Q3 number changed since Tuesday" stops being a conversation.
• Cost becomes a schedule, not a meter. You pay for one extract on a cadence you chose, instead of per-query against a warehouse billing by the second while a user fiddles with a filter.
• Partitioning is just file layout — year=/month=/region= directories, and DuckDB prunes them without a metastore.

Match the cadence to the question. Finance reporting: quarterly. Ops review: weekly. Sales pipeline: nightly. The interaction budget is the same 30 ms in every case, because a snapshot's freshness has nothing to do with a cube's speed.

Why a team should build this

For small and mid-sized use cases, the honest comparison isn't DuckDB-and-Mosaic versus a better dashboard. It's versus a platform licence, a semantic layer you don't control, and a roadmap you can't influence.

What building it yourself gets you:

• The visual grammar is yours. Your charts, your interactions, your empty states, your copy — not the six widgets a vendor exposes.
• Domain logic lives in SQL you own — your definition of an active customer, your fiscal calendar, your exclusions. Not in someone's modelling UI.
• It embeds where the work happens. Inside your internal app, next to the record the user is actually looking at, not behind a separate login.
• No per-seat cost, so analysis goes in front of everyone who'd benefit rather than everyone who can justify a licence.
• It's inspectable. Every query is SQL you can read, log, test and defend in a review.

The reason this is now a week of work rather than a quarter is that the hard parts — the engine, the format, the coordination — are solved and free. What's left is the part only you can do: deciding what your users should be able to ask.

When to reach for this, and when not to

Size it in rows and columns, not megabytes.

Above 10 M it isn't the queries that break — it's the load. Nobody should pull that into a tab before seeing anything.

Two rules worth keeping:

• Rows are cheap, columns are not. More rows cost load time and memory but not per-frame latency — the cube is sized by chart resolution, not row count. More projected columns cost you everywhere at once.
• Only the columns your charts touch count. Parquet reads what you select, so a 200-column fact table where the dashboard uses twelve behaves like a twelve-column table.

The trade underneath fits in one line: pay one large cost up front, at a moment the user isn't waiting, to make every subsequent question free.

What I'm building next

The snapshot cadence above is currently a human decision — someone chooses nightly or weekly and writes a cron entry. That's the obvious place for an agent.

I'm exploring a planner that looks at a question and decides: is this ad hoc, or is it becoming a workload?

A one-off exploration runs against the source and costs what it costs. But if the same question is asked forty times a week, the system should recognise that pattern and consider turning it into a scheduled extract — with its own Parquet snapshot and potentially its own pre-aggregated representation.

The agent wouldn't execute that optimisation. It would detect the pattern, propose the configuration, estimate the cost/benefit, and justify the decision against a defined budget — leaving the commit to a human.

Cost-aware query planning, one level above the optimizer. That's the next post.

Coordinator architecture per Heer & Moritz, Mosaic: An Architecture for Scalable & Interoperable Data Views (TVCG 2024). Data: the first million yellow-cab trips of 2010, from the Mosaic example datasets. Borough boundaries from NYC Department of City Planning, reprojected to NY State Plane. Every timing here was measured in-browser on a desktop machine — yours will differ, which is rather the point of shipping the instrument instead of the claim.