Copying data into a warehouse to join it is a habit born of necessity,

not design.

Trino queries sources in place. Postgres, S3, Kafka, ClickHouse - one SQL

dialect across all of them, no pipeline in between:

SELECT p.name, c.revenue
FROM postgres.public.customers p
JOIN clickhouse.default.sales c ON p.id = c.customer_id

No ETL ran. Nothing was copied. The join just happened across two systems that

have never met.

This removes an entire category of pipeline - and the failures that came with

it, like the sync job that silently stopped three weeks ago.

The trade is that federated queries are only as fast as the slowest source, so

it is not a replacement for a warehouse. It is a replacement for copying data

you did not need to copy.

What are you federating, and what turned out to be too slow to federate?

Most nightly batch jobs exist because something upstream cannot tell

you when it changes.

So you re-read the whole table every night, diff it yourself, and hope the

window is quiet enough.

Change data capture removes that entire class of job. Debezium reads the

database write-ahead log and emits every insert, update and delete as it

happens.

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "table.include.list": "public.orders",
  "topic.prefix": "cdc"
}

Now the downstream sees changes in milliseconds rather than tomorrow morning,

and you stop writing diff logic.

Two things that bite: the database user needs replication rights, and an

abandoned replication slot will fill your disk. Both are worth knowing before

you turn it on in production.

What did you replace a batch job with, and was it worth it?

The most common data infrastructure mistake is reaching for a cluster

before you have a cluster-sized problem.

A few hundred million rows is a laptop problem. DuckDB will aggregate it in

seconds from Parquet files, with no server, no loading step and no warehouse

bill.

import duckdb
duckdb.sql("""
    SELECT region, sum(amount) AS revenue
    FROM 'sales.parquet'
    GROUP BY region ORDER BY revenue DESC
""")

That is often faster than a warehouse for the same query, because nothing has

to travel.

When does a warehouse actually earn its place? Roughly:

• concurrent users, not one analyst

• data that will not fit on one disk

• retention requirements you have to enforce

• multiple teams querying the same tables

What was the actual size that made you move off files?