workspace_id % N: War stories on building resilience through application-level sharding
October 20–23
At Toggl, we run a multi-tenant analytics pipeline on top of vanilla PostgreSQL, sourcing data via logical replication from OLTP and molding it to meet analytic use cases with a Golang ETL fleet.
For years, we leaned on a single, generously sized Postgres cluster with PARTITION BY HASH (workspace_id) and trigger-driven transformations inside the ingest transaction. It worked, but it came with issues - the biggest one was reliance on sequential data processing. By coupling ingestion and transformation, we made one noisy tenant everybody's problem.
To overcome that, we chose application-level sharding over Citus, FDW, or vertical scaling.
This talk is about war stories and how we came to like our sharding solution after realizing that splitting the ETL pipeline into independent stages improved resilience, but on its own would have pushed us off near-real-time processing unless we threw a humongous amount of resources at the cluster. It's also a story about lean engineering: at every painful turn, the tempting move - and the one most engineering teams reach for - is to defer the problem to an established OLAP solution. We didn't, and this is what we learned by staying on vanilla Postgres.