Skip to content
Venkataraman ThyagarajanArchitect

Notes on building systems that can’t be wrong.

Four subjects, mostly: event-driven architecture, data platforms at scale, the design consequences of regulation, and where agentic AI genuinely belongs in a platform.

RSS feed

Showing 1–6 of 6 articles

Two horizontal bars comparing modelled monthly cost at 100 GB a day. Self-hosted Kafka is $22K to $38K, mostly engineering time. Confluent Cloud is $4K to $9K. The figures are a model, not a measurement.
Data platforms at scaleConfluent Cloud vs self-hosted Kafka: a TCO modelConfluent Cloud vs self-hosted Kafka: a TCO model built on current list prices, where the engineering cost lands, and when self-managed still wins.May 10, 2026 · 14 min read
Three routes from Kafka to Snowflake: classic Snowpipe at 1 to 5 minutes, Snowpipe Streaming at 5 to 10 seconds, and the managed Confluent Cloud sink, whose latency depends on its settings.
Data platforms at scaleKafka to Snowflake: real-time ingestion patterns at scaleClassic Snowpipe, Snowpipe Streaming or a managed sink: how the three Kafka to Snowflake paths compare on latency, cost and schema handling.May 8, 2026 · 13 min read
A service writes a payment and an outbox row in one transaction. Change data capture publishes the outbox to a Kafka topic, which feeds Elasticsearch, Redis and Snowflake read models. Failed records go to a dead letter topic.
Event-driven architectureFive Kafka patterns for event-driven microservicesFive event-driven patterns with Kafka: when each one fits, how to implement it, and the failure modes that cost teams months in production.May 5, 2026 · 16 min read
Data from 50+ source systems flows through change data capture into three layers: Bronze raw events on Kafka, Silver unified records on Databricks where entity resolution matches customers, and Gold views on Snowflake, serving 1M+ API calls at peak hour.
Data platforms at scaleA customer data platform for a regulated servicerHow I unified 500+ tables from 50+ systems into one real-time customer view serving 1M+ API calls at peak hour, under mortgage servicing constraints.March 10, 2026 · 14 min read
A six-week timeline of two Kafka clusters. The blue public cluster runs throughout. The green Private Service Connect cluster is built in weeks 1 and 2, producers move first in weeks 3 and 4, and consumers move in weeks 5 and 6, with no planned downtime.
Regulated systemsMoving Confluent Cloud Kafka to Private Service ConnectHow we moved a Confluent Cloud Kafka estate from public endpoints to Private Service Connect: dual-write, offset cutover and a cheap rollback.January 20, 2026 · 12 min read
Kafka, which moves data with fan-out and replay, feeds Databricks, which processes it with joins, aggregation and machine learning.
Event-driven architectureKafka vs Databricks: when to use each (decision framework)A decision framework for choosing between Kafka, Databricks, Kafka Connect and JDBC for streaming, batch and CDC ingestion.November 15, 2025 · 15 min read
subject: customer-value · v4 · backwardcompatibility: passing