I work where correctness and speed are in tension.
I design event-driven data platforms for regulated financial services. Eight years in, most of that spent in mortgage servicing and adjacent parts of US finance, building systems where the data has to be correct, provable after the fact, and available in tens of milliseconds.
I like this problem space because the constraints are real and non-negotiable. In most software you can trade correctness for speed quietly and nobody notices for a year. In regulated finance the trade is explicit, someone will audit it, and you have to be able to defend the choice you made. That pressure produces better architecture than any amount of design review.
Right now I am working on where agentic systems genuinely belong in a data platform. My position has got narrower the more of them I have shipped, which I take as a good sign.
Track record
Sep 2026-present
Apr 2025-Sep 2026
- Migrated the Kafka platform to private connectivity using Confluent PSC, an incremental, zero-data-loss rollout across every environment, delivered ahead of schedule.
- Architected a real-time data exchange platform for secure, low-latency sharing across distributed systems, using Kafka cluster-to-cluster streaming.
- Built platform-wide streaming analytics covering topic utilisation, data volumes and peak loads, for proactive capacity planning.
- Led the Data Modernization initiative, consolidating 100+ data sources into a centralised Snowflake platform.
- Extended the Streaming Marketplace with a per-project lineage dashboard that traces a producer-to-consumer break to the exact failing service.
- Mentored engineers and drove alignment across engineering, infrastructure and business teams.
Jun 2021-Apr 2025
- Directed a team of 5 engineers and 2 QA building the Customer Data Platform, consolidating 50+ sources and 500+ tables into a unified Kafka, Snowflake and Databricks platform.
- Implemented Medallion architecture, bronze, silver and gold, on Snowflake and Databricks Spark, and restructured the gold layer to cut a common downstream query from about three seconds to roughly 500 milliseconds for 15+ teams.
- Led the technical side of post-acquisition data integration with Rocket Companies under a regulatory mandate, using asymmetric encryption for PII exchange across roughly 20 to 50 topics between the two entities.
- Led the Azure to GCP migration of close to 200 stateless services, driven by cost reduction, using a lift-and-shift approach.
- Developed parallel consumer logic in Java and .NET, and enforced PI and NPI encryption across MongoDB and Confluent environments.
Jun 2019-Jun 2021
- Built the Streaming Marketplace from scratch: a self-service control plane for Kafka that still governs 5,000+ topics across 200+ teams.
- Built reusable Java and .NET Kafka libraries with authentication, schema validation and observability already wired in, adopted across the organisation.
- Delivered Docker-based Kafka Connect images that turned ingestion provisioning from a multi-day request into a self-serve workflow.
- Implemented real-time alerting on consumer lag and record-read failures.
Jun 2018-Jun 2019
- Designed and delivered Scout, an investor analytics portal giving real-time visibility into loan performance and portfolio health, replacing manual executive reporting.
Dec 2022-present
- Architected and still own a two-sided hiring platform built from zero: 9 production services on AWS, serving 1,000+ employers and 20,000+ jobseekers, with 3 engineers I mentor.
- Designed the matching service so all agentic work runs offline: a six-agent pipeline generating 1536-dimension embeddings into vector search, with the live path making zero model calls by design.
- Built fan-out notifications on change streams with backoff, crash recovery and deduplication. Hourly batching cut Slack traffic from 500 messages an hour to one.
- Own platform engineering and security, including the hardening that followed a 188,000-request injection campaign.
What I work with
Streaming
Apache Kafka and Confluent Platform · Confluent Flink · Kafka Streams · Kafka Connect · Schema Registry · Confluent PSC · Debezium and change data capture · MongoDB change streams
Data
Snowflake · Databricks · Apache Spark · Delta Lake · PostgreSQL · MongoDB, including Atlas Vector Search · Elasticsearch
Cloud and infrastructure
GCP · AWS · Azure · Kubernetes · Docker · Terraform · Helm · GitHub Actions
Backend
Java · Spring Boot · Spring WebFlux · Python and FastAPI · .NET · Node.js · REST and GraphQL
AI systems
LangGraph · vector search · embedding pipelines · agentic workflows kept off the critical path
Observability
New Relic · Splunk · structured logging · alerting · consumer-lag SLOs
Frontend
React · Next.js · TypeScript · Material UI
Education
Common questions
Who is Venkataraman Thyagarajan?
I design event-driven data platforms for regulated financial services. They have to be correct, auditable, and fast at the same time. Eight years building streaming platforms on Kafka and Confluent, most of it inside regulated environments.
What technologies does Venkataraman Thyagarajan specialise in?
Apache Kafka and Confluent Platform, Kafka Streams, Debezium and change data capture, and the schema governance and replay semantics that decide whether a streaming estate can be safely reprocessed. Alongside those: Snowflake, Databricks, GCP and AWS, and Java and Spring Boot on the service side.
What has Venkataraman Thyagarajan built professionally?
I led the customer data platform at Rocket India, directing a team of 5 engineers and 2 QA to unify 500+ tables from 50+ source systems into a real-time view serving 1M+ API calls at peak hour. Restructuring the Snowflake layer’s gold tables cut a common downstream query from about three seconds to roughly 500 milliseconds for 15+ teams. Before that I built the streaming marketplace that now governs 5,000+ Kafka topics across 200+ teams, including a per-project dashboard that maps the full data lineage and flags exactly where a link has broken. More recently I moved the whole estate to private networking incrementally, team by team, with zero data loss throughout.
What kind of work does Venkataraman Thyagarajan take on?
Architecture problems in regulated or high-scale environments: streaming platforms, ordering and replay semantics, and migrations that cannot take downtime. I also take speaking invitations and technical writing collaborations.
Where is Venkataraman Thyagarajan located?
I am based between Coimbatore and Chennai, in Tamil Nadu, India, and I work in IST.
What is heyvenkat.com?
This site is where I write about event-driven architecture, data platforms at scale, and regulated systems, alongside written-up case studies of the engagements behind that writing. It is the public artefact of the work, not a portfolio of it.
All of this shows up somewhere. The case studies are where the constraints are written down.
See the work