A Kafka platform that enforces its own standards across 200+ teams

The Streaming Marketplace is the control plane for Kafka at Rocket India. I designed and built it from scratch in 2019 and have extended it ever since, incrementally, over about six years. It now governs 5,000+ topics across 200+ teams and is still how Kafka is consumed at the company.

The visible problem was a ticket queue: a new topic or a schema change meant filing a request and waiting days for someone with cluster credentials to action it by hand. That is the part everyone complained about, and it is not the part that mattered.

The real problem was that there were no enforced standards. Naming, retention, partition counts and ownership were conventions written down somewhere and applied by whoever happened to pick up the ticket. At ten teams an inconsistency is an annoyance. At two hundred it means nobody can answer who produces to a topic, what shape the payload is, or who agreed to any of it, and no amount of documentation fixes that after the fact.

• Production had to stay closed. Teams can produce to the lower environments and read messages there, but not in production, because the payloads carry PII. Opening it would have made the self-service story considerably cleaner and it was refused.
• One writer per topic, with no exceptions. Two teams producing to the same topic makes ownership unanswerable and replay unsafe, so the naming and ownership convention has to make it impossible rather than discouraged.
• Least privilege by default. A service account gets exactly the topics it declared, enforced by the tool rather than by whoever reviews the request.
• Whatever replaced the queue had to be safe to hand to 200+ teams with no platform engineer standing between them and a cluster, and operable by whoever is on call, including people who have never seen the internals.

Provisioning validates the request rather than fulfilling it. Naming convention, retention, partition count and whether the requester is even eligible to ask are all checked automatically at request time, so a request that violates a standard cannot be submitted rather than being caught later by a reviewer who may or may not know the rule. A pull request review is the gate before anything goes live, and it is a merge approval rather than a second manual rule check: by the time a human sees it, the rules have already held.

The production lockdown is the decision the rest of the design bends around. Every other environment is genuinely self-service, and production is not: teams cannot produce to it and cannot read messages from it through the platform. That asymmetry costs the platform its cleanest story, and it is the reason a self-service tool can hold PII-bearing topics at all. Single ownership is enforced the same way, through the naming convention, so a topic has exactly one producing service by construction.

The rest accumulated over six years around that spine. The platform also covers topic and service-account analytics; consumer group lag alerts scoped to the owning service account; end-to-end producer to topic to consumer flow tracing, drawn as a graph, to pinpoint exactly where a flow has broken; DLQ monitoring that captures the exception in the record header and notifies the owning team; Flink query management in the lower environments; self-managed Kafka Connect connectors; clusters running on Kubernetes; schema request management including AVRO POJO, POCO and POPO generation; and role-scoped dashboards giving each team topic health, consumer group status, DLQ counts, throughput and alerts for their own slice and nobody else’s.

Reusable Java and .NET SDKs carry authentication, schema validation and observability already wired in, so the correct thing is also the least work. A team gets lag metrics and schema enforcement by using the SDK, not by reading a policy document.

A platform governing 5,000+ topics across 200+ teams, still in production and still being extended six years after the first version. Provisioning went from a multi-day request to a self-serve workflow, but the durable result is that the standards hold: because naming, ownership and access are declared rather than granted ad hoc, the question of who produces to a topic has an answer, and production has never been open enough for a team to put PII somewhere it should not be.

I built the request flows before the observability. Teams could get a topic on day one but could not see their own consumer lag until much later, so the platform team stayed in the loop for exactly the questions self-service was supposed to remove. The visibility should have shipped with the provisioning, not after it.