Kafka vs Databricks: When to Use Each (Decision Framework)

The Question That Comes Up in Every Architecture Review

Spend enough time in data engineering and you will inevitably sit in a meeting where someone asks: "Do we actually need Kafka here, or can Databricks just read it directly?" I've been in that meeting dozens of times. The answer is never obvious from a whiteboard, and getting it wrong costs real money — typically six figures over a year once you account for infrastructure, operational overhead, and engineering hours spent fighting an architecture that was never right for the workload.

This article is the framework I wish I had earlier. It covers when each technology excels, where each one fails, and how to make the call quickly — with concrete criteria rather than gut feel.

First, Get the Mental Model Right

These tools are not competitors. They solve different problems in the same data stack. Kafka is a distributed commit log — a durable, ordered transport layer for events. Databricks is a compute platform built on Apache Spark — designed for transformations, aggregations, and machine learning at scale. The typical production architecture has them working together:

Source Systems → Kafka → Databricks → Data Warehouse → Applications

Kafka moves data. Databricks processes it. Confusion arises because Databricks can technically consume from a database directly (via JDBC), and Kafka can technically do lightweight transformations (via Kafka Streams). Both capabilities exist. Neither is a replacement for what the other does well.

The third piece of the puzzle is ingestion connectors — specifically the difference between CDC-based and polling-based approaches — which changes the cost and reliability equation significantly.

When Kafka Is the Right Choice

Kafka's core value proposition is not throughput. It's durability plus fan-out plus replay in a single system. Those three properties together are what no other component in a typical data stack provides.

Sub-second to low-second latency requirements. Databricks Structured Streaming runs on micro-batch triggers. In practice, a well-tuned Databricks streaming job has end-to-end latency in the range of 5 to 30 seconds, depending on trigger interval and cluster warmup. Kafka brokers, by contrast, deliver messages to consumers in 5 to 200 milliseconds end-to-end. If a fraud detection model, a real-time recommendation system, or a live alerting pipeline needs results in under five seconds, Kafka is not optional — it's the only viable transport layer in the stack.

Multiple independent consumers on the same data stream. Kafka's consumer group model means ten services can independently read the same topic at their own pace, each maintaining their own offset. With direct database reads or file-based ingestion, adding a new consumer means adding another query against the source, another pipeline to maintain, another point of failure. With Kafka, adding a consumer is adding a consumer group — no upstream changes required.

Event replay and audit requirements. Kafka retains the log for a configurable window — hours, days, or indefinitely with tiered storage. When a downstream job fails, a new consumer comes online, or a bug is discovered in processing logic, you can replay from any offset. This is not possible with JDBC ingestion or most file-based approaches once the source data changes.

Event-driven architecture with decoupled producers and consumers. If your platform has more than three services that need to react to the same business events — order placed, payment processed, user registered — Kafka is the right distribution mechanism. Direct service-to-service calls or shared database writes create coupling that becomes increasingly expensive to unwind as the system grows.

Kafka's performance characteristics: 100K to 2M messages per second per broker, P99 end-to-end latency of 5 to 200 milliseconds in well-tuned clusters, near-linear horizontal scaling by adding brokers and partitions. Storage retention is bounded by disk, not by some architectural limit.

When Databricks Is the Right Choice

Databricks earns its place in the stack when the work to be done is genuinely computational: complex joins, aggregations across large windows, ML feature engineering, medallion architecture transformations, or anything that requires multi-step logic across historical data.

Complex transformations and aggregations. Kafka Streams can filter, map, and do simple stateful aggregations. It cannot do a nine-way join across a 2TB historical dataset while running a feature engineering pipeline for a recommender model. Spark can. Databricks is the right answer for any transformation that requires more than a few operations or touches data at TB scale.

Batch-oriented workloads with hourly or daily schedules. If a business report runs at 6 AM and nobody is waiting on it at 6:01 AM, there is no business case for a streaming pipeline. Databricks batch jobs are significantly cheaper to run and simpler to operate than continuous streaming infrastructure for the same workload. The cost delta between a daily Databricks job and a 24/7 Kafka cluster plus streaming consumer for the same use case is typically 30 to 50 percent.

Delta Lake and the lakehouse pattern. ACID transactions, time travel, schema evolution, and Z-order indexing are Delta Lake capabilities that make Databricks the right home for curated data layers. The medallion architecture — raw Bronze, validated Silver, aggregated Gold — is a proven pattern that Databricks implements well. Kafka is not involved in this layer.

Machine learning pipelines. Feature engineering at scale, model training on historical data, batch inference — all of this belongs in Databricks. Kafka's role ends when the events are delivered. Databricks takes over when the computation begins.

Typical Databricks performance: processing throughput of 1 to 100+ TB per hour depending on cluster size, configurable Structured Streaming trigger intervals from 500ms upward (with 5 to 30 seconds being practical for production stability), and horizontal scaling to 1,000-node clusters for large-scale batch work.

The Ingestion Question: CDC vs JDBC

Once you decide whether Kafka is in the path, the next question is how data gets into the pipeline from relational source systems. This is where teams make expensive mistakes most often.

Log-based CDC with Kafka Connect and Debezium reads the database transaction log directly. Every insert, update, and delete is captured as an event in a Kafka topic with sub-second latency from the original commit. There is no query load on the source database during steady-state operation. Deletes are captured reliably. This is the right approach for production databases with high write rates, tables larger than 50GB, or workloads requiring true real-time replication. The operational cost is higher — you need to configure and maintain Debezium connectors, ensure database binlog access, and manage schema evolution through Schema Registry — but the reliability and efficiency advantages are substantial at scale.

Kafka JDBC Source Connector polls the source database on a configurable interval, executing a SQL query to fetch rows where an incrementing column or timestamp has advanced. It is simpler to set up and works without transaction log access. The trade-offs are meaningful: polling creates periodic load on the source, delete events are not captured, and latency is bounded by the poll interval (typically 5 to 15 minutes in production configurations that avoid overwhelming the source). The JDBC connector is appropriate when multiple downstream consumers need the same database data buffered in Kafka and when near-real-time latency (minutes) is acceptable.

Databricks JDBC direct reads connect a Spark cluster directly to the source database and read on-demand during job execution. With parallel partitioning, throughput can reach 100K+ rows per second. There is no intermediate buffering, no replay capability, and no fan-out — if three teams need the same data, each team runs their own query. For single-consumer batch workloads with hourly or daily schedules, this is the simplest and often cheapest architecture. It becomes the wrong choice when the database cannot sustain the query load, when multiple consumers exist, or when the latency requirement tightens below the job schedule interval.

Both JDBC approaches put query load on source databases. For production OLTP systems with significant write rates or tables above 100GB, log-based CDC via Debezium is the right long-term approach. The short-term cost of setup is lower than the long-term cost of polling at scale.

Decision Matrix

The following table summarises which approach fits which requirement. "Recommended" means this is the natural fit. "Viable" means it works with trade-offs worth understanding. "Avoid" means the architecture fights the tool's design.

What This Costs: A Realistic TCO Comparison

Cost estimates vary significantly by cloud provider, region, instance types, and team configuration. The figures below are directional, based on AWS/GCP pricing, and meant for order-of-magnitude planning rather than budgeting.

Self-managed Kafka on cloud VMs: A three-broker cluster on m5.xlarge instances for a 100GB/day workload runs approximately $1,500 to $4,000 per month in infrastructure alone. At 1TB/day you are looking at $8,000 to $18,000 per month. The infrastructure cost is not the whole story — self-managed Kafka requires 1.5 to 3 dedicated engineers to operate reliably. At $150K to $200K per engineer per year, that operational cost typically exceeds the infrastructure cost for medium-scale deployments.

Confluent Cloud (managed Kafka): Roughly $400 to $1,500 per month at 100GB/day on the Basic tier, scaling to $4,000 to $12,000 per month at 1TB/day on Standard or Dedicated clusters. The engineering leverage is meaningful: managed Confluent reduces operational overhead to approximately 0.5 FTE versus 2 to 3 FTE for self-managed. Whether that delta justifies the premium depends on your team's size and expertise.

Databricks: At $0.07 to $0.65 per DBU plus underlying cloud compute, batch workloads at 100GB/day typically run $800 to $2,500 per month. At 1TB/day with continuous streaming jobs and auto-scaling, expect $4,000 to $12,000 per month. Cloud storage for Delta Lake adds roughly $23 per TB per month at standard tier.

The single most common cost mistake: teams spinning up Kafka for workloads where hourly Databricks batch jobs would suffice. The unnecessary streaming infrastructure adds $30,000 to $100,000 per year at medium scale while providing no business value for the latency profile of the workload. If your reporting dashboard refreshes every 15 minutes and your users would not notice the difference between 5-second and 10-minute staleness, that is a batch workload.

Five Patterns That Get Teams Into Trouble

Using Databricks as a low-latency event transport. Databricks Structured Streaming is a micro-batch engine. Its practical minimum trigger interval in production — accounting for cluster warmup, job scheduling, and checkpoint writes — is 5 to 30 seconds. Teams that configure a 500ms trigger interval and expect Kafka-like latency are consistently surprised. The trigger is not the only source of latency. If a job needs to emit events to downstream systems in under two seconds, Databricks is not the right transport. Use Kafka for delivery; use Databricks for processing after the fact.

Using Kafka for complex transformations. Kafka Streams handles stateless and simple stateful transformations well. Multi-way joins across large datasets, ML inference pipelines, and complex windowing operations against historical data are not what Kafka Streams is designed for. Teams that push this logic into Kafka end up with difficult-to-debug topology graphs and state store management problems that would not exist in Databricks. Move complex logic downstream.

Running JDBC full-table scans on production databases. A full scan of a 200GB production table via JDBC creates substantial I/O and CPU load on the source system. Scheduled at 2 AM this is manageable. Scheduled hourly it degrades production traffic. Scheduled every 15 minutes it causes incidents. Log-based CDC reads the transaction log rather than querying the table — it has near-zero impact on the source at steady state and captures deletes, which polling cannot.

Building separate pipelines per consumer instead of using a shared topic. When three teams each build their own JDBC job to pull from the same source table, you have three queries hitting that table, three pipelines to maintain, three different latency profiles, and three different failure modes. One Kafka topic with three consumer groups is strictly better: one query (or one CDC connector) on the source, identical events for all consumers, independent offsets per consumer group, and a single place to monitor ingest.

Introducing Kafka because "we might need real-time someday." Kafka adds operational complexity. Running a production Kafka cluster well requires understanding partition rebalancing, consumer lag monitoring, schema evolution, connector management, and retention policies. That is real work. If today's requirements are hourly batch and next year's might be five-minute near-real-time, build the batch pipeline. Migrating from Databricks batch to Kafka-backed streaming later is a predictable, well-documented exercise. Paying for streaming infrastructure for a year before you need it is waste.

Six Scenarios With Concrete Recommendations

Scenario: Real-time database replication to a data lake. You need database changes reflected in your lakehouse within seconds. The recommendation is Kafka Connect with Debezium for CDC, writing to Kafka topics, consumed by Databricks Structured Streaming into Delta Lake Bronze. Debezium's log-based CDC captures every change with sub-second latency and near-zero database overhead. Databricks handles the transformation and merge logic into your medallion layers.

Scenario: Daily log processing for BI dashboards. You have 500GB of daily application logs that need to be transformed for a business intelligence layer. The recommendation is Databricks only — batch jobs scheduled during off-peak hours, writing to a Delta Lake Gold table that the BI tool reads. Adding Kafka here would add cost and complexity without improving the outcome for users who refresh dashboards once a day.

Scenario: SQL Server to Snowflake every four hours, single consumer. One team, one destination, four-hour cadence. The recommendation is Databricks JDBC direct reads with partitioned parallel reads into Snowflake via the Spark Snowflake connector. Simple, cheap, easy to maintain. No Kafka required.

Scenario: Payment event fraud detection under one second. The fraud model needs to evaluate each payment in under 500 milliseconds. The recommendation is Kafka as the transport layer, with a lightweight fraud scoring service consuming the payments topic. Databricks is not suitable here — its micro-batch engine cannot guarantee sub-second delivery consistently.

Scenario: User clickstream for ML feature engineering. Clicks need to be captured in real-time and made available for both real-time personalisation and offline model training. The recommendation is Kafka for event capture and distribution, with a real-time consumer for live personalisation and Databricks consuming from Kafka for batch feature engineering into a feature store. Both paths consume from the same Kafka topic without duplicating ingestion work.

Scenario: Schema evolution with audit requirements. A regulatory workload requires that every version of every event be preservable, and the schema will evolve as the business changes. The recommendation is Kafka with Confluent Schema Registry. Avro or Protobuf schemas registered in Schema Registry, Kafka's log retention providing the historical record, and consumer-side schema evolution handling backward compatibility gracefully.

The Decision in Four Questions

When the architecture debate starts, these four questions cut through most of the ambiguity:

1. What is your actual P95 latency requirement? Not what you might need eventually — what the business needs today, measured at the consumer end. If it's above five minutes, you have a batch workload. If it's under five seconds, you need Kafka in the path.

2. How many independent consumers will read this data? If the answer is one, JDBC or direct ingestion is probably sufficient. If the answer is three or more, Kafka's fan-out model pays for itself quickly in reduced source database load and eliminated pipeline duplication.

3. Do you need to replay events? If a consumer fails and you need to reprocess the last 48 hours of events, Kafka's log retention gives you that for free. JDBC and file-based approaches do not. If replay is a hard requirement, Kafka is structurally necessary.

4. What is the transformation complexity? If the work is filtering and routing, Kafka Streams is fine. If it is multi-table joins, aggregations across days of history, or ML inference, that belongs in Databricks. Both can coexist — Kafka handles delivery, Databricks handles computation.

The Architecture Most Production Platforms Converge On

After building and iterating on data platforms at scale, the pattern that recurs is a three-path architecture:

• Hot path (sub-second to 5 seconds): Kafka producers → Kafka topics → Stream processing → Operational databases or caches → Real-time user-facing features
• Warm path (5 seconds to 5 minutes): Kafka topics → Databricks Structured Streaming → Delta Lake Silver → Near-real-time analytics
• Cold path (hourly to daily): Cloud storage or scheduled JDBC jobs → Databricks batch → Delta Lake Gold → Data warehouse → BI tools

Not every platform needs all three paths. Start with the cold path — it is the cheapest to operate and covers the majority of business intelligence workloads. Add the warm path when a critical use case requires minute-level freshness. Add the hot path when a product feature genuinely needs sub-five-second event delivery. This sequencing matches the cost and complexity curve to the actual business value delivered at each stage.

Conclusion

The Kafka vs Databricks question is rarely about which technology is better. It is about which problem you are solving. Kafka moves events reliably at low latency to multiple consumers with durable replay. Databricks transforms data at scale with rich compute capabilities. These are not competing answers to the same question — they are answers to different questions that arise at different points in the same data pipeline.

The practical rule: if your latency requirement is under five seconds or you have more than two independent consumers, Kafka is probably justified. If you are doing batch processing with a single consumer and no replay requirement, Databricks alone with JDBC ingestion is likely cheaper and simpler to operate. The cost of getting this wrong is real — both over-engineering (paying for streaming you don't use) and under-engineering (batch pipelines failing to meet latency SLAs) show up in engineering hours and infrastructure invoices. Pick the architecture that matches what the business actually needs today, and build the migration path to the next tier when the requirements evolve.