A customer data platform for a regulated servicer

The problem: data silos at mortgage scale

When I joined the data platform team at Rocket India in 2018, customer data was scattered across 50+ source systems: loan origination, servicing, payment processing, CRM, collections, escrow, and more. Every team that needed a complete customer view had to join data from multiple systems, sometimes taking hours, sometimes days. Critical decisions (risk assessment, customer outreach, fraud detection) were being made on stale or incomplete data.

By 2021, we had designed and built a Customer Data Platform (CDP) that unified all of this into a single, real-time accessible layer serving 1M+ API calls at peak hour. This article documents the architectural decisions, the tradeoffs we made, and what I'd do differently.

Architecture overview: the medallion approach

We adopted a medallion architecture (Bronze → Silver → Gold) built on Confluent Kafka, Databricks, and Snowflake:

• Bronze Layer (Raw Ingestion), Every change event from source systems lands here via Kafka Connect CDC connectors (Debezium). No transformation, just raw events with metadata
• Silver Layer (Cleansed & Unified), Databricks Structured Streaming jobs process bronze events, apply schema validation, deduplication, and entity resolution. Output to Delta Lake
• Gold Layer (Business-Ready): Aggregated customer profiles, 360-degree views, feature sets for ML models. Stored in Snowflake for SQL access, replicated back to Kafka for real-time consumers

Source DB → Debezium CDC → Kafka (Bronze topics)
                                  ↓
              Databricks Structured Streaming (Silver)
                                  ↓  
              Delta Lake (Silver tables) → Snowflake (Gold)
                                  ↓
              Kafka (Gold events) → Real-time consumers

Decision 1: CDC over batch polling

The first and most important architectural decision: how do we get data out of 10+ relational source systems?

We evaluated three options:

1. Batch polling (JDBC), Simple, no additional infrastructure, but poll intervals create 15–60 minute data lag and miss deletes entirely
2. Application-level events, Developers emit events on every write. Clean but requires coordination with 20+ application teams and introduces event drift over time
3. Database CDC (Debezium), Reads the database transaction log. Captures all changes (inserts, updates, deletes) with sub-second latency, no application changes required

We chose CDC. The setup cost (deploying and operating Debezium connectors) was worth it, we captured 100% of data changes including deletes, with <2 second lag from source to Bronze.

Lesson: If you have more than 2 source systems and need <5 minute data freshness, CDC is almost always the right choice despite the operational overhead.

Decision 2: entity resolution strategy

The hardest problem in any CDP is entity resolution, matching records across systems that represent the same customer but have different IDs, slightly different names, or inconsistent email addresses.

We built a three-tier matching strategy:

Tier 1: deterministic matching

Exact match on Social Security Number (SSN), loan account number, or email address (normalized to lowercase, trimmed). This handles ~75% of records.

Tier 2: probabilistic matching

For records without exact match keys, we used a scoring model combining:

• Name similarity (Levenshtein distance, normalized)
• Address similarity (street, city, zip)
• Phone number match
• Date of birth match

Records scoring above 0.85 were auto-merged. Records between 0.60–0.85 went to a human review queue.

A worked example

Two records with no shared exact-match key, one from loan origination, one from servicing:

Record A (origination)          Record B (servicing)
name:  "Robert J. Smith"        name:  "Rob Smith"
email: "rsmith@email.com"       email: "r.smith@email.com"
phone: "+1-555-0142"            phone: "+1-555-0142"
addr:  "123 Main St, Austin"    addr:  "123 Main Street, Austin"
dob:   "1985-03-14"             dob:   "1985-03-14"

Neither the name nor the email matches exactly, which is exactly the case Tier 1 is blind to. Normalized field scores: name similarity ≈ 0.78 (a nickname plus a dropped middle initial), address similarity ≈ 0.95 (street vs. street abbreviation), phone ≈ 1.0, date of birth ≈ 1.0. With equal weighting across the four fields, the composite score is ≈ 0.93, comfortably above the 0.85 auto-merge threshold. Email is not in the scoring model at all here: it is the field most likely to change or be re-typed inconsistently, so we treated it as a Tier 1 exact-match signal only, never a Tier 2 similarity input. Phone and date of birth carried the real weight, because they are the fields most stable across a customer's lifecycle in a mortgage servicing relationship.

Tier 3: manual override

Ops team could manually link or unlink records. These overrides were stored as first-class events in Kafka and took highest priority in all resolution decisions.

Decision 3: the 500+ table problem

Unifying 500+ source tables sounds daunting. Here's how we made it tractable:

Domain-based partitioning

We organized tables into 8 domains: Customer Identity, Loan, Payment, Escrow, Collections, Communications, Documents, and Risk. Each domain had an owning team responsible for its Silver-layer transformations.

Canonical schema

We defined a canonical Customer entity schema with ~120 fields. Every source system's data was mapped to this schema by the owning domain team. New fields could be added but existing fields couldn't be removed or renamed without a deprecation cycle.

-- Example: Canonical customer identity fields
customer_id          VARCHAR  -- CDP-generated UUID
ssn_hash             VARCHAR  -- SHA-256 of SSN (never plain SSN)
first_name           VARCHAR
last_name            VARCHAR
email                VARCHAR  -- Normalized
phone_primary        VARCHAR  -- E.164 format
address_line1        VARCHAR
city                 VARCHAR
state                CHAR(2)
zip                  CHAR(5)
created_at           TIMESTAMP
updated_at           TIMESTAMP  -- CDP-level, not source
source_system        VARCHAR  -- Origin system ID
source_system_id     VARCHAR  -- ID in origin system

Schema governance

All schemas were versioned in Confluent Schema Registry with backward compatibility enforced. Producers couldn't publish breaking schema changes. They had to be approved through a schema review process. This saved us from countless silent data breakages.

Decision 4: real-time vs. batch serving

Not all use cases have the same freshness requirements:

The key insight: don't build one serving layer to rule them all. Build the right serving layer for each latency tier and populate them all from the same unified Gold layer.

Operational lessons

1. Schema evolution is your biggest risk

Source systems change schemas without warning. We caught 40+ schema-breaking changes in the first year because Confluent Schema Registry rejected them at the connector level before they could corrupt downstream data. Invest heavily in schema governance early.

2. Monitor consumer lag, not just throughput

Throughput metrics looked healthy during several incidents because messages were still flowing. Consumer lag was the real signal. It told us when a processing bottleneck was forming 20 minutes before it became a customer-facing problem.

3. Data quality gates are non-negotiable

We added quality validation at the Silver layer: null checks, referential integrity, value range validation. Records failing validation went to a dead letter topic rather than silently corrupting the Gold layer. This increased trust across all downstream teams dramatically.

4. Backfill strategy matters

When you launch a CDP, you need to populate it with historical data. We underestimated this. Our backfill took 3 weeks because we tried to run it through the streaming pipeline at full volume. In retrospect: build a dedicated batch backfill path from day one, separate from the streaming path.

5. Ownership requires SLA commitment

We assigned each domain team a data quality SLA on event processing timeliness. This accountability created the right incentives for teams to own their connectors, schemas, and Silver transformations properly.

Results after 4 years

• 500+ tables unified from 50+ source systems
• 3 seconds to 500 milliseconds on medallion reads, an 83% improvement felt by 15+ downstream teams
• Redundant reporting ETL decommissioned, along with the per-product joins against systems of record
• Zero data quality incidents reaching production in the last 18 months
• 1M+ API calls at peak hour served across fraud, customer service, marketing, and compliance

Building a CDP is a multi-year investment. The first year is hard: schema fights, entity resolution edge cases, backfill headaches. But the compounding value as more use cases are unlocked from a single trusted data layer is transformative. Start with CDC, invest in schema governance, and resist the temptation to build one serving layer for all use cases.

Frequently asked questions

How is a customer data platform different from a data warehouse?

A data warehouse is optimized for analytical queries over historical data, typically refreshed hourly or daily. A CDP is an operational layer: it resolves entities across source systems in near real time and serves both low-latency lookups (fraud, customer service) and analytical workloads from the same unified model. In our architecture the Gold layer fed both a Snowflake warehouse and live Kafka consumers, so the CDP was the source that fed the warehouse, not a replacement for it.

Do you need Kafka to build a CDP, or would batch ETL work?

It depends entirely on your freshness requirement. If every consumer of the unified customer view can tolerate hourly or daily staleness, a batch ETL pipeline into a warehouse is simpler to build and operate. We needed sub-second freshness for fraud detection and sub-500ms for customer service lookups, which batch ETL cannot deliver. If you don't have a use case that needs that, don't build the streaming version, it's meaningfully more operational surface area for no benefit.

How is a customer's social security number handled in the canonical schema?

Never store plain social security numbers in the CDP. We stored a one-way hash for matching purposes and kept the plaintext value only in the originating system of record, accessed through a separate, audited service when a downstream consumer had a legitimate need for it. The same principle applied to other regulated fields: the CDP holds what's needed for resolution and serving, not a second copy of every sensitive field a source system has.

What's a realistic timeline for a CDP of this scope?

Budget at least a year before the platform is trustworthy enough for teams to build on. The first few months go into CDC infrastructure and the canonical schema for a handful of domains. Entity resolution accuracy takes longer to mature than the infrastructure does, expect to be tuning match thresholds against real production edge cases for two to three quarters after initial launch. The compounding value shows up in year two, once enough domains and consumers are on the platform that teams stop building their own point-to-point joins.