The Raw Zone Is Non-Negotiable: Storing Messages Before Deserializing

Shannon Lowder

Shannon Lowder

— 3 min read

I want to tell you about a pipeline I inherited. It consumed messages from a Kafka topic and wrote structured records directly into an analytics database — no intermediate storage, no raw landing. The producer was an application team in a different department that had, historically, not told anyone when they changed their message schema.

They changed their schema on a Tuesday. Our pipeline kept running. The deserialization did not crash — it just silently dropped fields it did not recognize and defaulted missing fields to null. For three days, we wrote subtly wrong data into our fact tables. The anomaly showed up in a report on Friday. The root cause took four hours to find. The data repair took most of the following week.

This is the canonical argument for the raw zone, and I have watched it play out more than once.

What the Raw Zone Is

The raw zone is the first landing spot for data entering your pipeline — specifically, the place where you write the data exactly as it arrived, before any interpretation. For a streaming pipeline, that means writing the serialized message payload to durable storage before your deserialization code ever sees it.

In a batch ETL context, the raw zone is the file drop folder or the staging table. In a streaming context, it is a topic, an object store, or a Delta table that holds messages in their original format with enough metadata to know where they came from.

The raw zone is not a cache. It is not a temp table that gets truncated after the load. It is the authoritative original record. You can derive everything downstream from it. You cannot derive it from anything downstream.

What You Lose Without It

Without a raw zone, your pipeline has exactly one opportunity to correctly interpret each message: when it first reads it. If that interpretation is wrong — because the schema changed, because your deserialization code had a bug, because the producer sent an unexpected variant — the original message is gone. The Kafka topic will retain it for a while (7 days by default), but if you did not notice the problem within the retention window, you have nothing to replay from.

With a raw zone, you can go back. Fix the deserializer. Reprocess. No data lost, no emergency escalation to the producer team asking them to resend.

What to Store

At minimum, store these fields for every message:

  • Raw payload — the actual bytes, decoded to a string if you need JSON storage, or kept as a binary blob
  • Source topic — where the message came from
  • Partition and offset — so you can correlate with the Kafka log for debugging
  • Arrived-at timestamp — when your consumer received it, not the event timestamp in the payload (the payload might not have one, or it might be wrong)
  • Producer or source identifier — if available from message headers
import json
import time
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'sensor_readings',
    bootstrap_servers=['kafka-broker-01:9092'],
    group_id='raw_landing_job',
    enable_auto_commit=False
)

def write_raw(message):
    record = {
        'source_topic':   message.topic,
        'partition':      message.partition,
        'offset':         message.offset,
        'arrived_at_ms':  int(time.time() * 1000),
        'raw_payload':    message.value.decode('utf-8', errors='replace')
        # store as string; deserialization is a separate job's problem
    }
    raw_store.append(record)  # write to your raw zone storage

for message in consumer:
    write_raw(message)
    consumer.commit()

Notice there is no JSON parsing of raw_payload here. The payload is stored as a string. The raw landing job has one job: write it down. Anything else — parsing, validating, transforming — belongs in the next stage.

The Three-Stage Architecture

Once you commit to a raw zone, the rest of the architecture follows naturally:

  1. Stage 1: Ingest — Kafka consumer writes raw messages to the raw zone. This job can be very fast because it does almost nothing. High throughput, low CPU.
  2. Stage 2: Deserialize — A separate job reads from the raw zone and parses messages into structured records. Schema validation happens here. Bad messages go to a dead letter table with enough context to debug them.
  3. Stage 3: Transform / Merge — A separate job reads the structured records and applies business logic, deduplication, and writes to your serving layer.

Three jobs instead of one. At first that sounds like more work. In practice it is less fragile — each stage can fail, be debugged, and be restarted independently without affecting the others. The ingest job does not need to stop when you deploy a schema fix to the deserializer.

The Objection

"This doubles my storage costs." Maybe. Raw message payloads are often small. JSON sensor readings are a few hundred bytes. Even at millions of messages per day, this is probably gigabytes, not terabytes. Compare that against the cost of data repair from a missed schema change, and the math is straightforward.

The raw zone is infrastructure overhead that pays for itself the first time you need it. I have never talked to someone who had a raw zone and regretted it. I have talked to several people who did not have one and very much regretted it. As always, I am here to help.

Read more