Oozie Ran Green and the Data Was Wrong: Making Hadoop Pipelines Observable

Shannon Lowder

Shannon Lowder

— 3 min read

The Oozie coordinator showed green. Every action node in the workflow completed with status SUCCEEDED. The daily aggregation job had run, written its output to HDFS, and Oozie was happy.

The output had zero rows.

Nobody noticed for four days, because there was no alerting on output volume. The downstream reporting table just showed the same numbers it had shown four days earlier — and because those numbers were plausible, nobody questioned them. The anomaly surfaced when a business analyst pulled a date-range query and noticed a suspicious flat line across a holiday weekend that had actually been a high-traffic period.

The root cause was a partition filter that had silently started producing empty results when the upstream partition naming convention changed. Oozie ran the job. The job ran successfully. Zero rows is a valid output from Oozie's perspective.

The Problem With "The Job Ran"

In software engineering, a successful HTTP response code tells you the request completed. The application layer validates whether the response made sense. You don't accept a 200 with an empty body as proof that your order processed — you check the order confirmation.

Data pipelines don't have that second check. Oozie's job is to execute a workflow and report whether it threw an exception. It has no opinion about whether the output is meaningful. That validation has to come from somewhere else — and right now, in most shops, it comes from nowhere.

Three Checks That Would Have Caught This

You don't need a sophisticated observability platform to fix this. You need three things:

1. Output row counts. After each pipeline stage writes to HDFS or a Hive table, count the output rows. If the count is zero or below a historical baseline, fail the job deliberately.

# Add this as a final Oozie shell action after each write stage
OUTPUT_COUNT=$(hive -e "SELECT COUNT(*) FROM reporting.daily_agg WHERE report_date='${REPORT_DATE}'" 2>/dev/null | tail -1)
if [ "$OUTPUT_COUNT" -eq "0" ]; then
  echo "ERROR: daily_agg wrote zero rows for ${REPORT_DATE}"
  exit 1
fi
echo "daily_agg row count: $OUTPUT_COUNT"

2. Partition existence checks. Before a job reads from a partition, verify the partition exists and is non-empty. A missing partition is not an error in Hive — it's just an empty result set. Make it an explicit error.

PARTITION_CHECK=$(hive -e "SHOW PARTITIONS events.raw_clickstream PARTITION (event_date='${EVENT_DATE}')" 2>/dev/null)
if [ -z "$PARTITION_CHECK" ]; then
  echo "ERROR: upstream partition events.raw_clickstream/event_date=${EVENT_DATE} does not exist"
  exit 1
fi

3. Staleness detection. If a pipeline hasn't produced new output in more than N hours, alert. This catches the silent failure where the coordinator stopped firing due to a misconfigured time zone or a missed dependency — and nobody noticed because the old data was still there.

Make Alerting Explicit

Oozie's <kill> node and email notification action are underused. If your workflow's error path is just "mark as KILLED and move on," you're relying on someone to check the Oozie console. That's not monitoring — that's hoping.

<action name="send-failure-alert">
  <email xmlns="uri:oozie:email-action:0.1">
    <to>data-team@yourcompany.com</to>
    <subject>PIPELINE FAILURE: ${wf:name()} at ${wf:lastErrorNode()}</subject>
    <body>
      Workflow: ${wf:name()}
      Failed node: ${wf:lastErrorNode()}
      Error: ${wf:errorMessage(wf:lastErrorNode())}
      Report date: ${reportDate}
    </body>
  </email>
  <ok to="end"/>
  <error to="end"/>
</action>

Wire every error path to this action. It's five minutes of XML. You'll thank yourself the first Monday it pages you before the business notices.

This Is a Production Service

The mindset shift that drives all of this: a scheduled data pipeline is a production service. It has consumers, it has an implicit SLA, and when it fails silently, those consumers make decisions on stale or wrong data. Software teams treat services this way — they have health checks, alerting, and on-call rotations. Data pipelines should get the same treatment.

If you've set up meaningful Hadoop pipeline monitoring and have patterns worth sharing, I'm all ears. As always, I'm here to help.

Read more