Azure Data Lake Storage: The First Object Store That Did Not Feel Like a Compromise

Shannon Lowder

Shannon Lowder

— 3 min read

Before Azure Data Lake Storage, storing large amounts of unstructured data in Azure meant Azure Blob Storage. Blob Storage worked, but it was an object store with a flat namespace — "folders" were really just prefixes in the object name. If you wanted to use it as the foundation of a data lake with a real directory hierarchy, enforce access controls at the folder level, or work with Hadoop-compatible file paths, you were working against the model rather than with it.

ADLS Gen 1 (launched in 2015, with preview in late 2014) was different. It was designed from the ground up as a data lake — a hierarchical file system with POSIX-style ACLs, optimized for analytical workloads, with native Hadoop compatibility. For the first time, the object store matched how data practitioners actually wanted to organize data.

What ADLS Got Right That Blob Storage Didn't

Hierarchical namespace with real ACLs. In Blob Storage, there was no folder — just a key with a "/" in it. ADLS had an actual directory tree, and you could set access control at any level in the tree. A data engineering team could have read access to the raw/ zone, read-write access to their processing/ zone, and no access to the finance/ zone. This is table-stakes for a multi-team data platform, and Blob Storage couldn't do it in 2014.

Optimized for large analytical reads. ADLS internally optimized for the access patterns of analytical workloads: large sequential reads of big files, rather than Blob Storage's optimization for small object retrieval. Read throughput for files measured in gigabytes was measurably better than Blob Storage for the same data.

Hadoop compatibility. ADLS supported the Azure Data Lake Store adapter for Hadoop, meaning the same MapReduce and Hive jobs that read from HDFS could read from ADLS with a path prefix change. For organizations running HDInsight (Microsoft's managed Hadoop) or migrating Hadoop workloads to Azure, this compatibility removed a significant integration cost.

The Zone Architecture

The data lake zone pattern I used on every ADLS deployment:

adls://yourlake.azuredatalakestore.net/
  raw/                    -- landing zone: data as received from sources
    noaa/
      storm-events/
        2014/
    salesforce/
      accounts/
  staging/                -- in-flight processing: cleaned and conformed
    storm-events/
    accounts/
  curated/                -- final: query-ready, governed, versioned
    analytics/
    reporting/
  archive/                -- cold storage: processed data retained for compliance

Each zone had separate ACL settings. Service accounts for ingestion had write access to raw/ only. ETL processing accounts had read access to raw/ and write access to staging/. Analytical tools had read access to curated/ and no access to raw/ or staging/. The zone boundaries enforced the data governance model at the storage layer — not as a policy document but as an access control configuration.

The ACL Model

ADLS ACLs followed the POSIX model: user, group, and other permissions, with named user and named group entries for more specific access. This was more expressive than the Shared Key / SAS token model Blob Storage offered in 2014.

# Set ACLs via Azure CLI (simplified)
# Grant ETL service account read access to raw/noaa/
azure datalake store filesystem permission set \
  --accountName yourlake \
  --path /raw/noaa \
  --aclSpec "user:etl-service-account:r-x"

# Grant analytics team read-only to curated/
azure datalake store filesystem permission set \
  --accountName yourlake \
  --path /curated \
  --aclSpec "group:analytics-team:r-x"

Where ADLS Made the Most Difference

The workloads where ADLS clearly outperformed Blob Storage were large-scale analytical jobs: U-SQL jobs in Azure Data Lake Analytics reading and writing files measured in tens of gigabytes, HDInsight jobs processing years of historical event data. The throughput advantage was real and measurable for these workloads.

For smaller workloads — files under a few hundred megabytes, infrequent access — the difference between ADLS and Blob Storage was less significant. The governance and ACL story made ADLS the right answer regardless of file size for multi-team environments, but the performance argument was strongest for the large analytical workloads it was designed for.

ADLS Gen 1 was eventually superseded by ADLS Gen 2 (Blob Storage with hierarchical namespace enabled), but the design choices it pioneered — hierarchical namespace, POSIX ACLs, analytical optimization — became the template for what a cloud data lake storage layer should look like. As always, I'm here to help.

Read more