Event Hubs with the Kafka Protocol: Managed Infrastructure, Zero Code Changes

Shannon Lowder

Shannon Lowder

— 2 min read

Azure Event Hubs Kafka protocol support went generally available this year, and it is a bigger deal than most of the announcement posts made it sound. The practical implication: any Kafka producer or consumer can connect to Event Hubs by changing a connection string. No SDK swap. No code changes. No rewrite.

For teams that have been writing to self-hosted Kafka but want to move to a managed service without a migration project, this is the path. For teams that want to evaluate Event Hubs without committing to a full integration, this lowers the bar to essentially zero.

What Changed

Event Hubs now exposes a Kafka-compatible endpoint on port 9093 using SASL/SSL authentication. The Kafka client library (kafka-python, the Java client, librdkafka, whatever you are using) connects to this endpoint exactly as it would connect to a Kafka broker. Topics map to Event Hubs. Consumer groups map to Event Hubs consumer groups. Offsets work the same way.

The connection string format is the main change. Instead of broker-hostname:9092, you use Event Hubs' FQDN on port 9093 with SASL credentials derived from the Event Hubs connection string.

# kafka-python producer connecting to Event Hubs
from kafka import KafkaProducer
import json

CONNECTION_STRING = "Endpoint=sb://mycompany-events.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-key-here"

# Parse the connection string into SASL components
# Event Hubs uses the connection string as the SASL password
# Username is always '$ConnectionString'

producer = KafkaProducer(
    bootstrap_servers='mycompany-events.servicebus.windows.net:9093',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
    sasl_plain_username='$ConnectionString',
    sasl_plain_password=CONNECTION_STRING,
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

producer.send('order-placed', value={'order_id': 1001, 'total': 129.99})
producer.flush()
# kafka-python consumer connecting to Event Hubs
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'order-placed',
    bootstrap_servers='mycompany-events.servicebus.windows.net:9093',
    security_protocol='SASL_SSL',
    sasl_mechanism='PLAIN',
    sasl_plain_username='$ConnectionString',
    sasl_plain_password=CONNECTION_STRING,
    group_id='order-raw-landing',
    auto_offset_reset='earliest',
    enable_auto_commit=False
)

for message in consumer:
    raw_store.write(message.partition, message.offset, message.value)
    consumer.commit()

The code is identical to the self-hosted Kafka version from earlier in this series. The only change is the bootstrap_servers address and the SASL configuration. Everything else — consumer groups, offset management, partition behavior — works the same way.

What the Kafka Protocol Surface Does Not Cover

Not everything in the Kafka API is available via the Event Hubs Kafka endpoint. The AdminClient API (creating topics programmatically, describing consumer groups via API, altering topic configurations) is not fully supported — you manage Event Hubs and consumer groups through the Azure portal or ARM/Terraform, not through Kafka admin tools. Schema Registry is not included — if you need schema enforcement, you still need to run your own Schema Registry or use a third-party service. Kafka Streams and Kafka Connect, which use the AdminClient API under the hood, require workarounds.

For simple producer/consumer use cases — which covers the majority of data pipeline ingestion patterns — the Kafka protocol support is complete and production-worthy. If you are building a full Kafka Streams topology or running Kafka Connect with dozens of connectors, the gaps matter more.

Migration Path

If you have existing Kafka producers and consumers and want to move to Event Hubs without downtime:

  1. Provision the Event Hubs namespace and create Event Hubs matching your Kafka topic names
  2. Update your connection configuration (bootstrap_servers, SASL settings) via environment variable or config file — no code changes
  3. Restart consumers one at a time, allowing them to commit new offsets to Event Hubs
  4. Drain the old Kafka topics (confirm consumer group lag reaches zero) before decommissioning

The migration is a configuration change, not a code change. For organizations with policy reasons to prefer managed services over self-hosted infrastructure, this is a significant unlock. I am here to help if you want to walk through the migration for a specific pipeline.

Read more