Azure Event Hub

1. What is Azure Event Hubs?

• It supports a publish-subscribe model with multiple consumers reading from the same event stream independently.

• Can integrate with various Azure services for analytics and processing such as Azure Stream Analytics, Azure Functions, Azure Data Lake, and more.

• Supports open protocols including AMQP and Kafka, enabling broad client compatibility.

2. Core Concepts and Architecture

3. Data Flow and Processing

• Producers publish events using SDKs or Kafka clients in batched fashion.

• Events are distributed to partitions based on a partition key (hash-based routing), round robin, or explicit partition assignment.

• Events are appended to partition logs in order, with immutable, durable storage.

• Consumers read events independently within consumer groups, using offsets to track progress.

• Checkpointing ensures consumers can resume processing without loss or duplication.

4. Supported Protocols & SDKs

• Protocols: AMQP 1.0, HTTPS, and Kafka protocol.

• SDKs: Official libraries for .NET, Java, Python, JavaScript, Go, and more.

• Compatibility with Kafka clients allows Event Hubs to be used as a Kafka endpoint.

5. Pricing Models

• Throughput Units (TU): Each TU allows 1 MB/sec ingress or 1000 events/sec, and 2 MB/sec egress.

• TUs determine ingress/egress capacity; can be scaled manually or automatically using Auto-Inflate in Standard/Premium tiers.

6. Scalability

• Achieved by partitioning event hubs to parallelize event ingestion and consumption.

• Scaling is primarily by adding throughput units or increasing partitions.

• Auto-Inflate automatically increases throughput units within limits to meet load.

• Partition count is fixed at creation: max 32 for standard tier, up to 1024 for Premium.

• Consumers scale by creating multiple consumer instances reading partitions independently under consumer groups.

7. Security and Identity Management (IAM)

8. Deployment and Configuration

Resource Creation

• Use Azure Portal, Azure CLI, ARM templates/Bicep, or PowerShell to create and configure namespaces and event hubs.

Sample: Create Event Hub Namespace and Event Hub Using Azure CLI

# Create resource group
az group create --name MyResourceGroup --location eastus

# Create Event Hubs namespace
az eventhubs namespace create --resource-group MyResourceGroup --name MyNamespace --location eastus --sku Standard

# Create Event Hub with 4 partitions and 7 days retention
az eventhubs eventhub create --resource-group MyResourceGroup --namespace-name MyNamespace --name MyEventHub --partition-count 4 --message-retention 7

Configuration Options

• Partition count (defined at creation)

• Message retention period

• Capture configuration for automatic storage

• Consumer Groups creation for parallel consumption

• Access policies (Manage, Send, Listen permissions)

• Max throughput units / auto-inflate settings

9. Development: Producer and Consumer Examples

Sending Events to Event Hub (.NET)

using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;

string connectionString = "<EventHubNamespaceConnectionString>";
string eventHubName = "<EventHubName>";

// Create producer client
await using var producerClient = new EventHubProducerClient(connectionString, eventHubName);

// Create a batch
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();

// Add events
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));

// Send batch
await producerClient.SendAsync(eventBatch);
Console.WriteLine("Events published.");

Receiving Events (.NET) Using Event Processor Client for checkpointing

using Azure.Messaging.EventHubs.Consumer;
using Azure.Messaging.EventHubs.Processor;
using Azure.Storage.Blobs;

string connectionString = "<EventHubNamespaceConnectionString>";
string eventHubName = "<EventHubName>";
string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;

string blobStorageConnectionString = "<BlobStorageConnectionString>";
string blobContainerName = "<BlobContainerName>";

// Create blob container client for checkpointing
BlobContainerClient storageClient = new BlobContainerClient(blobStorageConnectionString, blobContainerName);

// Create processor client to read events
EventProcessorClient processor = new EventProcessorClient(storageClient, consumerGroup, connectionString, eventHubName);

processor.ProcessEventAsync += async (PartitionContext eventArgs) =>
{
    Console.WriteLine($"Received event: {Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray())}");
    // Update checkpoint
    await eventArgs.UpdateCheckpointAsync(eventArgs.CancellationToken);
};

processor.ProcessErrorAsync += (PartitionContext eventArgs) =>
{
    Console.WriteLine($"Error on partition {eventArgs.PartitionId}: {eventArgs.Exception.Message}");
    return Task.CompletedTask;
};

await processor.StartProcessingAsync();

// Wait some time or use your app logic to stop processing later
Console.ReadLine();

await processor.StopProcessingAsync();

10. Advanced Features

• Event Capture: Automatically saves streaming data into Azure Blob Storage or Data Lake for batch analytics without writing custom code.

• Geo-disaster recovery: Namespace-level geo-pairing and failover across Azure regions.

• Kafka endpoint support: Use Event Hubs as a drop-in replacement for Kafka brokers.

• Consumer groups: Enable multiple independent consumers.

• Auto-Inflate: Scale throughput units automatically based on load.

• Partition Keys: Ensure related data is sent to the same partition, preserving order.

• Protocol support: AMQP, HTTPS, Kafka.

• Batching and Compression: For efficient network usage and performance.

11. Monitoring and Diagnostics

• Metrics in Azure Monitor: Incoming requests, outgoing events, throttled requests, partition metrics.

• Diagnostic logs for runtime events.

• Alerts and autoscale rules based on usage thresholds.

• Integration with Application Insights for deeper telemetry.

12. Summary Table for Azure Event Hubs

FAQ

• Namespace: Container for Event Hubs and their policies.

• Event Hub: The event ingestion endpoint partitioned for scalability.

• Partition: Unit of parallelism; holds ordered event logs.

• Producer: Sends events to Event Hub.

• Consumer Group: Separate views/readers of event streams allowing parallel processing.

• Consumer: Reads events from partitions using offsets.

• Checkpointing: Mechanism for storing consumer progress to ensure fault tolerance and replay.

Intermediate Interview Questions and Answers

Advanced and Tricky Interview Questions and Answers

Scenario-Based Interview Questions and Answers