Overview

Summary

Integration Hub is a subscription-based event distribution system composed of four cooperating microservices. It ingests platform events from upstream sources via the event bus, resolves active subscriptions to determine target endpoints, wraps events into OutboxEvent envelopes, routes them through domain-specific outbox topics, and delivers them to subscriber HTTP endpoints. Failed deliveries are forwarded to dead letter topics (DLTs) for automatic retry and operator-driven replay. All four services share a common technology stack and follow the Hexagonal Architecture pattern.

Why It Exists

Platform events (user lifecycle changes, access events, policy modifications, organization updates) need to reach external downstream services. Without Integration Hub, each downstream service would need to:

1. Directly consume event bus topics — requiring event bus client expertise, Schema Registry integration, and Protobuf deserialization in every consumer
2. Implement its own retry logic — handling transient failures, backoff strategies, and dead letter processing independently
3. Manage its own subscription state — tracking which events to consume and which to ignore

Integration Hub solves these problems by providing a managed webhook delivery layer: downstream services register subscriptions with a delivery endpoint and authentication credentials, and Integration Hub handles ingestion, routing, delivery, retry, and dead letter management.

Where It Fits in Keymate

Integration Hub sits between the platform's internal event sources and external downstream subscribers:

Internal Event Sources ──Event Bus──▶ Integration Hub ──HTTP──▶ Downstream Subscribers
        │                              │
        ▼                              ▼
   Event Hub                   Audit & Observability

Event Hub produces raw platform events to event bus topics. Integration Hub consumes these events, resolves subscriptions, and delivers them to subscriber endpoints via HTTP. Every step of the process is audited through the Audit & Observability component.

Boundaries

In scope:

• Event ingestion from the event bus (and gRPC/REST ingest endpoints)
• Subscription and event type management (CRUD)
• Subscription resolution and topic routing
• HTTP delivery to subscriber endpoints with authentication
• Automatic retry and dead letter processing
• Operator API for DLT inspection and manual replay

Out of scope:

• Raw event production — handled by upstream services and Event Hub
• Event bus consumer logic on the subscriber side — subscribers receive events via HTTP callbacks
• Event content transformation — events are delivered as-is in their original format

How It Works

The Four Services

Integration Hub consists of four independent microservices that form an event processing pipeline:

1. Subscription Management Service

The subscription registry. Stores which downstream services are subscribed to which event types and with what delivery configuration. Exposes both gRPC and REST APIs for CRUD operations on subscriptions and event type definitions.

• gRPC server: Exposes subscription and event CRUD methods
• REST API: Provides filterable list, create, update, and delete operations for subscriptions and event types
• Database: Relational database storage for event type definitions and subscription registrations
• Consulted by: Event Processor (via gRPC) during routing

2. Event Processor Service

The ingestion and routing engine. Consumes raw platform events from the event bus, resolves active subscriptions from the Subscription Management service, wraps each event into an OutboxEvent envelope (containing the subscriber's delivery endpoint and authentication), and routes the envelope to the appropriate domain-specific outbox topic.

• Event bus consumer: Consumes EventHubEvent messages from the upstream event topic
• gRPC client: Calls the Subscription Management service to resolve active subscriptions
• Event bus producers: Routes OutboxEvent messages to one of four domain outbox topics
• Topic routing: Events are categorized by operation/resource type and mapped to user lifecycle, access, policy, or organization topics

3. Outbox Processor Service

The delivery engine. Consumes OutboxEvent messages from all four domain outbox topics, extracts the delivery endpoint and authentication from the envelope, and makes HTTP POST calls to subscriber endpoints. Successful deliveries are acknowledged; failures trigger NACK, which routes the message to the corresponding DLT topic.

• Event bus consumers: Merges four domain outbox topic streams
• HTTP client: Asynchronous HTTP client with configurable timeout and authentication
• DLT strategy: Failed messages are automatically routed to corresponding DLT topics
• Retry: Configurable retry count (default: 3) before DLT routing

4. DLT Processor Service

The dead letter processor. Consumes messages from all four DLT topics, persists them to the relational database for durability, and automatically retries delivery on a configurable schedule. Exposes a REST API for operators to inspect failed events and manually trigger replays.

• Event bus consumer: Consumes from all DLT topics
• Database: Relational database storage for failed events and retry history
• Scheduler: Retries failed events at a configurable interval (default: 3 seconds), up to a configurable number of attempts (default: 3)
• REST API: Provides list, detail, and manual replay operations for failed events
• Cleanup: Configurable deletion of processed events (default interval: 60 minutes, disabled by default)

End-to-End Event Flow

1. Ingestion — A platform event (e.g., user created, role assigned, client updated) arrives at Event Processor via the event bus, gRPC, or REST
2. Subscription resolution — Event Processor extracts the event's sourceService, eventType, resourceType, and operationType, then calls Subscription Management via gRPC to find all active subscriptions matching this event key
3. Envelope creation — For each matching subscription, Event Processor wraps the event into an OutboxEvent protobuf envelope containing the subscriber's delivery endpoint, authentication credentials, and protocol
4. Topic routing — The OutboxEvent is routed to the appropriate domain outbox topic based on the event's category (user lifecycle, access, policy, or organization)
5. HTTP delivery — Outbox Processor consumes the OutboxEvent, authenticates with the subscriber endpoint, and delivers the event via HTTP POST
6. Success → Broker ACK, audit log recorded
7. Failure → Retry up to 3 times → Broker NACK → message routed to DLT topic
8. DLT processing — DLT Processor persists the failed event, retries delivery on schedule, and exposes it for operator inspection

The OutboxEvent Envelope

The OutboxEvent protobuf message is the central contract flowing between the three pipeline services (Event Processor → Outbox Processor → DLT Processor):

OutboxEvent
├── event        : EventHubEvent    (the original platform event)
├── protocol     : string           ("http", "grpc", "event-bus")
├── endpoint     : string           (subscriber delivery URL)
└── auth         : SubscriberAuth   (authentication credentials)

Supported Authentication Types

Subscribers can configure one of the following authentication models for event delivery:

Auth Type	Description
BASIC	HTTP Basic Authentication (username, password)
OAUTH2	OAuth 2.0 client credentials flow (client_id, client_secret, token_url, scope)
JWT	Pre-shared JWT token in the Authorization header
APIKEY	API key in a configurable header (e.g., X-API-KEY)

Diagram

Example Scenario

Scenario

A new user is created in the identity provider. Two downstream services have active subscriptions for user creation events and need to be notified.

Input

• Actor: Identity provider (upstream event source)
• Resource: User creation event (EventHubEvent)
• Action: Event flows through the Integration Hub pipeline
• Context:
  • Event type: user event
  • Operation type: CREATE
  • Resource type: USER
  • Two active subscriptions exist in Subscription Management

Expected Outcome

1. Event Processor consumes the EventHubEvent from the event bus
2. Event Processor calls GetSubscriptionsByEvent with the event key → returns 2 subscriptions
3. Two OutboxEvent envelopes are created (one per subscription, each with its own endpoint and auth)
4. Both are routed to the user lifecycle outbox topic
5. Outbox Processor consumes both, delivers via HTTP POST to each subscriber's endpoint
6. Both deliveries succeed → broker ACK, audit logs recorded
7. Both subscribers receive the user creation event at their registered endpoints

Common Misunderstandings

• "Integration Hub transforms event payloads" — Integration Hub delivers events as-is. The OutboxEvent envelope wraps the original EventHubEvent with delivery metadata but does not modify the event content.

• "Subscribers must consume the event bus directly" — Subscribers receive events via HTTP callbacks. They do not need event bus clients, Schema Registry access, or Protobuf deserialization capabilities — Integration Hub handles all of that.

• "Failed events are lost" — Failed deliveries go through multiple retry layers: Outbox Processor retries up to 3 times, then routes to DLT. DLT Processor retries on schedule and persists events for operator-driven manual replay. Events are only lost if the DLT store itself fails.

warning

Integration Hub provides at-least-once delivery. In edge cases (successful HTTP delivery but broker ACK failure), the same event may be delivered more than once. Subscriber endpoints must implement idempotent processing using the eventId field.

Design Notes / Best Practices

• Register specific subscriptions — Subscribe to specific (sourceService, eventType, resourceType, operationType) combinations rather than broad patterns. This reduces unnecessary delivery traffic and simplifies subscriber logic.

• Implement idempotent endpoints — Use the eventId from the delivered event as a deduplication key. At-least-once delivery means duplicates are possible.

• Monitor DLT event age — Regularly inspect the DLT Processor's REST API. Events that remain in FAILED status after all retry attempts indicate a persistent subscriber issue.

• Use OAuth2 for production subscribers — While all four authentication types are supported, OAuth2 client credentials flow provides the best security posture for service-to-service communication.

tip

All four Integration Hub services send audit logs to the Audit & Observability component via gRPC. The audit trail includes events like EVENT_RECEIVED, EVENT_PROCESSED, OUTBOX_SEND, HTTP_SEND, and EVENT_ERROR — providing full end-to-end traceability.

Related Use Cases

• Notifying external services when users are created, updated, or deleted in the identity provider
• Broadcasting access events (login, logout, token exchange) to monitoring or analytics systems
• Distributing policy change events to enforcement points across the platform
• Delivering organization membership changes to downstream directory services

Next Step

Continue with Consumer Model to learn how downstream services register subscriptions and configure delivery endpoints.

Related Docs

Consumer Model

How to register subscriptions and configure delivery endpoints.

gRPC Ingestion Surface

How events enter Integration Hub via gRPC and REST.

Event-Driven Distribution

Domain-based topic routing and fan-out mechanism.

Connector Runtime

Delivery execution, retry strategy, and DLT processing.

Event Hub

The upstream event routing component that feeds events into Integration Hub.

Audit & Observability

Audit logs from all Integration Hub services.

How many microservices does Integration Hub consist of?

Integration Hub consists of four microservices: Subscription Management (subscription registry), Event Processor (ingestion and routing), Outbox Processor (HTTP delivery), and DLT Processor (failed event handling and replay).

Do subscribers need to consume the event bus directly?

No. Subscribers receive events via HTTP POST callbacks at their registered delivery endpoints. Integration Hub handles all event bus consumption, Protobuf deserialization, and delivery logic. Subscribers only need an HTTP endpoint that can accept the event payload.

What happens when a delivery fails?

The Outbox Processor retries delivery up to 3 times (configurable). If all retries fail, the message is routed to a dead letter topic (DLT). The DLT Processor persists the failed event to the relational database, retries delivery on a configurable schedule, and exposes a REST API for operators to inspect and manually replay failed events.

What authentication types are supported for subscriber endpoints?

Integration Hub supports four authentication types: HTTP Basic Auth, OAuth 2.0 (client credentials flow), pre-shared JWT tokens, and API keys in configurable headers.

Does Integration Hub guarantee exactly-once delivery?

No. Integration Hub provides at-least-once delivery. In rare edge cases, the same event may be delivered more than once. Subscriber endpoints should implement idempotent processing using the event's unique eventId as a deduplication key.