The Outbox Pattern in Orleans: The Grain Is the Relay

I honestly wasn't sure the outbox pattern belonged in an Orleans codebase. Grains are single-threaded. State writes are atomic. The runtime handles failover. Bringing in a relational-era workaround felt like admitting Orleans hadn't solved the problem, when I was fairly sure it had.

Then I watched a silo die between WriteStateAsync() and PublishAsync(). The order was saved. Nobody downstream heard about it. Flip the two lines and you get the opposite bug: an OrderPlaced event for an order that never reached storage. The customer can't tell the difference. Either way, no confirmation email.

So, after building this twice, here is where I landed. The dual-write problem is real in Orleans. The outbox pattern is the right fix. But Orleans throws away about two-thirds of the pattern's machinery, and if you copy the microservices.io version line for line you will build infrastructure you don't need.

This post covers when you need it, when you don't, and the ~60 lines of reusable code that make a grain's external side effects survive a crash. No outbox table, no relay service, no CDC, no distributed lock.

Do You Even Need This? (I Wasn't Sure Either)

Dual-write is when one operation has to land in two systems that don't share a transaction. In Orleans those two systems are your grain storage provider (Azure Table, Cosmos, Postgres via ADO.NET, Redis) and whatever sits outside the cluster: Kafka, Azure Service Bus, RabbitMQ, SendGrid, a partner webhook, a reporting database.

There are exactly two failure windows. Be precise about them.

1. Crash between the writes. State is saved, the process dies, the publish never happens. The grain reactivates with no memory that it owed anyone a message.
2. Destination down after the state is saved. Storage succeeds, the broker throws. The exception reaches the caller, who now has to guess whether the order exists. Retry and you risk a duplicate order. Give up and you've lost the event without a trace.

Orleans does not save you here. The single-threaded activation model guarantees ordering inside the grain. It says nothing about the durability of side effects that leave the grain. WriteStateAsync() is atomic for the state object and only the state object. The Orleans Transactions docs are clear that transactions coordinate ITransactionalState<T> across grains. A broker is not a participant.

This is the naive version everyone writes first. I've marked where it breaks.

public async Task PlaceOrderAsync(PlaceOrder cmd)
{
    state.State.Order = Order.From(cmd);
    await state.WriteStateAsync();
    // CRASH POINT 1: order persisted, event never sent, grain forgets on reactivation

    await bus.PublishAsync(new OrderPlaced(this.GetPrimaryKeyString(), cmd.Total));
    // CRASH POINT 2: broker throws, caller sees failure for an order that exists
}

Both crash points are silent in production. Nothing reaches a dashboard for crash point 1. You find out three weeks later when finance asks why 14 orders have no invoices.

When NOT to use it

I'll take explicit positions here, because this pattern gets cargo-culted.

Both sides are grains in the same cluster. An outbox is waste. If you need atomicity across grains, use Orleans Transactions with [Transaction(TransactionOption.Create)] and ITransactionalState<T>. Otherwise make the second grain's method idempotent and let the caller retry. Orleans won't retry an application exception for you, but a grain call that fails tells you it failed, synchronously, inside the same runtime. There is no second system to drift out of sync with. An outbox between two grains adds a storage write and solves nothing.

The side effect is best-effort. A SignalR notification, a cache invalidation, a metrics counter. If the message is lost, the user refreshes or the cache expires. Nobody loses money. Don't persist it.

You're already on JournaledGrain. Your event log is the outbox. You don't need a second list of pending messages, you need a cursor recording which log version you've published up to. I cover that cursor in the Orleans 10 JournaledGrain event sourcing post and come back to it in Section 4.

My rule, which I now apply without a second thought: use the outbox when the destination can't join grain storage's write and losing the message costs money. Everything else is over-engineering.

The Orleans Twist: The Grain Is the Relay

The canonical transactional outbox has three moving parts. An outbox table written in the same transaction as the business data. A message relay that reads the table and forwards to the broker. And something to guarantee only one relay runs at a time: leader election, row locks, or Debezium-style CDC so the database log becomes the relay.

In Orleans, every one of those maps to something you already have.

The lifecycle:

1. Mutate state and append the pending message to the same state object.
2. One WriteStateAsync(). This is the only write that must succeed for the operation to count.
3. Dispatch pending messages in order, immediately, in the same turn.
4. On ack, remove the message (or move a cursor) and write state again.
5. On failure, arm a grain timer (in-memory, exponential backoff, KeepAlive = true) and a reminder (durable, one-minute period). Stop draining. Later messages must not overtake the failed one.
6. On reactivation, OnActivateAsync drains whatever is left.

The timer covers the common case: a transient broker hiccup, retried a few seconds later while the grain is still warm. The reminder covers what the timer can't: the grain was deactivated or the silo died with messages still pending. Per the timers and reminders docs, reminders are persisted, tied to the grain rather than the activation, and will re-create the activation when they tick.

Two rules keep reminder costs down. Only register the reminder when the outbox is non-empty, and unregister it once drained. A grain that publishes successfully 99.9% of the time never touches the reminder table.

Look at what step 3 doesn't have: a poller. Traditional outbox setups pay a polling interval on every message, because the relay is a separate process that can't know when a row appears. In Orleans the code that wrote the row is the code that sends it, so normal-path latency is zero. You also don't need a .NET job scheduler like TickerQ next to the cluster to sweep the outbox. Timers and reminders are already distributed and already scoped to the grain that owns the data.

One thing on the horizon. The Orleans 10.4.0 release ships Microsoft.Orleans.DurableJobs as a 10.4.0-alpha.1 preview. Durable Jobs are time-sharded, at-least-once, one-shot scheduled work that survives deactivation. That is exactly the shape of the reminder safety net above, minus the one-minute floor. I expect it to replace the reminder half of this pattern once it's stable. I'm not building on it today, and you shouldn't either for anything that touches money.

Example 1: OrderGrain Publishing OrderPlaced to Service Bus

Start with the message and state types. Anything that goes through WriteStateAsync() needs [GenerateSerializer] and explicit [Id(n)] attributes.

[GenerateSerializer]
public sealed record OutboxMessage(
    [property: Id(0)] string Id,
    [property: Id(1)] string Type,
    [property: Id(2)] string Payload,
    [property: Id(3)] long Sequence,
    [property: Id(4)] int Attempts,
    [property: Id(5)] DateTimeOffset CreatedAt)
{
    public static OutboxMessage Create(string type, string payload) =>
        new("", type, payload, -1, 0, DateTimeOffset.UtcNow);
}

[GenerateSerializer]
public sealed class OrderState
{
    [Id(0)] public Order? Order { get; set; }
    [Id(1)] public List<OutboxMessage> Pending { get; set; } = [];
    [Id(2)] public long NextSequence { get; set; }
}

The publisher abstraction is one method. The Azure Service Bus implementation is six lines, and the only one that matters is MessageId = msg.Id. Service Bus duplicate detection keys on MessageId alone, so turn it on for the queue and redeliveries from a crashed grain get dropped on the broker side. Kafka has nothing like it, which is why consumers dedupe too (more on that below).

public interface IOutboxPublisher
{
    Task PublishAsync(OutboxMessage message, CancellationToken cancellationToken);
}

public sealed class ServiceBusOutboxPublisher(ServiceBusSender sender) : IOutboxPublisher
{
    public Task PublishAsync(OutboxMessage msg, CancellationToken ct) =>
        sender.SendMessageAsync(
            new ServiceBusMessage(msg.Payload) { MessageId = msg.Id, Subject = msg.Type },
            ct);
}

Now the grain. First half: the command and the drain loop.

public sealed class OrderGrain(
    [PersistentState("order", "orders")] IPersistentState<OrderState> state,
    IOutboxPublisher publisher,
    ILogger<OrderGrain> logger) : Grain, IOrderGrain, IRemindable
{
    private const string RetryReminder = "outbox-retry";
    private IGrainTimer? _retryTimer;
    private int _backoffSeconds = 2;
    private bool _reminderArmed;

    public async Task PlaceOrderAsync(PlaceOrder cmd)
    {
        var order = Order.From(cmd);
        var evt = new OrderPlaced(this.GetPrimaryKeyString(), order.Total, order.CustomerEmail);

        state.State.Order = order;
        Enqueue(OutboxMessage.Create("OrderPlaced", JsonSerializer.Serialize(evt)));
        await state.WriteStateAsync(); // the one write that makes the order real
        await TryDrainAsync(CancellationToken.None);
    }

    private void Enqueue(OutboxMessage msg)
    {
        var seq = state.State.NextSequence++;
        state.State.Pending.Add(msg with { Id = $"{this.GetPrimaryKeyString()}:{seq}", Sequence = seq });
    }

    private async Task TryDrainAsync(CancellationToken ct)
    {
        var pending = state.State.Pending;
        while (pending.Count > 0)
        {
            var msg = pending[0];
            try { await publisher.PublishAsync(msg, ct); }
            catch (Exception ex)
            {
                logger.LogWarning(ex, "Outbox publish failed for {MessageId}", msg.Id);
                pending[0] = msg with { Attempts = msg.Attempts + 1 };
                await ArmRetryAsync();
                return; // stop here so later messages cannot overtake this one
            }
            pending.RemoveAt(0);
            await state.WriteStateAsync(); // ack; a crash before this line means redelivery
        }
        await DisarmRetryAsync();
    }

Second half: the retry machinery and lifecycle hooks.

    private async Task ArmRetryAsync()
    {
        _retryTimer?.Dispose();
        _retryTimer = this.RegisterGrainTimer(
            static (g, ct) => g.OnRetryTickAsync(ct),
            this,
            new GrainTimerCreationOptions
            {
                DueTime = TimeSpan.FromSeconds(_backoffSeconds),
                Period = Timeout.InfiniteTimeSpan, // one-shot; re-armed on the next failure
                KeepAlive = true
            });
        _backoffSeconds = Math.Min(_backoffSeconds * 2, 30);

        if (_reminderArmed) return;
        await this.RegisterOrUpdateReminder(RetryReminder, TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1));
        _reminderArmed = true;
    }

    private Task OnRetryTickAsync(CancellationToken ct) => TryDrainAsync(ct);

    private async Task DisarmRetryAsync()
    {
        _retryTimer?.Dispose();
        _retryTimer = null;
        _backoffSeconds = 2;
        if (!_reminderArmed) return;
        var reminder = await this.GetReminder(RetryReminder);
        if (reminder is not null) await this.UnregisterReminder(reminder);
        _reminderArmed = false;
    }

    public Task ReceiveReminder(string reminderName, TickStatus status) =>
        reminderName == RetryReminder ? TryDrainAsync(CancellationToken.None) : Task.CompletedTask;

    public override async Task OnActivateAsync(CancellationToken cancellationToken)
    {
        if (state.State.Pending.Count > 0)
        {
            _reminderArmed = true; // assume a reminder exists; DisarmRetryAsync cleans up either way
            await TryDrainAsync(cancellationToken);
        }
        await base.OnActivateAsync(cancellationToken);
    }
}

A few things I want to be upfront about.

This is two writes per side effect. One to persist the order plus the pending message, one to ack. You can't avoid that without a transaction spanning storage and broker, which is the whole reason you're reading this. The second write is cheap, and you could take it off the caller's critical path by returning before draining. I don't. I'd rather the caller know the event went out.

The attempt counter isn't persisted on failure. It sits in the in-memory list until the next successful ack writes it. If the grain deactivates mid-outage, attempts reset. I accept that rather than pay a storage write on every failed publish while the broker is down.

The grain timer is one-shot. Each tick either drains everything and disarms, or fails and re-arms with a doubled DueTime, capped at 30 seconds. The callback follows the Orleans 8.2+ RegisterGrainTimer API: Func<TState, CancellationToken, Task> plus a GrainTimerCreationOptions struct. The old RegisterTimer is obsolete and interleaves by default. Don't use it.

It is at-least-once. A crash between the broker's ack and the second WriteStateAsync() means the message goes out again on reactivation. That's the contract. Consumers dedupe on Id, which is {grainId}:{sequence} and therefore stable across redeliveries:

// Consumer side: idempotency key is the outbox message Id, carried as MessageId
if (await processed.ContainsAsync(message.MessageId)) return;
await Handle(message);
await processed.AddAsync(message.MessageId);

Example 2: Make It Reusable, Then Use It in 15 Lines

Everything above except PlaceOrderAsync is boilerplate. Lift it into a base class and your domain grains shrink to the part you care about.

The state wrapper holds your domain data plus the outbox bookkeeping. I've switched from "delete on ack" to a published-up-to cursor. I'll explain why in a moment.

[GenerateSerializer]
public sealed class OutboxState<TState> where TState : class, new()
{
    [Id(0)] public TState Data { get; set; } = new();
    [Id(1)] public List<OutboxMessage> Pending { get; set; } = [];
    [Id(2)] public long NextSequence { get; set; }
    [Id(3)] public long PublishedUpTo { get; set; } = -1;
}

public abstract class OutboxGrain<TState>(
    IPersistentState<OutboxState<TState>> state,
    IOutboxPublisher publisher) : Grain, IRemindable
    where TState : class, new()
{
    private const string RetryReminder = "outbox-retry";
    private const int CompactionThreshold = 50;
    private IGrainTimer? _retryTimer;
    private int _backoffSeconds = 2;
    private bool _reminderArmed;

    protected TState Data => state.State.Data;

    protected async Task CommitAsync(Action<TState> mutate, params OutboxMessage[] messages)
    {
        mutate(state.State.Data);
        foreach (var m in messages)
        {
            var seq = state.State.NextSequence++;
            state.State.Pending.Add(m with { Id = $"{this.GetGrainId()}:{seq}", Sequence = seq });
        }
        await state.WriteStateAsync();
        await TryDrainAsync(CancellationToken.None);
    }

CommitAsync is the whole public surface. Mutate your state, hand over zero or more messages, and the base class guarantees they're persisted with the mutation and eventually delivered.

The drain loop with the cursor:

    private async Task TryDrainAsync(CancellationToken ct)
    {
        var s = state.State;
        foreach (var msg in s.Pending.Where(m => m.Sequence > s.PublishedUpTo))
        {
            try { await publisher.PublishAsync(msg, ct); }
            catch (Exception)
            {
                await state.WriteStateAsync(); // persist whatever progress the cursor made
                await ArmRetryAsync();
                return;
            }
            s.PublishedUpTo = msg.Sequence; // ack is a cursor move, not a list mutation
        }

        if (s.Pending.Count > CompactionThreshold)
            s.Pending.RemoveAll(m => m.Sequence <= s.PublishedUpTo); // lazy compaction

        await state.WriteStateAsync();
        await DisarmRetryAsync();
    }

    // ArmRetryAsync, DisarmRetryAsync, ReceiveReminder and OnActivateAsync
    // are lifted verbatim from OrderGrain above.
}

The cursor instead of deleting rows: deleting after each ack is the lazy default, and it costs one storage write per message. With a cursor you drain the whole batch and write once. Published messages stay in the list until there are more than 50, then one compaction pass clears them. The trade is a wider redelivery window after a crash (the whole batch instead of one message). That's fine, because you're at-least-once anyway and the consumer dedupes. The bonus is a free audit trail of what this grain recently emitted, which has saved me more than one late debugging session.

A domain grain on top of it. Changing an email address has to persist the pending address and send a confirmation link via SendGrid or a webhook. If that email never goes out, the user can't confirm and you get a support ticket.

public sealed class UserGrain(
    [PersistentState("user", "users")] IPersistentState<OutboxState<UserProfile>> state,
    IOutboxPublisher publisher)
    : OutboxGrain<UserProfile>(state, publisher), IUserGrain
{
    public Task ChangeEmailAsync(string newEmail)
    {
        var token = Guid.NewGuid().ToString("N");
        var payload = JsonSerializer.Serialize(new { Email = newEmail, Token = token });

        return CommitAsync(
            u => { u.PendingEmail = newEmail; u.ConfirmationToken = token; },
            OutboxMessage.Create("EmailConfirmationRequested", payload));
    }
}

Fifteen lines, and whoever writes the grain never thinks about timers, reminders or crash windows again. That's the 75% in practice. The hard part is written once, and the domain code reads like it has no failure modes.

If inheritance offends you, the same thing works as a component. Inject an Outbox<TState> helper that wraps the IPersistentState and expose CommitAsync on it. The grain still has to implement IRemindable and forward ReceiveReminder and OnActivateAsync, so you save a base class and pay two forwarding methods. I use the base class because forwarding methods are exactly the kind of boilerplate people forget.

If you're on JournaledGrain, you don't need a list

The event log already stores everything in order. A Pending list would just duplicate it. Keep a cursor instead:

// Inside a JournaledGrain<TState, TEvent>; _publishedVersion lives in a small IPersistentState<int>
private async Task PublishNewEventsAsync(CancellationToken ct)
{
    await ConfirmEvents();
    var unpublished = await RetrieveConfirmedEvents(_publishedVersion.State, Version);
    foreach (var e in unpublished) await publisher.PublishAsync(ToOutboxMessage(e), ct);
    _publishedVersion.State = Version;
    await _publishedVersion.WriteStateAsync();
}

Same retry and reminder wiring as above, same at-least-once contract. The Orleans 10 JournaledGrain event sourcing post has the full grain this snippet drops into.

Orleans Streams are not an outbox

I get asked this a lot. Orleans Streams are for fan-out inside the cluster: one grain emits, many grains or observers subscribe, the runtime handles delivery. They're the right tool for a projection grain reacting to an order. They're the wrong tool for reaching Kafka, because the stream provider's persistence (if it has any) still isn't transactional with your grain storage. You can have both. The grain commits the outbox message, and a downstream projection grain subscribes to a stream for the in-cluster view. The outbox is for the one hop that leaves the cluster.

Honest Trade-Offs

Two writes per side effect, minimum. Covered above. If your storage provider bills per operation, this doubles the cost of any command that emits an event. For most systems that's noise. For a chatty telemetry grain it isn't, and that grain shouldn't be using an outbox anyway (see "best-effort" earlier).

The outbox grows while the broker is down. Each failed command appends a message and nothing drains. Cap it. I reject new commands with a clear error once Pending.Count passes 1,000, because an Azure Table entity tops out at 1 MB and a Cosmos item at 2 MB. Blow through either and a broker outage becomes a storage outage. Fail loudly at the edge, not silently in the middle.

ETag conflicts with reentrant grains. Mark the grain [Reentrant], or set Interleave = true on the retry timer, and the drain loop can run at the same time as a command that also writes state. One of the two writes fails with an InconsistentStateException. Keep the grain non-reentrant and leave Interleave at its default false. The single-threaded model is doing real work for you here.

Reminders have a one-minute floor and a storage cost. Each registration and unregistration is a write to the reminder table. Arming only when needed keeps this to a rounding error, but ten thousand grains hitting a dead broker at once means ten thousand reminder writes. Acceptable in an outage. Worth knowing about.

Ordering is per grain, not global. Messages from OrderGrain("42") arrive in sequence. Messages from grains 42 and 43 can interleave however the broker likes. If you need cross-aggregate ordering, that's a different problem and the outbox pattern was never going to solve it.

It's still at-least-once. Every decision above assumes consumers dedupe on message Id. If a downstream team tells you they can't, you don't have a reliable integration no matter what you do on the Orleans side.

The cost of doing none of this: silent data loss over weeks, found by whoever reconciles the numbers. I'll take two storage writes.

Where I Landed

The outbox pattern belongs in Orleans in exactly one place: where a grain must persist state and tell a system outside the cluster that can't join the write. Everywhere else it's dead weight.

My checklist:

• Is the destination outside grain storage (broker, email, webhook, another database)? If no, stop. Use Transactions or idempotent retry.
• Does losing the message cost money or a support ticket? If no, stop. Fire and forget.
• Are you on JournaledGrain? If yes, keep a published-version cursor, not a list.
• Otherwise, inherit OutboxGrain<TState>, call CommitAsync, move on.

No outbox table. No relay service. No CDC connector. No leader election. The grain is the relay, the state is the outbox, and the runtime you already run gives you the scheduler and the single-writer guarantee.

Pick one grain this week that publishes to Kafka or Service Bus, drop the base class under it, and delete the poller you've been babysitting. If you think I've drawn the boundaries wrong, especially the "grain-to-grain never needs it" position, I want to hear the case.