Skip to content

nestbox

A tiny, zero-dependency transactional outbox for Postgres, in Go: append in the business transaction, deliver at least once, scale across workers.

The transactional outbox solves the dual-write problem: a service that both commits a row and publishes an event can crash between the two, losing the event, or publish one for a transaction that later rolls back. nestbox is that pattern as a small, dependency-free Go library for Postgres. Append the event inside the business transaction, and a relay delivers it at least once, scaled across as many workers as you like.

The dual-write problem

A write and a publish are two actions, not one. A service that commits a row to its database and then publishes an event to a broker has a gap between the two: crash after the commit and the event is lost forever; publish first and then roll back, and you have announced something that never happened. Retries and two-phase commit each bring their own failure modes. The transactional outbox closes the gap by making the event part of the same transaction as the write.

One transaction, one truth

Append the event where the write already is. nestbox takes the transaction you are already using and adds the event to it, so the row and the event commit together or not at all. The library package depends only on Go’s standard library; a Postgres driver is needed only to open the connection.

tx, _ := db.BeginTx(ctx, nil)

// ... your business write(s) on tx ...

nestbox.Append(ctx, tx, nestbox.Event{
    ID:            newUUID(),
    AggregateType: "sighting",
    AggregateID:   sightingID,
    Type:          "SightingRecorded",
    Payload:       []byte(`{"species":"Turdus merula"}`),
    OccurredAt:    time.Now().UTC(),
})

tx.Commit() // the row and the event commit together, or neither does

Delivered at least once, by any number of workers

Scaling is adding workers, not partitions. A relay drains the outbox on an interval, claiming a batch with SELECT … FOR UPDATE SKIP LOCKED, delivering each event to a handler, and marking it published, all in one transaction. Because locked rows are skipped rather than waited on, every relay takes a disjoint set, so more throughput is just more workers, with no leader election and no partitioning. A handler failure rolls the whole batch back, so an event is never marked published until it is delivered, which is what makes delivery at-least-once (and handlers idempotent).

sink := nestbox.HandlerFunc(func(ctx context.Context, e nestbox.Event) error {
    return publishToKafka(ctx, e) // or SNS, a webhook, anything
})

relay := nestbox.NewRelay(nestbox.NewPostgresStore(db), sink)
relay.Run(ctx) // blocks until ctx is cancelled; run several for throughput
nestbox transactional-outbox flowA database transaction commits a business write and an outbox row together into Postgres. Relay workers claim undelivered outbox rows using SELECT … FOR UPDATE SKIP LOCKED, so each worker takes a disjoint batch, and deliver every event at least once to a handler such as Kafka, SNS or a webhook.one transactionbusiness writenestbox.Appendcommitoutbox tableundelivered rows,oldest firstpublished_at IS NULLSKIP LOCKEDrelay worker 1relay worker 2relay worker NAt-least-once delivery to the handler (Kafka · SNS · webhook); handlers are idempotentA handler error rolls the batch back, so nothing is marked published until it is delivered
One transaction commits the business row and the event together, so neither is lost nor phantom. A pool of relays then claims undelivered rows with FOR UPDATE SKIP LOCKED (each relay gets a disjoint set, so adding workers adds throughput with no partitioning or leader election) and delivers each event at least once to the handler.

Run it

The repo ships a runnable worker and a producer behind a Docker Compose file, so the whole loop is one command: a producer enqueues events in a single transaction, and the worker delivers them.

docker compose up --build -d        # Postgres + the relay worker
docker compose run --rm producer    # enqueue events in one transaction
docker compose logs worker          # watch them get published

The same pattern, a different language

Range is the point, not novelty. The outbox already appears, in Kotlin, inside the bird-platform backend; nestbox is the same idea rebuilt as a standalone, idiomatic Go service, with the concurrency pushed further: Go goroutines and a context-cancelled poll loop, graceful shutdown, and the SKIP LOCKED claim that lets the relay fan out horizontally. It is small on purpose, with unit tests for the relay and Postgres integration tests for the drain and its at-least-once rollback.