projects/browser-orchestrator/

02 · browser session orchestrator

Browser Orchestrator

Rust · Restate · axum · Go tester · Jan '26 · source

A Rust-based microservice orchestration system aided by Restate for spawning multiple browser sessions exposed through a single axum server. It uses Restate for durable, stateful execution and axum as a thin HTTP compatibility layer, and it brings durable execution to fragile browser sessions across 1,000+ concurrent sessions.

Summary

client ──HTTP──▶ axum :3000 ──▶ Restate ingress :8080 ──▶ Pool (virtual object) :4000
                 schema adapter        durable, stateful execution  │
                                                                    ├─▶ steel-browser :port₁
                                                                    ├─▶ steel-browser :port₂
                                                                    └─▶ steel-browser :portₙ

Restate is responsible for:

• Persisting the worker/session pool state across failures.
• Performing durable side effects (starting and stopping steel-browser processes) via ctx.run.
• Coordinating retries and recovery during worker startup and health checks.

Each steel-browser instance runs exactly one browser session and exposes its own HTTP API. The orchestrator:

• Spawns workers on dynamically allocated ports.
• Maintains a mapping between sessions and workers.
• Routes session-scoped requests to the correct worker based on persisted state.

Axum serves only as a schema adapter, exposing endpoints compatible with the original steel-browser API and forwarding requests to Restate. All orchestration logic and state management live exclusively in Restate.

The primary design principle is statefulness first: session and worker state is always derived from Restate-managed state, never from in-memory caches.

How the pool behaves

• Workers are reused across sessions. spawn_worker looks for an existing worker that is available, and spawns a new steel-browser only when no worker is free.
• The stale-session and stale-worker pollers schedule themselves as recurring loops through Restate delayed self-calls (every 10s and 60s).
• Health checks run in parallel: one ctx.run future per worker, awaited together with join_all. Sessions on unhealthy workers are dropped too.
• Session TTL expiry uses each session's created_at.
• A missing session returns 404 instead of a 500.

Testing

The tester has been written in Go, encompassing TTL timeout checks, session/worker scaling, health/status checks, and notable edge cases.

Trade-offs

This project optimizes for durability, explicit, persisted state, and correctness under retries and partial failures. It intentionally sacrifices simplicity of deployment, stateless request handling and HTTP-layer ergonomics.

The focus was on Restate's execution and persistence model with orchestrating parallel reliable sessions, rather than building a fully production-ready control plane.

Production gaps & improvements

• Process lifecycle management: add explicit worker supervision (restart, timeout, cleanup on failure).
• Resource exhaustion: replace ad-hoc port scanning with a robust port allocation strategy, and add CPU, memory and file-descriptor limits per worker.
• Backpressure and admission control: session creation is unbounded today, with no rate limiting or queuing during spikes.
• Observability: metrics for active sessions, worker startup latency, and retry counts and failure rates.
• Typed state models instead of raw JSON blobs in Restate, structured responses instead of raw strings, and formal idempotency guarantees for session creation and deletion.

Scaling

The current scaling model assumes one orchestrator per region, with each orchestrator managing workers local to that region. In a Kubernetes-based deployment, workers would be scheduled as pods rather than OS processes, and session routing would rely on Kubernetes networking instead of local ports. Longer-term, the worker abstraction would be generalized so steel-browser is just one backend among others.