Skip to content

Architecture

SnerdMQ uses an embedded sidecar architecture. Instead of running a separate queue server (like Redis or RabbitMQ), each application instance spawns its own lightweight Rust daemon as a child process. Multiple instances can share the same queue by pointing at the same file on a shared volume.

Overview

┌──────────────────────────────────────────────────┐
│  Application Server (Node, Python, Go, etc.)      │
│                                                    │
│  ┌──────────────────────────────────────────────┐ │
│  │  SnerdMQ SDK (thin client)                   │ │
│  │                                              │ │
│  │  • Enqueue tasks       • Register handlers   │ │
│  │  • Stream progress     • Serve dashboard     │ │
│  │  • JSON-RPC over stdin/stdout                │ │
│  └──────────────────┬───────────────────────────┘ │
│                     │                              │
│  ┌──────────────────▼───────────────────────────┐ │
│  │  snerdmq daemon (Rust sidecar process)       │ │
│  │                                              │ │
│  │  ┌─────────────┐  ┌──────────────────────┐  │ │
│  │  │ Append-only │  │ Priority Queue       │  │ │
│  │  │ Log (disk)  │  │ (Binary Max-Heap)    │  │ │
│  │  └─────────────┘  └──────────────────────┘  │ │
│  │                                              │ │
│  │  ┌─────────────┐  ┌──────────────────────┐  │ │
│  │  │ Rate        │  │ Cron Scheduler       │  │ │
│  │  │ Limiter     │  │                      │  │ │
│  │  └─────────────┘  └──────────────────────┘  │ │
│  │                                              │ │
│  │  ┌─────────────────────────────────────────┐ │ │
│  │  │ OS File Locking (flock / fs3)           │ │ │
│  │  └─────────────────────────────────────────┘ │ │
│  └──────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────┘

The Daemon

The snerdmq binary is a single, statically-compiled Rust executable (~5 MB) that handles all queue orchestration:

Responsibility Implementation
Persistence Append-only JSON-lines log (.snerdata/tasks/tasks.log)
Concurrency control OS-level file locking via fs3 (flock on Linux/macOS, LockFileEx on Windows)
Task ordering Binary Max-Heap keyed on urgency_score for priority dispatch
Retry logic Configurable backoff with retry_after_hours
Rate limiting Rolling-window velocity enforcement per rate_limit_group
Scheduling Cron expression parsing for recurring jobs
Deduplication xxHash-based payload fingerprinting
Webhook dispatch HTTP POST with X-SnerdMQ-Event headers
Hard timeouts tokio::time::timeout enforcement per task
Log compaction Periodic rewrite to reclaim space from deleted/completed tasks

The daemon is deliberately not a network service. It communicates exclusively over stdin/stdout pipes, which means:

  • No port conflicts
  • No firewall rules
  • No authentication tokens
  • No network latency

The SDK Contract

Every SDK — regardless of language — follows the same contract:

  1. Spawn the snerdmq binary as a child process
  2. Send JSON messages to the daemon's stdin
  3. Read JSON messages from the daemon's stdout
  4. Dispatch incoming messages to registered handlers

JSON-RPC Protocol

The protocol is newline-delimited JSON. Each line is a complete JSON object with an action field.

SDK → Daemon (enqueue a task): 

{
  "action": "enqueue",
  "task_id": "email-123",
  "task_type": "send_email",
  "task_data": "{\"to\": \"user@example.com\"}",
  "max_retries": 3,
  "retry_after_hours": 0.5,
  "auto_dedupe": true,
  "urgency_score": 0.0,
  "rate_limit_group": "email_api",
  "max_per_minute": 100
}

Daemon → SDK (execute a task): 

{
  "action": "execute",
  "task_id": "email-123",
  "task_type": "send_email",
  "task_data": "{\"to\": \"user@example.com\"}"
}

Daemon → SDK (acknowledge enqueue): 

{
  "action": "ack",
  "task_id": "email-123"
}

SDK → Daemon (report result): 

{
  "action": "result",
  "task_id": "email-123",
  "success": true
}

Message Types

Direction Action Purpose
SDK → Daemon enqueue Add a task to the queue
SDK → Daemon result Report task execution outcome
SDK → Daemon progress Send progress update for a running task
Daemon → SDK execute Dispatch a task for execution
Daemon → SDK ack Confirm task was enqueued
Daemon → SDK error Report an error (e.g., duplicate task)
Daemon → SDK progress Forward progress event to dashboard
Daemon → SDK max_retries_reached Dead Letter Queue event

The Append-Only Log

All task state is persisted to a single file: .snerdata/tasks/tasks.log

Each line is a JSON object representing the latest state of a task. When a task is updated (retry, completion, deletion), a new line is appended — the old line is never modified in place.

{"task_id":"email-123","task_type":"send_email","task_data":"{...}","retry_count":0,...}
{"task_id":"email-123","task_type":"send_email","task_data":"{...}","retry_count":1,"last_error":"timeout",...}
{"task_id":"email-123","task_type":"send_email","task_data":"{...}","deleted_at":"2026-08-18T10:30:00Z",...}

Why Append-Only?

  • Crash safety — No partial writes. Each line is atomically appended.
  • Audit trail — Full history of every state transition.
  • Simplicity — No database engine, no WAL, no checkpointing.
  • Compaction — Periodically, the daemon rewrites the log keeping only the latest state of each task, reclaiming space.

File Locking

Multiple processes can safely read and write to the same log file concurrently using OS-level file locks:

  • Linux/macOS — flock(2) system call
  • Windows — LockFileEx API

This is what enables the shared-volume scaling model (see Deployment).

Task Lifecycle

                    ┌──────────┐
                    │ Enqueued │
                    └────┬─────┘
                         │
                    ┌────▼─────┐
              ┌─────│  Active  │─────┐
              │     └────┬─────┘     │
              │          │           │
         ┌────▼────┐ ┌──▼───┐ ┌────▼────────┐
         │ Failed  │ │ Done │ │ Timed Out   │
         └────┬────┘ └──────┘ └────┬────────┘
              │                     │
         ┌────▼──────────┐         │
         │ Retry?        │◄────────┘
         │ (if retries   │
         │  remaining)   │
         └────┬────┬─────┘
              │    │
          Yes │    │ No
              │    │
    ┌─────────▼┐ ┌─▼───────────┐
    │ Scheduled│ │ Dead Letter │
    │ (backoff)│ │ Queue (DLQ) │
    └──────────┘ └─────────────┘
  1. Enqueued — Task is persisted to the log and awaiting dispatch
  2. Active — Task has been dispatched to an SDK handler for execution
  3. Completed — Handler returned success; task is soft-deleted from the queue
  4. Failed — Handler threw an error; task is scheduled for retry or moved to DLQ
  5. Timed Out — Execution exceeded max_execution_seconds; treated as a failure
  6. Retry — Task's retry_after_time is set; it will be re-dispatched after the backoff
  7. Dead Letter Queue — All retries exhausted; max_retries_reached event is fired

Embedded Libraries vs. Daemon SDKs

SnerdMQ offers two ways to use the queue engine:

Daemon SDKs (Node, Python, Go, Ruby, PHP, Java, .NET) Embedded Libraries (snerd-rust, snerd-go)
Architecture SDK spawns the Rust daemon as a child process Engine runs in-process (no separate binary)
Language Any language with SDK Rust or Go only
Performance IPC overhead (microseconds per message) Zero IPC — direct function calls
Polyglot Multiple SDKs share the same log file Same log file format — can share with daemon SDKs
Binary required Yes (auto-downloaded on install) No (compiled into your app)
Use when You want language flexibility or polyglot services You want maximum performance in Rust/Go

Both approaches use the same log file format and same OS file locking, so an embedded Rust worker and a Node.js worker can share the exact same queue.