GitHub - raiyanyahya/zapq: ⚡ A single‑binary Go microservice that exposes a First‑In First‑Out message queue completely in RAM.

⚡ ZapQ In‑Memory FIFO Queue Server

A single‑binary Go microservice that exposes a First‑In / First‑Out message queue completely in RAM. This README explains what every endpoint does, why it works internally and the key computer‑science principles behind each subsystem. This queue deliberately pairs a plain slice + mutex (for data integrity) with lock-free atomic counters (for hot-path metrics) to balance simplicity and high-throughput telemetry. Wrapping that lean core in HTTP/TLS, structured logging, and health endpoints so it can drop straight into micro-service stacks without extra glue.

ZapQ is aiming at a different sweetspot: a single-process, RAM only FIFO that you can spin up with virtually zero config, drop next to a container, and hit at ~µs latencies. No clustering logic, no external store, just a fast, transparent buffer. If you outgrow that or need fan-out, persistence, or stream replay, NATS (or Kafka, or Redis Streams) is absolutely the right move.

TL;DR

Ultra‑low‑latency queue for transient messages (≤ 128 KB each).

Hard caps (maxBytes, maxMsgs) guard RAM.

Concurrency‑safe via a mutex + atomic counters.

JSON metrics, TLS option, graceful shutdown.

1. Architecture Overview

 ┌────────────┐   HTTP/HTTPS    ┌────────────────────────┐   Goroutines   ┌─────────────┐
 │  Producers │ ───────────────▶│  queue ([]byte slices) │◀──────────────▶│  Consumers  │
 └────────────┘                 └────────────────────────┘                └─────────────┘

queue.data – a slice of byte‑slices ([][]byte). Index 0 is always the next item to pop, satisfying FIFO.
queue.mu – a single sync.Mutex serialises structural modifications while permitting parallel HTTP clients.
queue.bytes – running total for O(1) memory‑cap checks.
Atomic counters record events without taking the mutex (cache‑friendly, lock‑free).

Time & Space Complexity

Operation	Time	Space Comments
`enqueue`	O(1) append; amortised as Go grows slice capacity exponentially.
`dequeue`	O(1) slice header moves, old entry set to `nil` for GC.
`clear`	O(n) nils everything so GC can reclaim.

Mutex ensures linearizability (each op appears instantaneous). Because we hold the lock only long enough to modify the slice, throughput scales linearly until contention on enqueue/dequeue dominates.

2. Endpoint Walk‑Through & CS Background

Path / Verb	Semantics	Response	Internals & Rationale
`POST /enqueue`	Push a message (≤ 128 KB).	`202 Accepted` or `429 Queue Full` or `413 Payload Too Large`.	Reads the body into a byte‑slice. Calls `enqueue`. Checks both count‑cap (`maxMsgs`) and byte‑cap (`maxBytes`) in O(1). On success increments `enqueueCnt` atomically.
`GET /dequeue`	Pop next message.	`200` with raw bytes or `204 No Content` if empty.	`dequeue` grabs mutex, returns first item, shifts slice head. Failure path increments `dequeueMiss` without lock.
`POST /clear`	Purge all data.	`200 OK`.	Nils every slice entry so Go’s tri‑colour GC can reclaim memory quickly; resets counters.
`POST /persist?file=path.json`	Snapshot queue to disk (blocking).	`200` or `500`.	Snapshot copy removes lock early to minimise stall, then JSON‑encodes for human‑readability. O(n) IO.
`POST /load?file=path.json`	Replace in‑memory queue with file contents.	`200` or `500`.	Validates file, recomputes `bytes` in one pass.
`GET /metrics`	Human‑friendly JSON telemetry.	`{…}`	Shows uptime, counters, queue size, Go goroutines, and FD usage (reads `/proc/self/fd`). Ideal for dashboards or Prometheus “textfile” collector.
`GET /health`	Liveness probe.	`ok`	Constant‑time, lock‑free.

Why JSON not Protobuf?

Zero client codegen, trivial curl debugging, small payload (few hundred bytes).

3. Compile & Run

Prerequisites

Go ≥ 1.22
(Linux) raise FD limit for heavy load tests: ulimit -n 65535

Quick start (HTTP)

# Build & run with 2 GiB RAM cap and 100 k messages cap
QUEUE_MAX_BYTES=2G go run fifo_queue_server.go \
   --addr :8080 --max-msgs 100000

Enable HTTPS

openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
        -keyout key.pem -out cert.pem -subj "/CN=localhost"

go run fifo_queue_server.go --addr :8443 \
        --tls-cert cert.pem --tls-key key.pem

Now curl -k https://localhost:8443/health ➜ ok.

Flags & Env

Flag / Env	Default	Description
`--addr`	`:8080`	Where to listen (`host:port`).
`--tls-cert/key`	empty	Enable TLS.
`--max-bytes` / `QUEUE_MAX_BYTES`	`256M`	Memory cap (bytes, M, G).
`--max-msgs` / `QUEUE_MAX_MSGS`	`50000`	Count cap.

Flags beat env vars.

4. Graceful Shutdown Explained

The server goroutine listens for SIGINT/SIGTERM:

sigCh := make(chan os.Signal,1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh // blocking

Then runs srv.Shutdown(ctx) which:

Stops Listen accepting new sockets.
Waits for in‑flight handlers to return (10 s timeout).
Active goroutines finish enqueue/ dequeue, so no message corruption.

This follows Go’s graceful‑shutdown contract, ensuring linearizability even during redeploys.

5. Load‑Testing Examples

# 100 parallel enqueues, 50k messages total
hey -m POST -d @<(head -c 16 /dev/urandom | base64) \
    -c 100 -n 50000 http://localhost:8080/enqueue

# In another shell
watch -n1 'curl -s http://localhost:8080/metrics | jq "{len:.queue_length, bytes:.queue_bytes}"'

Observe queue_length grow, then drain as consumers dequeue.

6. Production Checklist (Why It Matters)

Item	Reason
`ulimit -n` ≥ 10 k	Prevent `EMFILE` under heavy concurrency; FD count exposed via `/metrics`.
Caps tuned	Protect node‑RAM; shrink caps in staging.
Process supervisor	Auto‑restart on crash; passes SIGTERM for graceful shutdown.
TLS or upstream termination	Protect messages on the wire (privacy, integrity).
Alerting (`queue_length`, `uptime_sec`)	Spot backlog or unexpected restarts.
Regular `/persist` cron (optional)	Hot‑backup if message loss is costly.
Client retry w/ jitter	Avoid thundering herd when server returns 429.

7. Extending the Server (Ideas)

Replace global mutex with segment locks or a lock‑free ring buffer for higher parallelism.
Add priority queues using a heap instead of FIFO slice.
Swap JSON snapshot with binary protobuf for faster I/O.
Integrate with systemd‑sdnotify for readiness signals.

Pull requests welcome! 🎉

License

MIT – Hack, learn, enjoy.