Background Jobs in Go with Asynq and Valkey

Every Go backend eventually needs background jobs. You start with a goroutine, then you need retries, then deduplication, then you realize a server restart just killed 2,000 half-processed tasks.

I hit this wall on Markwise, an all-in-one productivity app I run as a hobby project. I use side projects like this to experiment with tools and patterns I'd never try directly in production at work. The learnings feed back into my day job, and the rough edges make for better blog posts.

Here's how I set up durable background processing with Asynq (v0.25) and Valkey (v8), and why I picked them over the alternatives (goroutines, Redis, AWS SQS, etc.)

The problem

Markwise's Go API needs to do several things that don't belong in an HTTP request/response cycle:

• Lifecycle emails -- activation nudges, search prompts, and daily digests on cron schedules
• Bulk imports -- process hundreds of URLs, fetch metadata, categorize with AI
• Third-party sync -- push 2,000+ contacts to Resend with rate limiting (1.5s per contact = ~50 minutes)

My first attempt was goroutines:

go func() {
    result, err := audienceSvc.SyncAllUsers(ctx)
    if err != nil {
        slog.Error("sync failed", "error", err)
    }
}()

This works until it doesn't. The ctx gets cancelled the moment the HTTP handler returns, so the sync may die mid-flight even without a restart. No retries. No visibility into whether it's running. No way to prevent duplicate runs. And behind Cloudflare, any request over 100 seconds gets a 524 timeout anyway.

The stack

Valkey is a Redis-compatible in-memory store, forked from Redis after the license change. It stores tasks in lists, supports pub/sub for worker coordination, and persists data to disk so tasks survive restarts.

Asynq is a Go library that turns Valkey into a proper job queue. Enqueueing, worker pools, retries, scheduling, deduplication, and timeouts. Think Sidekiq for Go.

Why Asynq + Valkey over alternatives

For a solo-founder SaaS on a single VPS, Asynq hits the sweet spot. No extra infrastructure beyond the Valkey instance I already had.

Why Valkey over Redis

Valkey is Redis with a BSD license. Same protocol, same commands. Every Redis client library works with it. I swapped redis:// for valkey:// in my config and nothing else changed.

Setting it up

1. Define task types

const (
    TaskActivationEmails  = "lifecycle:activation_emails"
    TaskDailyResurface    = "lifecycle:daily_resurface"
    TaskImportProcessURLs = "import:process_urls"
    TaskAudienceSyncAll   = "audience:sync_all"
)

Namespace tasks by domain (lifecycle:, import:, audience:). Makes logs readable when you're tailing output at 2am.

2. Create the scheduler

type Scheduler struct {
    scheduler *asynq.Scheduler
    server    *asynq.Server
    client    *asynq.Client
    mux       *asynq.ServeMux
    db        *mongo.Database
}

func New(db *mongo.Database) (*Scheduler, error) {
    redisOpt := asynq.RedisClientOpt{Addr: "localhost:6379"}

    return &Scheduler{
        db: db,
        scheduler: asynq.NewScheduler(redisOpt, nil),
        server: asynq.NewServer(redisOpt, asynq.Config{
            Concurrency: 3,
            Queues: map[string]int{
                "imports":   1,  // priority weight, not worker count
                "lifecycle": 1,
                "default":   1,
            },
        }),
        client: asynq.NewClient(redisOpt),
        mux:    asynq.NewServeMux(),
    }, nil
}

Three queues with equal priority weights. Concurrency: 3 means 3 workers total, distributed across queues by weight ratio (1:1:1 = equal). This runs in-process alongside the API server. No separate worker binary needed.

3. Register handlers

s.mux.HandleFunc(TaskActivationEmails, func(ctx context.Context, _ *asynq.Task) error {
    ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
    defer cancel()
    return s.OnActivationEmails(ctx)
})

s.mux.HandleFunc(TaskAudienceSyncAll, func(ctx context.Context, _ *asynq.Task) error {
    ctx, cancel := context.WithTimeout(ctx, 2*time.Hour)
    defer cancel()
    return s.OnAudienceSyncAll(ctx)
})

Derive from Asynq's context, not context.Background(). Asynq's context carries cancellation signals from the Timeout option and server shutdown. If you create a fresh background context, your task won't know when the server is stopping gracefully.

Handlers are thin wrappers. Business logic lives in domain services, injected via callbacks to avoid circular imports.

4. Register cron schedules

s.scheduler.Register("0 10 * * *", asynq.NewTask(TaskActivationEmails, nil),
    asynq.Queue("lifecycle"),
    asynq.Unique(24*time.Hour),  // skip if already ran today
)

s.scheduler.Register("0 8 * * *", asynq.NewTask(TaskDailyResurface, nil),
    asynq.Queue("lifecycle"),
    asynq.Unique(24*time.Hour),
)

s.scheduler.Register("*/5 * * * *", asynq.NewTask(TaskCampaignRetries, nil),
    asynq.Queue("lifecycle"),
    asynq.Unique(5*time.Minute),
)

Unique prevents duplicate enqueuing within the specified window. If the cron fires while an identical task is already queued or recently completed, Asynq rejects the duplicate. Note that Unique uses a Redis key with a TTL, so it guards against concurrent enqueuing, not concurrent execution. For most cron jobs, this is exactly what you need.

5. Enqueue on-demand tasks

func (s *Scheduler) EnqueueAudienceSyncAll() error {
    task := asynq.NewTask(TaskAudienceSyncAll, nil)
    _, err := s.client.Enqueue(task,
        asynq.Queue("default"),
        asynq.MaxRetry(1),
        asynq.Timeout(2*time.Hour),
        asynq.Retention(24*time.Hour),
        asynq.Unique(1*time.Hour),
    )
    return err
}

The HTTP handler enqueues and returns 202 Accepted immediately:

func HandleSyncAllUsers(w http.ResponseWriter, r *http.Request) {
    if err := EnqueueSyncAllFunc(); err != nil {
        httputil.EncodeErrorResponse(w, 500, "ENQUEUE_ERROR", err.Error())
        return
    }
    httputil.EncodeSuccessResponse(w, 202, map[string]string{
        "status": "queued",
    }, "Sync started. Check logs for progress.")
}

No Cloudflare timeout. No orphaned goroutine. The task runs to completion even if the HTTP connection dies.

Task configuration cheat sheet

Lessons from production

Start with a queue from day one. I wrote goroutines first because "I'll add proper queuing later." That later arrived when a server restart killed a 50-minute sync halfway through. The migration to Asynq was easy. Cleaning up the data inconsistency wasn't.

Use asynqmon from the start. Asynq ships a web UI for inspecting queues, retrying failed tasks, and viewing task history. I went months relying on slog output alone. Don't repeat that mistake.

Set Retention on every task. Without it, completed tasks disappear immediately. With 24h retention, you can answer "did that cron actually run last night?" without digging through log aggregation.

Running in production

The full setup runs on a single Hostinger VPS ($10/mo):

Valkey barely registers on resource usage. Asynq workers add negligible overhead when idle. The entire job infrastructure costs nothing beyond the Valkey container memory.

For a solo founder running a SaaS, this is the right level of complexity. Not a goroutine with a prayer, not a Temporal cluster with three containers. Just a queue that works. The full scheduler implementation is about 300 lines of Go.

If you found this useful, I write about Go, platform engineering, and building products solo. Follow me on Twitter or check out my other blog posts.