Async Task Processing in Python with Celery: Queues, Workers, and an Ops Checklist

Sending email, generating reports, converting thumbnails, calling external APIs in bulk. Handle these inside a web request and responses get slow, timeouts fire, and when something fails there is no way to retry. The answer has been the same for a long time: the request only enqueues the job; execution happens in a separate process. In Python, the de facto standard for this pattern is Celery. This post covers Celery’s architecture, basic usage, and the operational points everyone eventually hits.

The architecture: broker, worker, result backend #

Celery consists of three parts.

When the web app calls task.delay(), a message lands in the broker, and a worker pulls and runs it. Web processes and workers are fully separated, so each scales independently. Broker choice reduces to something simple: if you already run Redis, start with Redis; if message loss is intolerable or you need complex routing, consider RabbitMQ.

The minimal setup #

uv add "celery[redis]"

# tasks.py
from celery import Celery

app = Celery(
    "myapp",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)


@app.task
def send_welcome_email(user_id: int) -> str:
    # real email sending goes here
    return f"sent to user {user_id}"

# run a worker
celery -A tasks worker --loglevel=info

# web-side code: enqueue and return immediately
result = send_welcome_email.delay(42)
print(result.id)          # the task ID
print(result.get(timeout=10))  # wait for the result (not recommended inside a request)

delay() returns immediately. Waiting on result.get() inside a web request defeats the purpose of the queue, so when a result is needed, the common design returns the task ID and lets the client poll a status endpoint.

Retries: failure is the default assumption #

Background jobs mostly talk to the outside world (mail servers, external APIs), so transient failure is routine. Retries are declared as decorator options.

@app.task(
    autoretry_for=(ConnectionError, TimeoutError),  # auto-retry on these exceptions
    max_retries=5,
    retry_backoff=True,       # exponential backoff: 1s, 2s, 4s...
    retry_backoff_max=600,    # backoff cap at 10 minutes
    retry_jitter=True,        # mix in random delay to avoid retry stampedes
)
def call_external_api(payload: dict) -> dict:
    ...

• Naming the retryable exceptions matters. Retry everything and your code bugs (a KeyError, say) get executed five times over.
• Exponential backoff plus jitter is the standard guard against piling retries onto a service that is already down.

Idempotency: design for at-least-once execution #

Celery’s delivery guarantee is fundamentally at-least-once. If a worker dies mid-execution, the same message can run again on another worker; add retries on top, and a task running twice is a matter of when, not if. So the rule is to write tasks so that running twice produces the same result — idempotent.

• Instead of “add 1,000 points,” design it as “record the point grant for order X (ignore if already recorded).” Unique constraints and processing-history tables are the tools.
• For work that is hard to make idempotent — payments, dispatch — use the external service’s idempotency keys.

Operational settings: turn these on before the incident #

app.conf.update(
    task_acks_late=True,           # remove from queue after completion (redeliver on worker death)
    worker_prefetch_multiplier=1,  # limit prefetching when long tasks are in the mix
    task_time_limit=600,           # kill after 10 minutes
    task_soft_time_limit=540,      # raise at 9 minutes to allow cleanup
)

• acks_late: the default is “acknowledge on receipt,” so a worker dying mid-run loses the job. Turning it on switches to acknowledge-after-completion — at the cost of the duplicate-execution possibility above. It is a package deal with idempotency.
• Time limits: one hung task without limits permanently removes a worker slot. The soft limit gives you a cleanup window; the hard limit is the last line.
• Monitoring: run Flower for a web UI over queue length, task successes and failures, and worker status. At minimum, alert on queue length — a steadily growing queue is the earliest signal that workers cannot keep up.

When you do not need Celery #

Celery is powerful, but it brings infrastructure costs: broker operations, worker deployment, monitoring. Lighter options are often the right fit.

• FastAPI BackgroundTasks: the lightest option, running in the same process after the response. No retries, no persistence — use it only for side work that may fail harmlessly (log shipping, cache refresh). Covered in Modern Python in Practice #5.
• RQ, arq: simple Redis-only task queues. RQ is sync, arq is asyncio-based. Far less configuration than Celery — plenty for projects with a handful of task types.
• When Celery fits: many task types, a real need for retries, scheduling (celery beat), and routing, and a team that can carry the operations.

Reduced to one question: “is losing this job an incident?” If yes, go with Celery (or the RabbitMQ combination) and its broker durability and retries. If not, start with a lighter tool.

Summary #

• A task queue is three parts — broker (queue), worker (execution), result backend (status) — with web and workers scaling independently.
• delay() enqueues and returns immediately. Avoid designs that wait for results inside a web request.
• Declare retryable exceptions explicitly and enable exponential backoff plus jitter. Delivery is at-least-once, so design tasks to be idempotent.
• acks_late, time limits, and queue-length alerts are the settings to enable before the incident.
• Work that may fail harmlessly goes to BackgroundTasks; simple queues to RQ or arq; Celery when the features and scale genuinely call for it.