The interesting thing about await is that it is a place, not an operation.
When execution reaches an await and the awaited thing is not ready, CPython does not block the
thread. It stops the frame, keeps the instruction pointer, keeps every local variable exactly where
it was, and hands control back to the event loop. That frame sits in memory as an ordinary heap
object until someone sends a value into it. The whole point of async def is that a function can be
half-finished and still be a value you can hold.
Hold that thought next to a second one: if a paused function is a value, and a value can be serialised, then a paused function can be moved. That is the entire premise behind distributed coroutines: treating paused functions as units of distributed work rather than units of concurrency.
What a suspended Python coroutine is made of
Concretely, a coroutine object owns a frame, and a frame owns four things worth naming: a reference
to the code object, the instruction offset at which it stopped (f_lasti), an array of local
variable slots, and a value stack. Everything else in async/await — the event loop, the future,
the task wrapper — is machinery around that.
You can look at some of it from Python. coro.cr_frame.f_locals will show you the named locals of a
suspended coroutine. coro.cr_code.co_varnames tells you which names exist. coro.cr_suspended
(3.11+) tells you whether it is currently parked at an await. What you cannot do is the obvious
next thing:
import pickle
async def transfer(account, amount):
ok = await debit(account, amount)
return ok
c = transfer("acct_9931", 4200)
pickle.dumps(c)
# TypeError: cannot pickle 'coroutine' objectCPython declines, and the refusal is correct: a frame’s value stack can hold arbitrary interpreter temporaries, and there is no general way to write those down. Which is why every runtime working in this area stops trying to serialise the object and starts serialising a description of it.
What travels when a coroutine goes distributed
A distributed runtime does not send the frame. It sends: the qualified name of the function, the arguments it was called with, the offset it stopped at, and a dictionary of the local variables that have been assigned so far. On the far side, it imports the function, builds a fresh frame, writes the locals back into their slots, and resumes at the recorded offset.
This is why the constraint that surfaces in the API is always about values, not about control
flow. Coroutines in Python can branch, loop, nest and call other coroutines freely; what they may
not do is hold a local that cannot be written down. A socket. An open file. A database cursor. A
thread lock. A weakref. Each of those is a handle to something that only exists in the process
that made it, and the moment one is live across an await that the runtime intends to migrate, the
suspension point becomes un-serialisable.
The practical shape of the rule, in one line: acquire resources after the await, not before it.
Where CPython makes this hard
Three specific places, all of which explain why the projects in this space are conservative about version support, and all three are reasons Python coroutines are harder to move than the premise suggests.
The frame layout moved. Python 3.11’s specialising interpreter (the same rework that
changed how CPython assembles a stack trace)
reorganised frames
substantially: locals moved into the interpreter’s data stack, f_locals became a proxy computed
on demand rather than a real dictionary, and f_lasti changed units. Code that read frames on 3.10
did not read frames on 3.11. This is not a stable interface and was never advertised as one.
f_locals writes did not stick. Historically, assigning through frame.f_locals was a no-op
for real function frames, because the proxy was rebuilt from the fast-locals array each time you
touched it. PEP 667, in 3.13, made the proxy write-through, which is the change that turned “read
the locals” into “restore the locals” without a C extension.
Offsets are not portable across builds. f_lasti is an index into a specific compilation of a
specific source file by a specific interpreter version. Serialise a suspension point on 3.12 and
resume it on 3.13 and you are pointing into a different bytecode sequence entirely. Every payload
therefore has to carry, and check, the interpreter and code-object identity it was produced against.
The failure mode if you skip that check is not an exception. It is resuming in the middle of a
different function, which is considerably worse.
The security edge, since it comes up late
Resume means pickle.loads on bytes that arrived from somewhere else. pickle executes code by
design; that is not a bug in pickle, it is its contract. So the store holding suspended coroutines
is, exactly, a remote code execution surface for every worker that reads it. Sign the payloads,
authenticate the transport, and do not accept a resume request from anything you would not accept a
shell command from. This piece is not going to work through a scheme for that; it is a real design
problem and it deserves more than a paragraph. But a design that treats the queue as trusted is
a design with a hole in it, and the queue brings
its own catalogue of failure modes besides.
What async and await buy here, and what they do not
The gain is that a long-running process stops being tied to one machine’s uptime. A coroutine parked on a five-minute external call does not need a worker sitting there holding a frame; the frame can be written down, the worker can be recycled, and the resume can land wherever there is capacity. That is the whole argument for distributed coroutines in Python, and it is a narrow argument: it applies to waiting, not to computing. Failure stops being “restart the job” and becomes “resume the frame”, which is a genuinely different operational posture, and the reason Python coroutines keep coming up in this context at all.
The cost is that async and await acquire a second meaning. In ordinary Python, await says
“this may take a while, let something else run”. In a distributed setting it also says “this is a
point at which my state may be written to disk and my process may cease to exist”. Those are very
different promises to be carrying on the same keyword, and reviewing code where they overlap is
genuinely harder than reviewing either alone.
Whether that trade is worth it depends on something measurable, which is a relief after all the architecture: how large are the locals at your suspension points? Small and named (a few IDs, a cursor position, an amount) and migration is nearly free. Large and opaque — a parsed document, a model, a buffer — and you are paying serialisation costs on every hop to avoid a restart that would have cost less. Python coroutines are cheap to pause and cheap to resume; only the payload decides whether they are cheap to move. The frame is cheap to move only while it is small, and nothing in the API tells you when you have crossed that line.