LUPINE is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines.
Quick Start
Use the published GHCR images. The examples below pin CUDA 13.3.1 on
Ubuntu 24.04; other published tags use the same
cuda-<cuda-version>-ubuntu<ubuntu-version> format.
Run the server on the GPU machine:
docker run --rm --gpus all -p 14833:14833 \ ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04
Run the client pointing at that server:
docker run --rm -it \ -e LUPINE_SERVER=<server>:14833 \ ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \ nvidia-smi
Example output from a real run against a remote RTX 4090:
Mon May 18 15:40:46 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.288.01 Driver Version: 590.48.01 CUDA Version: 13.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:01:00.0 On | Off |
| 30% 52C P8 22W / 450W | 8MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
Inside the client container, LD_LIBRARY_PATH=/opt/lupine/lib is already set,
so CUDA driver users pick up the LUPINE libcuda.so.1 shim and NVML users such
as nvidia-smi pick up the LUPINE libnvidia-ml.so.1 shim automatically.
Prometheus Metrics
Linux servers built with CUDA and NVML expose Prometheus metrics on the RPC port without monitoring-specific configuration:
curl http://<server>:14833/metrics
The endpoint reports host GPU memory capacity, memory use and utilization,
plus memory and utilization for each connected client process. It also exports
the mapping between client identity, the Lupine connection child, and the host
PID reported by NVML. Values are collected when /metrics is requested, so
the server does no background NVML polling.
Client compatibility
Each production server executable embeds the matching Linux, macOS, and
Windows client objects for amd64 and arm64; there is no client-bundle directory
to deploy beside it. Python clients fetch the current object from
/.well-known/lupine/client/v1/<os>/<arch>, verify its strong ETag, content
digest, manifest, and file hashes, and cache it locally. The selected ETag is
also asserted when the RPC connection opens, closing the race between
discovery and a server upgrade. LUPINE_LIBDIR remains an explicit local
override for development.
Linux client objects target the manylinux2014 ABI (glibc 2.17) and statically include their private C++, HTTP/2, and TLS dependencies. They therefore work on newer glibc distributions, including Ubuntu 22.04, without requiring host copies of libstdc++, nghttp2, or OpenSSL.
Graceful Server Checkpoints
On Linux, SIGTERM stops the server from accepting connections, asks every
connection child to finish its in-flight CUDA calls, and waits for those
children to exit. This graceful drain happens in the open-source server with
no extra runtime dependency.
Each connection child looks for liblupinecr.so.0, then liblupinecr.so, and
uses the versioned provider ABI in
checkpoint_provider.h. A missing or incompatible
provider is a no-op; the server still drains and exits normally. The provider
is loaded before the child's first CUDA call so it can observe RM/UVM activity
needed to discover allocations.
Set LUPINE_SESSION in the client to attach a stable connection identifier.
The optional provider receives that identifier to restore the connection
before its first CUDA RPC and checkpoint it after shutdown has drained. For an
unkeyed connection, restore is skipped and checkpoint receives a null
identifier. Providers own storage configuration, file layout, and any fallback
policy for unkeyed connections; Lupine does not select a checkpoint directory.
LUPINE_CHECKPOINT_LIBRARY can override the provider library path for a
private deployment.
Connection Stability
Each client/server connection is a single long-lived TCP stream. Long-running workloads sit idle for long stretches (between training steps, during host-side data loading, inside long kernels), and stateful middleboxes — cloud load balancers, NAT gateways, conntrack tables, firewalls — silently reap idle flows far sooner than the kernel's default 2-hour keepalive. The next RPC then fails fatally. Lupine keeps these connections alive and resilient without retrying RPCs (which would break CUDA semantics):
RPC request and response bodies require content-encoding: lz4. Compression
is applied transparently as one LZ4 frame per HTTP/2 body; peers do not
negotiate or fall back to another encoding.
- TCP keepalive is enabled on every connection (client and server) with a 60s idle interval, 15s between probes, and 3 unanswered probes before giving up. Probes are sent only while idle, so active transfers pay no latency cost, and a dead peer is detected in ~105s instead of hanging on the TCP retransmit timer.
- Connect retry rides out a server that is not reachable yet (e.g. still provisioning): a connection is attempted a few times with exponential backoff, and each attempt is bounded by a deadline so a packet-filtered port is detected quickly rather than blocking for the full SYN-retransmit window.
Socket buffer sizes are left to the OS, which auto-tunes on modern kernels.
Trace Logging
Set LUPINE_TRACE on the client, server, or both to enable trace logging.
LUPINE_TRACE=0 or an unset value disables tracing. LUPINE_TRACE=1 writes
trace output to stdout, LUPINE_TRACE=2 writes it to stderr, and any other
non-empty value is treated as a file path opened in append mode.
# trace to stdout LUPINE_TRACE=1 ./your_cuda_program # trace to stderr LUPINE_TRACE=2 ./server # trace to a file LUPINE_TRACE=/tmp/lupine.trace ./your_cuda_program
The same LUPINE_TRACE variable controls both client and server tracing;
LUPINE_SERVER_TRACE is no longer used.
Device printf Forwarding
LUPINE inspects uploaded PTX and cubin symbol data for vprintf, the CUDA device
printf implementation. Until an image that may use device stdout is loaded,
synchronization avoids stdout redirection and its process-global lock, allowing
independent RPC lanes to synchronize concurrently. Fully opaque compressed
fatbins are treated conservatively as potentially using device stdout.
After a device-output-capable image is loaded, context, stream, and event
synchronization captures server fd 1 and forwards the bounded CUDA printf
buffer to the client's stdout. Capture remains process-global so output from
concurrent synchronization lanes is not misattributed.
Multi-GPU Across Multiple Servers
The client accepts a comma-separated LUPINE_SERVER list. Devices are exposed as
one local ordinal list in server order: all GPUs from the first server, then all
GPUs from the next server, and so on.
Run a server on each GPU machine:
# on gpu-host-a docker run --rm --gpus all -p 14833:14833 \ ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04 # on gpu-host-b docker run --rm --gpus all -p 14833:14833 \ ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04
Point the client at both servers:
docker run --rm --network host \ -e LUPINE_SERVER=gpu-host-a:14833,gpu-host-b:14833 \ ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \ nvidia-smi -L
Expected output lists both remote GPUs:
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
CUDA driver applications use the same LUPINE_SERVER value:
docker run --rm --network host \ -e LUPINE_SERVER=gpu-host-a:14833,gpu-host-b:14833 \ ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 \ ./your_cuda_program
Cross-server device-to-device and peer (cuMemcpyDtoD / cuMemcpyPeer) copies are
supported: when the source and destination live on different servers, the client
transparently stages the data through itself (device->host on one server, then
host->device on the other). Direct server-to-server transfers that avoid that
client hop, cross-server peer-access enablement, and cuMemcpy3DPeer are not
implemented yet.
Same-server operations route by handle ownership.
Prefix an endpoint with https:// when the Lupine server is behind a
TLS-terminating proxy. Both CUDA applications and NVML tools such as
nvidia-smi use the scheme and verify the proxy certificate against the
system trust store. HTTPS defaults to port 443; plain and http:// endpoints
default to port 14833.
For a specific CUDA version:
docker pull ghcr.io/lupinemachines/lupine-client:cuda-12.4.1-ubuntu22.04 docker pull ghcr.io/lupinemachines/lupine-server:cuda-12.4.1-ubuntu22.04
Client images contain the CUDA driver, CUDA runtime, cuBLAS, cuBLASLt, cuFFT, cuDNN, cuRAND, cuSPARSE, cuSPARSELt, cuSOLVER, cuSOLVERMg, NVRTC, NCCL, nvJitLink, nvJPEG, NPP, cuFile, CUPTI, nvSHMEM, NVML, and HIP shims, their runtime dependencies,
and nvidia-smi. They are based on Ubuntu and contain neither the CUDA nor ROCm
SDK. The -slim tags remain available as compatibility aliases with the same
SDK-free contents, for example
ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04-slim.
The server image is also based on Ubuntu. It installs the CUDA compatibility runtime and, on amd64, the ROCm HIP runtime so one image can serve either NVIDIA or AMD GPUs; the older separate HIP server Dockerfile is no longer needed.
Slow Start for the Skeptics
This path derives a small PyTorch client image from the published LUPINE client
image and runs the microgpt_train test against a remote GPU. It is
intentionally explicit so it is easy to see which side is the CPU-only client
and which side owns the GPU.
Create a PyTorch client Dockerfile in the repo root:
# Dockerfile.pytorch-lupine FROM ghcr.io/lupinemachines/lupine-client:cuda-13.3.1-ubuntu24.04 ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ python3 \ python3-pip \ && rm -rf /var/lib/apt/lists/* RUN pip3 install --break-system-packages \ --index-url https://download.pytorch.org/whl/cu132 \ torch COPY test/pytorch_lupine_tests.py /opt/lupine/test/pytorch_lupine_tests.py ENV LD_LIBRARY_PATH=/opt/lupine/lib:${LD_LIBRARY_PATH} CMD ["python3", "/opt/lupine/test/pytorch_lupine_tests.py", "microgpt_train"]
Build it:
docker build -f Dockerfile.pytorch-lupine -t lupine-pytorch:cuda-13.3 .Run the server on the GPU machine:
docker run --rm --gpus all -p 14833:14833 \ ghcr.io/lupinemachines/lupine-server:cuda-13.3.1-ubuntu24.04
Run the PyTorch client from the CPU-only machine:
docker run --rm \ -e LUPINE_SERVER=<server>:14833 \ lupine-pytorch:cuda-13.3
Expected success looks like:
microgpt first_loss=... last_loss=...
microgpt_train: PASS
Local development
Building the binaries requires running codegen first. The repository provides a containerized runner so local development and CI use the same CUDA and HIP headers, Python, parser, and formatter versions. Docker is the only host dependency.
Run codegen
Ensure there are no errors in the output of the codegen.
Run cmake
cmake -S . -B build
cmake --build buildCMake builds the CUDA driver shim at build/libcuda.so.1, the CUDA runtime shim
at build/libcudart.so.<major>, the cuBLAS, cuBLASLt, cuFFT, cuRAND,
cuSPARSE, cuSOLVER, cuSOLVERMg, NVRTC, nvJitLink and nvJPEG shims at
build/libcublas.so.<major>, build/libcublasLt.so.<major>,
build/libcufft.so.<major>, build/libcurand.so.<major>,
build/libcusparse.so.<major>, build/libcusolver.so.<major>,
build/libcusolverMg.so.<major>, build/libnvrtc.so.<major>,
build/libnvJitLink.so.<major>, build/libnvjpeg.so.<major>, the NPP shims at
build/libnppc.so.<major> and its image and signal libraries
(build/libnppial.so.<major> through build/libnpps.so.<major>) (when the
toolkit's library headers are
present; nvJitLink needs CUDA 12.4 or newer), the cuDNN shim at build/libcudnn.so.9 (when cuDNN 9 headers are found beside
the toolkit's or through -DLUPINE_CUDNN_INCLUDE_DIR=<dir>), the NCCL shim at
build/libnccl.so.2 on Linux (when NCCL 2.14.3 or newer headers are found
beside the toolkit's or through -DLUPINE_NCCL_INCLUDE_DIR=<dir>), the cuFile
shim at build/libcufile.so.0 on Linux (when cufile.h is found beside the
toolkit's or through -DLUPINE_CUFILE_INCLUDE_DIR=<dir>), the CUPTI shim at
build/libcupti.so.<major> on Linux (build/libcupti.so.11.8 on CUDA 11, whose
CUPTI carries the minor in its SONAME; when cupti_result.h is found beside the
toolkit's or through -DLUPINE_CUPTI_INCLUDE_DIR=<dir>), the NVML
toolkit's or through -DLUPINE_CUFILE_INCLUDE_DIR=<dir>), the nvSHMEM shim at
build/libnvshmem_host.so.3 on Linux (when nvSHMEM 3 headers carrying
nvshmem_host.h are found beside the toolkit's or through
-DLUPINE_NVSHMEM_INCLUDE_DIR=<dir>), the NVML
toolkit's or through -DLUPINE_CUFILE_INCLUDE_DIR=<dir>), the cuSPARSELt shim
at build/libcusparseLt.so.0 (when cuSPARSELt 0.6 or newer headers are found
beside the toolkit's, through CUSPARSELT_HOME or through
-DLUPINE_CUSPARSELT_INCLUDE_DIR=<dir>), the NVML
shim at build/libnvidia-ml.so.1, the HIP shim at build/libamdhip64.so.1, and
the server at build/lupine_driver_server. The runtime and library shims cover
their whole APIs: they forward cuda*, cublas*, cublasLt*, cufft*,
cudnn*, curand*, cusparse*, cusparseLt*, cusolver*, nvrtc*,
nvJitLink*, nccl*, nvjpeg* and npp* calls on the driver shim's
connections, so all of them must come from the same build. NVRTC compiles and nvJitLink links on the server, so their
output matches the server's toolkit and driver; the files a program includes or
links from the client's disk are sent along with it. The server loads the
machine's libcudnn.so.9, libnccl.so.2 and libcusparseLt.so.0 by name. nvJPEG decodes and encodes
on the server with the server library's default allocators, so a buffer it
hands back through a retrieve call is a server address.
An NPP call with a stream context runs on the server that owns the context's
stream, or on the current device's server for the default stream, and its
images, scratch buffers and results must be device memory on that server. The
contour calls that fill host lists sized by an earlier call's outputs
(nppiCompressedMarkerLabelsUFInfo_32u_C1R_Ctx and its geometry list and
interpolation calls) return NPP_NOT_IMPLEMENTED_ERROR.
A cuSPARSELt object (handle, matrix descriptor, matmul descriptor, algorithm selection, plan) is caller storage the library fills with state it links to its other objects by address, so it lives on the server and the caller's storage holds its address there. The initializing call allocates it and the matching Destroy releases it; storage that was never initialized through the shim names no object. Its matrices, compressed buffers, workspaces and pruning validity flags are device memory on the server that owns the handle.
cuFile is the exception to that forwarding. GPUDirect Storage moves bytes
between a storage device and GPU memory without the host, and no DMA spans a
client and a server, so the shim runs the compatibility path cuFile itself
falls back to without nvidia-fs, with its halves on the two machines: the
client reads the file and the driver shim moves the staging buffer. cuFileRead
and cuFileWrite keep their contract, the transfer being staged rather than
direct, while cuFileDriverGetProperties reports no GPUDirect capability and
the nvidia-fs tunables return CU_FILE_PLATFORM_NOT_SUPPORTED, so a program
asking what the platform supports is told. Nothing reaches the server but the
copies, and it needs no libcufile of its own.
CUPTI is the other exception, and a starker one. It profiles the process it is loaded into by hooking the driver and runtime calls that process makes, and a client makes none: its CUDA calls are RPCs and the work runs on the server. A server-side CUPTI would report the server's threads, clock, correlation ids and
- where a server holds more than one client's connections - the other clients'
work, so a timeline built from it would be wrong in ways its reader could not
see. The shim therefore profiles nothing and says so:
cuptiGetVersion,cuptiGetResultString,cuptiGetErrorMessageandcuptiGetLastErroranswer, every other entry point returnsCUPTI_ERROR_NOT_SUPPORTEDand the first refusal prints one line to stderr.torch.profilerand Nsight Systems' CUPTI path report no GPU activity on a lupine client, rather than a fabricated timeline; the library loads, which is what PyTorch'slibtorch_cpu.soneeds from itsNEEDEDentry. nvSHMEM is the other exception, and a starker one: it forwards nothing at all. Its PEs hold a partitioned global address space over the GPUs of a job, reading and writing each other's symmetric heap from inside kernels, and a client is not one of them. The process that owns a GPU here is the server, which serves many clients at once, while nvSHMEM keeps its PE identity, its symmetric heap and its teams in process-global state with no handle to tell one job from another; its heap sits outside the identity VA arena a client reserves; and its device half never reaches a host shim. So the shim loads, answers the version and status queries truthfully, and refuses everything else -nvshmemid_hostlib_init_attrwithNVSHMEMX_ERROR_NOT_SUPPORTED, the allocators and peer pointers with null, the collectives with an error - rather than return a value it cannot mean. That is enough for PyTorch, whoselibtorch_cuda.soreacheslibnvshmem_host.so.3throughlibtorch_nvshmem.soin itsNEEDEDand never calls it unless a program asks for symmetric memory.
A communicator whose ranks sit behind different servers needs those servers to
reach each other: NCCL's bootstrap and transport run between the server
processes, so settings such as NCCL_SOCKET_IFNAME belong in each server's
environment. Drive such ranks from a thread each, as separate processes would;
a group reaching two servers from one thread returns ncclInvalidUsage.
Redistributable server builds pass LUPINE_CLIENT_BUNDLE_INPUT with staged
native client directories. CMake deterministically assembles all six platform
routes and links them into lupine_driver_server. The Docker server target
requires that generated registry, so a server image cannot be produced without
the clients.
The Lupine server must be running before initiating client commands.
If successful, the server will start:
Server listening on port 14833...
Running the client
For local development, preload the built libcuda.so.1 before executing CUDA
commands. The published client image sets LD_LIBRARY_PATH for you instead.
Once the server above is running:
# update to your desired IP/port export LUPINE_SERVER=<server>:14833 LD_PRELOAD=./build/libcuda.so.1 python3 -c "import torch; print(torch.cuda.is_available())" # or LD_PRELOAD=./build/libcuda.so.1 nvidia-smi
You can also use the local shell script to run your commands.
Questions
- What does LUPINE stand for? Nothing, it just looks cool in all caps.
- Does this support authentication? TLS? Indirectly, yes. It's a plain HTTP/2 server, so you can front it with whatever TLS/auth server you want.
- Was this repo AI-generated? A chunk of it, yes. I mean, would you want to hand write hundreds of tedious API stubs? No? Me neither.
- Doesn't this incur a lot of latency? Surprisingly, no! You will see device transfers get slower because this is basically bottlenecking a PCIe link over the network, but there is very little overhead besides that. For things like model training and inference, once the model is on the GPU very little data transfer happens to the host. As a result, it might be faster than you expect.
- Can I do remote video encoding/decoding? This is probably one use case we wouldn't recommend because that's a lot heavier on the PCIe link. It works in theory though, so if you do have access to a 1 Tbps link it might work for you.
Prior Art
This project is inspired by some existing proprietary solutions: