Pod-first development - Runpod Documentation

Developing machine learning applications often requires powerful GPUs, making local development challenging. Instead of repeatedly deploying your worker to Serverless for testing, you can develop on a Pod first and then deploy the same Docker image to Serverless when ready. This “Pod-first” workflow lets you develop and test interactively in a GPU environment, then seamlessly transition to Serverless for production. You’ll use a Pod as your cloud-based development machine with tools like Jupyter Notebooks and SSH, catching issues early before deploying your worker to Serverless.

Requirements

• You’ve created a Runpod account.
• You’ve installed Python 3.x and Docker and configured them for your command line.
• Basic understanding of Docker concepts and shell scripting.

Step 1: Set up your project structure

Create a directory for your project and the necessary files:

This creates the following project structure:

Step 2: Create the handler

This Python script will check for a MODE_TO_RUN environment variable to determine whether to run in Pod or Serverless mode. Add the following code to handler.py:

Key features:

• MODE_TO_RUN = os.getenv("MODE_TO_RUN", "pod"): Reads the mode from an environment variable, defaulting to pod.
• async def handler(event): Your core logic.
• if mode_to_run == "pod" ... else: This conditional controls what happens when the script is executed directly.
  • In pod mode, it runs a sample test call to your handler function, allowing for quick iteration.
  • In serverless” mode, it starts the Runpod Serverless worker.

Step 3: Create the start.sh script

The start.sh script serves as the entrypoint for your Docker container and manages different operational modes. It reads the MODE_TO_RUN environment variable and configures the container accordingly. Add the following code to start.sh:

Here are some key features of this script:

• case $MODE_TO_RUN in ... esac: This structure directs the startup based on the mode.
• serverless mode: Executes handler.py, which then starts the Runpod Serverless worker. exec replaces the shell process with the Python process.
• pod mode: Starts up the JupyterLab server for Pod development, then runs sleep infinity to keep the container alive so you can connect to it (e.g., via SSH or docker exec). You would then manually run python /app/handler.py inside the Pod to test your handler logic.

Step 4: Create the Dockerfile

Create a Dockerfile that includes your handler and startup script:

Key features of this Dockerfile:

• FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04: Starts with a Runpod base image that comes with nginx, runpodctl, and other helpful base packages.
• ARG WORKSPACE_DIR=/workspace and ENV WORKSPACE_DIR=${WORKSPACE_DIR}: Allows the workspace directory to be set at build time.
• WORKDIR $WORKSPACE_DIR: Sets the working directory to the value of WORKSPACE_DIR.
• COPY requirements.txt ./requirements.txt and RUN pip install ...: Installs Python dependencies.
• COPY . .: Copies all application files into the workspace directory.
• ENV MODE_TO_RUN="pod": Sets the default operational mode to “pod”. This can be overridden at runtime.
• CMD ["$WORKSPACE_DIR/start.sh"]: Specifies start.sh as the command to run when the container starts.

Step 5: Build and push your Docker image

Now you’re ready to build your Docker image and push it to Docker Hub:

Step 6: Testing in Pod mode

Now that you’ve finished building our Docker image, let’s explore how you would use the Pod-first development workflow in practice. Deploy the image to a Pod by following these steps:

1. Navigate to the Pods page in the Runpod console.
2. Click Deploy.
3. Select your preferred GPU.
4. Under Container Image, enter YOUR_USERNAME/dual-mode-worker:latest.
5. Under Public Environment Variables, select Add environment variable and add:
   • Key: MODE_TO_RUN
   • Value: pod
6. Click Deploy.

Once your Pod is running, you can:

• Connect via the web terminal, JupyterLab, or SSH to test your handler interactively.
• Debug and iterate on your code.
• Test GPU-specific operations.
• Edit handler.py within the Pod and re-run it for rapid iteration.

Step 7: Deploy to a Serverless endpoint

Once you’re confident with your handler.py logic tested in Pod mode, you’re ready to deploy your dual-mode worker to a Serverless endpoint.

1. Navigate to the Serverless page in the Runpod console.
2. Click New Endpoint.
3. Click Import from Docker Registry.
4. In the Container Image field, enter your Docker image URL: docker.io/YOUR_USERNAME/dual-mode-worker:latest, then click Next***.
5. Under Environment Variables, add:
   • Key: MODE_TO_RUN
   • Value: serverless
6. Configure your endpoint settings (GPU type, workers, etc.).
7. Click Deploy Endpoint.

The same image will be used for your workers, but start.sh will now direct them to run in Serverless mode, using the runpod.serverless.start() function to process requests.

Step 8: Test your endpoint

After deploying your endpoint in to Serverless mode, you can test it by sending API requests to your endpoint.

1. Navigate to your endpoint’s detail page in the Runpod console.
2. Click the Requests tab.
3. Use the following JSON as test input:

4. Click Run.

After a few moments for initialization and processing, you should see output similar to this:

Explore the Pod-first development workflow

Now, let’s explore the recommended iteration process for a Pod-first development workflow:

This iterative loop (write your handler, update the Docker image, test in Pod mode, then deploy to Serverless) enables you to rapidly develop and debug your Serverless workers.