Unified DataCenter & Local Foundation Model Serving: Beyond Docker Way

Is generalized Docker solution good for Foundation Model serving?

Right now, when we talk about deploying AI models, most people think about putting these models and their framework dependencies into a Docker image and then exposing a RESTful API endpoint. Cloud services divide a big server into smaller virtual machines (VMs), each with a few GPUs. Kubernetes (k8s), helps manage small VM resources, pull docker images, and orchestrate everything during deployment and scaling.

This method is neat and clean for providing model inference as a microservice, but it has its downsides. One big issue is the ‘cold start time’ — the time it takes to get the service running. It can sometimes be a few minutes. Plus, the whole system is pretty complex.

Another important question is whether this general-purpose container method is the most efficient for serving AI models, especially in the era of foundation models.

Our solution: Multiple Layers Model Caching System

In early 2023, we realized that Docker + Framework solution had an issue: they were packed with too many unnecessary dependencies. That’s why we created HippoEngine. It can do inference without needing any extra libraries. The compiled model can be as small as 5MB, making the Docker image 100 times smaller.

But there was still a problem with model weights. Even though the runtime Docker image could be around 100MB, modern models themselves can be huge — over 5GB, even hundreds GB. Loading these weights takes a lot of time. We also started thinking about future hardware such as GraceHopper, and how to design AI infrastructure for it.

We came up with a new approach: breaking the traditional separation of GPU and CPU memory. We looked at it from a cache perspective — how to efficiently cache model weights and how this affects AI inference. We made sure that the weight representation is the same across all memory levels to maximize memory bandwidth usage.

With this new cache design, we found Small VM + Docker + K8s to be overcomplicated for the needs. A process-based runtime, without dependency issues, is more efficient. Our Multiple Layers Model Caching System not only solves the cold start problem but also opens up powerful edge use cases. Plus, using GPU memory as cache helps with scaling.

This caching strategy is versatile and can be implemented for both inter-model and intra-model scenarios. For instance, in the case of Mixture of Experts (MoE) models, each expert network can utilize the cache individually to reduce memory requirements.

Hosting all (112) CivitAI SDXL Models on a Single Server

Consider the example of running CivitAI SDXL model inference on Lambda’s standard 4U 8 x L40S Server. Each model is like a “filter” in an image generation application, offered to users for enhaced creative purposes. The challenge is to serve all 112 different SDXL models with minimal delay. We can’t have users waiting minutes to try out a filter, although that’s the norm. Our goal is to ensure that when a user selects a filter, the transition is almost instantaneous, eliminating the frustrating wait times often seen today.

After applying HippoML’s 8bit optimization, 112 models only need 362GB of memory, which can easily fit into Level 2 cache.

With this setup, we don’t need a complex Kubernetes system for resource management. A simple process-based request queue is enough. All CivitAI models are run on a single server, and the cache system handles scaling for the most used models. To activate an arbitrary model in the GPU runtime, it only takes ~450ms. If with GraceHopper system, it can be an extra 5X faster.

How powerful is this server? Using standard setting, each GPU can generate more than 38 images per minute, totaling over 437K images per day. At a standard rate of $0.01 per image, this server could potentially save $4.3k API bill per day ($1.56 million per year). If running at full capacity, it would take just 29 days to pay off the investment in this server.

When we expand this system to multiple servers, models are moved from Level 3 Cache (NVMe SSDs), to Level 2 Cache (DDR5 memory). This is based on the existing cache pattern. In the worst case, where we need to read the entire 362GB of data into DDR, it takes no more than 150 seconds. It’s still super fast and efficient.

This caching approach also fits perfectly with GPU instances in the cloud. The key difference here is that we use the full instance instead of breaking it down into smaller VMs. We prefer to keep things simple and efficient.

Empower Local AI: How HippoML’s PrivateCanvas works

HippoML’s PrivateCanvas is a powerful local AI image generation and editing app for Windows, Linux, MacOS, NVIDIA GPUs, and Apple GPUs.

We used the same Multiple Layer Model Caching System designed for datacenter GPU inference. This makes PrivateCanvas work well even on devices with less memory, like NVIDIA RTX 3060 GPUs with 8GB GPU memory & 16GB CPU memory, or Apple Macs with 16GB unified memory.

We checked how fast SDXL can be activated on different devices. On an RTX 4070, it is similar to the server use case, which is less than 500ms. On a M3 Max Macbook, the activation time is around 950ms. The cache loading time is even faster than the network latency of most hosting API providers.

Conclusion

We’ve shared our Multiple Layers Model Cache system and its applications in data centers and local AI. We think that the K8s + Docker + small VM approach isn’t efficient. As foundation AI models become more standardized, and the use cases shift to inference of the fine-turned foundation models, specialized systems like ours will be more beneficial than general-purpose systems.

If you are interested in HippoML’s technology, contact us at info@hippoml.com, or join our discord: https://discord.com/invite/ZP8weNy5yQ