Show HN: Nexa SDK – Build powerful and efficient AI apps on edge devices
github.comHey HN! Alex and Zack here from Nexa AI. We're excited to share something we've been working on.
Our journey began with the Octopus series --- action models for mobile AI agents (https://huggingface.co/NexaAIDev/Octopus-v2). We focused on making sub-billion parameter models excel at function calling, making high accurate and fast function-calling possible on mobile and edge devices. But as we delved into developing full-fledged on-device applications, we hit a roadblock.
We realized that optimizing for function calling (tool-use) alone wasn't enough. Building powerful on-device AI apps requires a diverse set of tools: language models with domain expertise, speech processing, image generation, embedding models and more. That's when we decided to create Nexa SDK --- a comprehensive toolkit that brings together everything developers need to build powerful and efficient AI applications that run entirely on-device.
Here's what Nexa SDK offers:
- Support for both ONNX and GGML models.
- An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements.
- An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models.
- An OpenAI-compatible API server with optimization in function calling.
- A Streamlit UI for rapid prototyping.
- An intuitive CLI for easy model management.
- Backend optimizations for latency and power consumption on edge devices.
To showcase its capabilities, we've built several demo apps running entirely on your device (https://github.com/NexaAI/nexa-sdk/tree/main/examples):
- AI soulmate with uncensored model and audio-in/audio-out interaction.
- A quick interface for uploading and chatting with PDFs like your personal finance documents.
- A meeting transcription app supporting multiple languages and real-time translation.
But we're just getting started! There are lots of exciting developments in our pipeline, and we can't wait to share them with you soon!
Check it out: (https://github.com/NexaAI/nexa-sdk)
Docs: (https://docs.nexaai.com/)
If you're excited about the future of on-device AI, we'd really appreciate your support. A star on our GitHub repo goes a long way in helping us reach more developers!
Cheers,
Alex & Zack Super excited, cannot wait to try it! Thank you! Any specific on-device app idea in mind? Any project demos? Here's some demos we built with the Nexa SDK: (https://github.com/NexaAI/nexa-sdk/tree/main/examples) We're also planning to have an app gallery where developers can upload their on-device AI apps built with the Nexa SDK. We'll aim to make it as out-of-the-box as possible for others to download and run on their devices.