Our journey began with the Octopus series --- action models for mobile AI agents (https://huggingface.co/NexaAIDev/Octopus-v2). We focused on making sub-billion parameter models excel at function calling, making high accurate and fast function-calling possible on mobile and edge devices. But as we delved into developing full-fledged on-device applications, we hit a roadblock.

We realized that optimizing for function calling (tool-use) alone wasn't enough. Building powerful on-device AI apps requires a diverse set of tools: language models with domain expertise, speech processing, image generation, embedding models and more. That's when we decided to create Nexa SDK --- a comprehensive toolkit that brings together everything developers need to build powerful and efficient AI applications that run entirely on-device.

Here's what Nexa SDK offers:

  - Support for both ONNX and GGML models.
  - An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements.
  - An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models. 
  - An OpenAI-compatible API server with optimization in function calling.
  - A Streamlit UI for rapid prototyping.
  - An intuitive CLI for easy model management.
  - Backend optimizations for latency and power consumption on edge devices.

To showcase its capabilities, we've built several demo apps running entirely on your device (https://github.com/NexaAI/nexa-sdk/tree/main/examples):

  - AI soulmate with uncensored model and audio-in/audio-out interaction.
  - A quick interface for uploading and chatting with PDFs like your personal finance documents.
  - A meeting transcription app supporting multiple languages and real-time translation.

But we're just getting started! There are lots of exciting developments in our pipeline, and we can't wait to share them with you soon!

Check it out: (https://github.com/NexaAI/nexa-sdk)

Docs: (https://docs.nexaai.com/)

If you're excited about the future of on-device AI, we'd really appreciate your support. A star on our GitHub repo goes a long way in helping us reach more developers!

Cheers,

Alex & Zack