Amazing!

How was your experience using Rust on this project? I'm considering a project in an adjacent space and I'm trying to decide between Rust, C, and Zig. Rust seems a bit burdensome with its complexity compared to C and Zig. Reminds me of C++ in its complexity (although not as bad). I find it difficult to walk through and understand a complicated Rust repository. I don't have that problem with C and Zig for the most part.

But I'm wondering if I just need to invest more time in Rust. How was your learning curve with the language?

Hoping the author can answer, I'm still learning about how this all works. My understanding is that inference is "using the model" so to speak. How is this faster than established inference engines specifically on Mac? Are models generic enough that if you build e.g. an inference engine focused on AMD GPUs or even Intel GPUs, would they achieve reasonable performance? I always assumed because Nvidia is king of AI that you had to suck it up, or is it just that most inference engines being used are married to Nvidia?

I would love to understand how universal these models can become.

We also wrote our inference engine in rust for mixlayer, happy to answer any questions from those trying to do the same.

Looks like this uses ndarray and mpsgraph (which I did not know about!), we opted to use candle instead.

I just spun up a AWS EC2 g6.xlarge instance to do some llm work. The GPU is NVIDIA L4 24GB and costs $0.8048/per hour. Starting to think about switching to an Apple mac2-m2.metal instance for $0.878/ per hour. Big question is the Mac instance only has 24GB of unified memory.

Can you explain the type of quantization you support?

would https://docs.unsloth.ai/basics/kimi-k2-how-to-run-locally be faster with mirai?

Have you added it to HomeBrew and other package managers yet?

Also any app deployed to PROD but developed on Mac need to be consistent i.e. work on Linux/in container.

How does this compare to https://github.com/Anemll/Anemll?

In practice, how often do the models use the ANE? It sounds like you are optimizing for speed which in my experience always favors GPU.

>faster than llama cpp in all of the use cases

What's your deliberate, well-thought roadmap for achieving adoption similar to llama cpp?

"trymirai", every time I hear the word Mirai I think of the large IOT DDoS botnet. Maybe it's just me though.

How does this bench compared to MLX?

I'm curious about why the performance gains mentioned were so substantial for Qwen vs Llama?

Wow! Sounds super interesting

What are the units on the benchmark results? I’m guessing higher is better?

Somewhat faster on small models. Requires new format.

Not sure what the goal is for this project? Not seeing how this presents adequate benefits to get adopted by the community

just curios, will it be supported on iOS, it would be great to build local llm app with this project.

that’s exactly we are looking for not to waste on apis. Wonder how significant trade offs are

cooollll

nice

Wondering why use Rust other than C++