FilterHN

Rust implementation of Mistral's Voxtral Mini 4B Realtime runs in your browser

142 points

by Curiositry

5 hours ago

| past

| 9 comments

| github.com

| HN

▲

HorizonXP

2 hours ago

[-]

If folks are interested, @antirez has opened a C implementation of Voxtral Mini 4B here: https://github.com/antirez/voxtral.c

I have my own fork here: https://github.com/HorizonXP/voxtral.c where I’m working on a CUDA implementation, plus some other niceties. It’s working quite well so far, but I haven’t got it to match Mistral AI’s API endpoint speed just yet.

▲

Ygg2

1 hour ago

[-]

There is also another Mistral implementation: https://github.com/EricLBuehler/mistral.rs Not sure what the difference is, but it seems to be just be overall better received.

▲

NitpickLawyer

42 minutes ago

[-]

mistral.rs is more like llama.cpp, it's a full inference library written in rust that supports a ton of models and many hardware architectures, not just mistral models.

▲

mentalgear

54 minutes ago

[-]

Kudos, this is were it's add: open-models running on-premise. Preferred by users and businesses. Glad Mistral's got that figured out.

▲

simonw

1 hour ago

[-]

I tried the demo and it looks like you have to click Mic, then record your audio, then click "Stop and transcribe" in order to see the result.

Is it possible to rig this up so it really is realtime, displaying the transcription within a second or two of the user saying something out loud?

The Hugging Face server-side demo at https://huggingface.co/spaces/mistralai/Voxtral-Mini-Realtim... manages that, but it's using a much larger (~8.5GB) server-side model running on GPUs.

▲

refulgentis

24 minutes ago

[-]

It's not fast enough to be realtime, though you could do a more advanced UI and a ring buffer and have it as you describe. (ex. I do this with Whisper in Flutter, and also inference GGUFs in llama.cpp via Dart)

This isn't even close to realtime on M4 Max. Whisper's ~realtime on any device post-2022 with an ONNX implementation. The extra inference cost isn't worth the WER decrease on consumer hardware, or at least, wouldn't be worth the time implementing.

▲

Jayakumark

3 hours ago

[-]

Awesome work, Would be good to have it work with handy.computer. Also are there plans to support streaming ?

▲

JLO64

2 hours ago

[-]

Just tried out Handy. This is much better and lightweight UI than the previous solutions I've tried out! I know it wasn't you intention, but thank you for the recommendation!

That said, I now agree with your original statement and really want Voxtral support...

▲

ChadNauseam

2 hours ago

[-]

Handy is awesome! and easy to fork. I highly recommend building it from source and submitting PRs if there are any features you want. The author is highly responsive and open to vibe-coded PRs as long as you do a good job. (Obviously you should read the code and stand by it before you submit a PR, but I just mean he doesn't flatly reject all AI code like some other projects do.) I submitted a PR recently to add an onboarding flow to Macs that just got merged, so now I'm hooked

▲

Retr0id

4 hours ago

[-]

hm, seems broken on my machine (Firefox, Asahi Linux, M1 Pro). I said hello into the mic, and it churned for a minute or so before giving me:

panorama panorama panorama panorama panorama panorama panorama panorama� molest rist moundothe exh� Invothe molest Yan artist�� Yan Yan Yan Yan Yanothe Yan Yan Yan Yan Yan Yan Yan

▲

jszymborski

3 hours ago

[-]

Man, I'd love to fine-tune this, but alas the huggingface implementation isn't out as far as I can tell.

▲

Nathanba

2 hours ago

[-]

I just tried it, I said "what's up buddy, hey hey stop" and it transcribed this for me: " وطبعا هاي هاي هاي ستوب" No, I'm not in any arabic or middle eastern country. The second test was better, it detected english.

▲

7moose

2 hours ago

[-]

fwiw, that is the right-ish transliteration into arabic. It just picked the wrong language to transcribe to lol

▲

refulgentis

2 hours ago

[-]

Notable this isn't even close to realtime. M4 Max.

▲

sergiotapia