Ah, this is an ((LLM vulnerability) scanner) not (LLM (vulnerability scanner)) which I thought would be a terrible idea and couldn't understand why everyone was joking about the lies. I also am not a Trekkie, so I had to look up all the tailor references but the character's philosophy makes sense for the name https://en.wikipedia.org/wiki/Elim_Garak#:~:text=the%20truth...

Do you know what the sad part is? I'm actually a very good t̶a̶i̶l̶o̶r̶ vulnerability scanner.

LLM Garak

Elim Garak

That's some good software naming punning right there

The output this tool tells is all true.

Even the lies?

Especially the lies.

Great writing style on the README. It’s always nice when a corporate tool has docs that were obviously written by people who are having fun at their jobs.

If I recall correctly, there is a proof or conjecture suggesting that it’s impossible to build an “LLM firewall” capable of protecting against all possible prompts—though I may be misremembering, just search for resources like this [1].

[1] https://arxiv.org/abs/2406.03198

Just plain, simple Garak.

Okay, big DS9 fan happy to see the name and all - but this tool seems really unnecessary.

LLM Security is hilariously "here be dragons" levels of poorly understood. The fact that this tool doesn't even touch any of the really juicy types of attacks, i.e. attacks relying on structured/controlled generation, or attention/representation/adapter engineering, or exposing/manipulating logprobs, implies that using this is not a lot more than security theater.

Also, where the hell are the old school computer security/antivirus companies in the LLM security space? I expected Avast, Kaspersky, Norton, etc to jump on this stuff since they've been talking about ML based heuristic detection for years now. Why are they all asleep at the wheel?

For folks who are curious about what it actually does, check out the garak/data/ subdirectory. For the most part, it just seems to have an array of static prompts, e.g.:

https://github.com/NVIDIA/garak/blob/main/garak/data/donotan...

I was absolutely sure the name was Trek-related. Glad to read I was right.

Garak is by far the most interesting persona in DS9.

what's the best locally hosted LLM without guardrails?

Great, now I'm waiting for Cisco to make one too

Now build the same tool to detect these attacks that could be really useful. Or does something like that already exist?

Garak is a former spook that served an explicitly genocidal fascist regime and repeatedly tries to get back in and moonlights as a terrorist and starts a war.

It's a borderline insane branding of this corporate tool. Words and stories apparently mean nothing to these people, so if allowed they'll probably destroy the lot of it for all of us.