Turning a pile of documents into a searchable useable knowledge base
86 points
6 hours ago
| 7 comments
| github.com
| HN
linuxrebe1
6 hours ago
[-]
I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted: 1) To keep my data local 2) be able to filter out PII and other data 3) Be able to find and delete duplicates 4) Get short synopsis of what a document is 5) Semantic and keyword search 6) All of this kept local to me requiring no internet access and no tokens spent to train someone elses AI.

The result I call DocuBrowser and in it's current form is FOSS (GPL-3) licensed for your personal use. The UI is in your browser. The AI models used are held local and are tiny, Available for Linux(RPM,Deb, and tgz) Windows and Mac. Let me know what you think and thanks for taking the time to try it out.

reply
seb1204
3 hours ago
[-]
Sounds similar to https://docs.paperless-ngx.com/

Key difference I see is that you point it to a folder instead of uploading to a system.

reply
vsviridov
44 minutes ago
[-]
I think paperless devs are working on AI integration, and there are 3rd party solutions. I'm holding out for an official one, so far.

It's pretty cool, I've set up a share where the scanner scans, and it automatically picks it up from there and ingests it into the system.

reply
bobim
5 hours ago
[-]
Could it be extended so it also extracts pictures from pptx and xlsx and run vision to get a description to be added to the text content before indexing?
reply
linuxrebe1
3 hours ago
[-]
Let me look into this
reply
clif_mcIrvin
1 hour ago
[-]
How about jpegs or other scanner images files? We have hundreds of scanned documents that were never pdf wrapped.
reply
password4321
1 hour ago
[-]
Personal use? I need this at work, dragging useful info from tarpits like Teams and GitLab.

Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).

reply
asciimoo
5 hours ago
[-]
We need projects like this. Automatically classifying the files is smart.

I'm working on a similar application called Hister (https://github.com/asciimoo/hister). I should borrow some of your ideas. =]

reply
jphorism
2 hours ago
[-]
Nice, what are you hoping to accomplish with this project?
reply
passwordoops
52 minutes ago
[-]
Care to elaborate?
reply
NamlchakKhandro
1 hour ago
[-]
A resume
reply
NKosmatos
4 hours ago
[-]
Looks good, definitely going to try it. Extra thanks for creating something fully local, we need more projects like this one!
reply
aucisson_masque
5 hours ago
[-]
I'm a huge fan of recall, going to test this out. This looks very interesting.
reply
drizzler
2 hours ago
[-]
I just installed this and, after a few hiccups, got it up and running on my Ubuntu system. Works great, looks great. Thank you for this. Half of my documents are OpenDocument format. Is there any chance you'll be supporting ODF in the future?
reply
toomuchtodo
4 hours ago
[-]
How do you feel about supporting an S3 compatible target as a feature request?
reply