If you're interested in the idea of writing a database, I recommend you checkout https://github.com/thomasjungblut/go-sstables which includes sstables, a skiplist, a recordio format and other database building blocks like a write-ahead log.
Also https://github.com/BurntSushi/fst which has a great Blog post explaining it's compression (and been ported to Go) which is really helpful for autocomplete/typeahead when recommending searches to users or doing spelling correction for search inputs.
>I really liked the README, that was a good use of AI.
Human intelligences, please start saying:
(A)I wrote a $something in $language.
Give credit where is due. AIs have feelings too.
Ohh boi, that’s exactly how the movie "Her" started! XD
We are soon entering into the territory of "no one cares if you did it, but can you say something interesting?". I created X software is soon leaving the ranks of cool stuff.
When you think OP vibe-coded the project but can’t prove it yet
Dexter's memes have been popping up recently and I am loving them
I don't know who bay harbor butcher is though :sob: but I don't want spoilers, I will watch it completely some day
My friend says that he watched complete dexter just via clips lol.
Should a multi-part blog would've been better?
I mean, no biggie and I don't understand the reaction but hey, I'm old so...
Is vibe-commented a thing yet? :D
Wanted to give fellow readers a good on-ramp for understanding the FTS internals. Figured leaning into readability wouldn’t hurt
For me this makes the structure super easy to grok at a glance
https://github.com/wizenheimer/blaze/blob/27d6f9b3cd228f5865...
That said, totally fair read on the comments. Curious if they helped/landed the way I intended. or if a multi-part blog series would’ve worked better :)
I'm wondering is it really worth dumping a general knowledge articles into code comments? To me it feels like the wrong place. Would just the wikipedia link be enough here?
I also notice a lot of comments like this
// IsEnd checks if this is the EOF sentinel
//
// Example usage:
//
// if pos.IsEnd() {
// // We've reached the end, stop searching
// }
func (p *Position) IsEnd() bool {
return p.Offset == EOF
}
This is one of the typical issues when AI creates "code comments", because it always describes "What" is happening. A good comment should answer the question "Why" instead.
For the linked skip list module, a good comment could say why skip list was chosen over b-tree or other data structure and which trade offs were made. AI will never know that.
If someone has ever written any code is well aware of what can be done in a weekend and especially that no one doing something "in a weekend" will ever add all those useless comments everywhere, literally a few thousand lines of comments. That takes more time than writing code. Comments in Claude style. Other claude-isms all around.
It's ok to vibe things, but just say so, no shame.
And yes, after 5 minutes of looking around I had enough evidence to "prove it". Any moderately competent engineer could.
What was the motivation to kick this project off? Learning or are you using it somehow?
It ended up being a clean, reusable component, so I decided to carve it out into a standalone project
The README is mostly notes from my Notion pages, glad you found it interesting!
Could you explain more why you avoided parsing strings to build queries? Strings as queries are pretty standard for search engines. Yes, strings require you to write an interpreter/parser, but the power in many search engines comes from being able to create a query language to handle really complicated and specific queries.
If you notice there are vv obvious optimizations we could make. I’m planning to collect them and put a these as code challenges for readers, and building string-based queries would make a great one :)
I see you are using a positional index rather than doing bi-word matching to support positional queries.
Positional indexes can be a lot larger than non-positional. What is the ratio of the size of all documents to the size of the positional inverted index?
For example the phrase query "United States of America" doesn't occur in the document "The United States is named after states of the North American continent. The capital of America is Washington DC". But "United States", "states of" and "of America" all appear in it.
There's a tradeoff because we still have to fetch the full document text (or some positional structure) for the filtered-down candidate documents containing all of the bi-word pairs. So it requires a second stage of disk I/O. But as I understand most practitioners assume you can get away with less IOPS vs positional index since that info only has to fetched for a much smaller filtered-down candidate set rather than for the whole posting list.
But that's why I was curious about the storage ratio of your positional index.
One’s for browsing HN at work, the other’s for home, and the third one has a username I'm not too fond of.
I’ll stick to this one :) I might have some karma on the older ones, but honestly, HN is just as fun from everywhere
I intended this to be an easy on-ramp for folks who want to get a feel for how FTS engines work under the hood :)
I appreciate the technical depth of the readme, but I’m not sure it fits your easy on-ramp framing.
Keep going and keep sharing.