I am amazed how quickly APL changed the way I think.
Also strongly recommend watching Aaron Hsu on youtube.
There is no better time to re-learn programming, try APL, Forth, LISP, z80 machine code, UXN TAL, just try new things.
"Men are born soft and supple; dead they are stiff and hard. Plants are born tender and pliant; dead, they are brittle and dry. Thus whoever is stiff and inflexible is a disciple of death. Whoever is soft and yielding is a disciple of life. The hard and stiff will be broken. The soft and supple will prevail."
[1]: https://github.com/jackdoe/gnu-apl-wasm source of the repl and learn playground and also how to compile gnu apl to wasm (vibecoded)
If anyone is curious how queries in this language look, you can see it here: https://github.com/ClickHouse/ClickBench/pull/939/changes#di...
The default CBQN "make o3" on x86-64 also results in it only using SSE2 (utilizing function multiversioning is on the ever-infinite TODO list, though somewhat-low on it considering it's strictly-unnecessary in any specific situation; there's also AVX-512 usage on a branch, but mostly only AVX2 on mainline; and no arm SVE)
That all said, CBQN doesn't currently do any loop fusion, so being significantly-slower for sequences of operations over larger-than-cache arrays would kinda just be expected. BQN also just isn't particularly intended for database work anyway.
(didn't look much at the specific query impls, though "Pair" in utils.bqn is at least an overlong version of "Pair ← ⋈¨"; and some if not all of those Pairs would be better as "≍˘" to avoid nested arrays and ensuing pointer chasing; and, of course, if some of the columns are bools/int8/int16/int32, it'd be beneficial to store & load them as such instead of float64)
"Site by Claude. Runtime by hand"
I think that's completely acceptable.
We are used to this, hence why folks here point it out.
https://github.com/l-labs/master-benchmark
Which can also be run with minimal modifications on ngn and others.
For database the h2o.ai for duckDB really is a solid hard bench IMHO:
https://github.com/l-labs/rust_ipc/blob/master/tests/ipc_tes...
Which is ~6k tests (Rust is actually what I use to drive release testing)
Unknown quality is the absence of information, you can’t have a signal for it.
If you are using “unknown” as a euphemism for “poor”, which is the most obvious thing that makes the signal meaningful in the direction that seems apparent from context, why?.
That's also why "closed-source" was mentioned in the same sentence. Two missing signals.
That's also why the author keeps pointing to the benchmarks and test suite, which is the signal they'd like us to judge by. Seems entirely reasonable to me!
Typically if a site is well designed, that used to be a signal that a lot of resources had been invested into the project. Now that signal is rather noisy.
Here, is particularly bad, as the main page is just complete slop bullet points. What does 'end-to-end column compression' mean? or 'uniform bytes at every layer'?
It's difficult to understand at a glance now, if this project is going to be abandonware in a week or if someone actually spending time on this. If the creator can't be bothered to read it, why should I?
Personally, I'm also very weary of Claude design-isms. Once you learn them, you see them everywhere.
That said the mail group has more details and the blog posts are effectively internal posts turned into shorter (less code) versions. The binary itself is years of effort - with small improvements over many years and shaving a few kb each time ... 'built the old way' like amish furniture :)
[1] https://web.archive.org/web/20160306154854/http://kx.com/q/d...
Columnar databases are array languages, after all.
Seems wild that the much-lauded "reference engine" leaves ~2x perf improvements on the table.
https://github.com/l-labs/master-benchmark
Or on the database side of the house try the h2o.ai base that duckdb excels at
> But where Claude was especially helpful (other than the web design w/ Claude Design for the prototype) was analyzing ASM output of functions and optimizing those E.g. lots of small mistakes would have been missed by not having restrict/const in places.
Second, in my personal experience, most quants refer to the platform as "kdb+", which uses the languages q (directly), k (indirectly), and qSQL. Is there a reason why the website landing page does not mention kdb+? I assume this is intentional.
L used the two open ones that are easy to replicate: H2O.ai (great bench) https://github.com/l-labs/db-benchmark TSBS (less great but useful) https://github.com/l-labs/tsbs
If there are others (will do ClickBench) they'll go there as well
primitive speedup
sum price 7.7x
avg price 33x
//100k random 32b ints ...
l>v:100000?255
l>v
196 124 18 216 63 169 151 173 126 99 90 133 92 158 217 169 201 191 138 105 13..
// but actually they are 1/4 the size e.g. int8
l>-17!`v
1b // is compressed?
100032j // compressed bytes
400000j // original bytesI don't necessarily like that it lacks a dark mode, but ultimately there's very little to complain about here. It accomplishes what many other websites only wish they could.
Memory got faster, no?
the early versions of L (circa ... 2011/2012) couldn't really deliver any substantial performance improvement over what is out there (BQN/ngn/Kona). The memory bandwidth was the limit - I simply couldn't keep the cores active. ~2018/2019 I started from scratch with arrays (vectors) using compression by default. That was a massive unlock - and I could not keep most of the CPU doing actual compute! Then it was years of working on compression native operations - some of which were obvious[3] like sum/reductions ... others not so much!
[1] https://www.emergentmind.com/topics/memory-wall [2] https://www.cse.iitd.ac.in/~rijurekha/col216/quantitative_ap... [3] https://lv1.sh/blog/compute-on-compressed/
Did you mean APL?
x@&/x?/:/:(+;!9;/:x)https://en.wikipedia.org/wiki/Array_programming
EDIT: Did someone downvoted this because it is wrong or is someone I am arguing with rage-downvoted this ?
Extremely likely to be AI, though I’m not sure that matters for rules re: submissions
The language and its ideas come from Ken Iverson's famous paper "Notation as a Tool of Thought". This is a common understanding for people in the APL, K/J/Q etc array language communities, who are likely this website and product's intended audience.
That much makes sense in context: K is an array language, like APL, J, etc. From what little experience I have with J, ‘vector as a unit of thought’ seems like a reasonable description.
> Extremely likely to be AI
I had the same thought though.
For moderators, I’d suggest that for the community, it’s spam. That’s one example, there are many more like it. The individual statement may be defensible but is still pseudoscientific language. This sort of content is a massive burden to community. Unanswerable anrguments about AI writing, whether the ability for an individual to have a parse-able reading is the same as writing being parable. The net effect is negative experiences for many and copy-editing for someone who did not do copy editing.
I’ve had similar ideas in the past: clearly simd is the way to get the most out of your cpu. Can we design a language where all operations are automatically simd, and it takes effort to do anything in scalars?
And I guess these array languages are what you might get.
It’s not ‘unit vector of thought,’ btw, which is weirder than what it says.