This is a classic bikeshedding issue. When Go and Rust were first being designed, I brought up support for multidimensional arrays. For both cases, that became lost in discussions over what features arrays should have. Subarrays, like slices but multidimensional? That's the main use case for striding. Flattening in more than one dimension? And some people want sparse arrays. Stuff like that. So the problem gets pushed off onto collection classes, there's no standard, and everybody using such arrays spends time doing format conversion. This is why FORTRAN and Matlab still have strong positions in number-crunching.
(Of course Python provides wrappers to Fortran code, eg FITPACK, but this code specifically was mostly written in the 80s, with small updates in this century, and is probably used more thru the numpy wrappers, stride conversion issues and all, than thru its native Fortran interface)
Fortran at least used to be very common in heavy scientific computing, but I would bet that relies on GPUs or other accelerators these days.
That is more of an urban legend than reality in 2024. Fact is, although the original BLAS implementation was fortran, it has been at least a decade since every Linux distribution ships either OpenBLAS or ATLAS or the MKL, which are both written in a mix of C and assembly. All modern processor support is only available in these implementations.
LAPACK itself is still often built from the Fortran code base, but it's just glue over BLAS, and it gets performance solely from its underlying BLAS. Fortran doesn't bring any value to LAPACK, it's just that rewriting millions of algebraic tricks and heuristics for no performance gain is not enticing.
Is Fortran legitimately faster than C with the restrict keyword? Regardless of the function call cost diffs between the two - meaning, even if we assume Fortran is somehow faster here - there's no way numeric production code does enough function calls for this to matter. If Fortran was faster than C at any point in time I can only imagine pointer aliasing issues to have been the culprit and I can't imagine it still being relevant, but I am of course ready to stand corrected.
Emphasis on modern because maintaining 70s-80s numeric mathematician cowboy code is a ninth circle of hell. It has a bad rep for that reason.
They did not even have while-loops in the language until Fortran 77. While-loop functionality used to be implemented using DO-loops mixed with GOTOs. You can't fault the people of that era for using the only tools that were available to them.
I thought blas/lapack is still written in Fortran, so most numerical code would still be built on top of Fortran.
Fortran still dominates certain areas of scientific computing / HPC, primarily computational chemistry and CFD. - https://fortran-lang.org/packages/scientific - you don't hear about most of them because they're generally run on HPC centers by scientists in niche fields. But you do get their benefit if you buy things that have the chemical sector in their supply chain.
The common thread is generally historical codes with a lot of matrix math. Fortran has some pretty great syntax for arrays and their operations. And the for-loop parallelization syntax in parallel compilers (like openmp) is also easy to use. The language can even enforce function purity for you, which removes some of the foot guns from parallel code that you get in other languages.
The kinds of problems those packages solve tend to bottleneck at matrix math, so it's not surprising a language that is very ergonomic for vector math found use there.
Same for Matlab, it's mostly used by niche fields and engineers who work on physical objects (chemical, mechanical, etc). Their marketing strategy is to give discounts to universities to encourage classes that use them. Like Fortran, it has good syntax for matrix operations. Plus it has a legitimately strong standard library. Great for students who aren't programmers and who don't want to be programmers. They then only know this language and ask their future employer for a license. If you don't interact with a lot of engineers at many companies, you aren't going to see Matlab.
Just the other day, the solar physicist I work with said “yeah, that code that runs on the supercomputer needs to be rewritten in Fortran” (from C, I think.
He’s nearing retirement, but it’s not that he’s behind the times. He knows his stuff, and has a lot of software experience in addition to physics
This surprises people who just write software, but consider:
- The documentation and support is superb. This alone can justify the cost for many orgs.
- Did I mention the support? MathWorks has teams of application support engineers to help you use the tool. The ability to pay to get someone on the phone can also justify the price.
- The toolkits tend to do what specific fields want, and they tend to have a decent api. In contrast, you might end up hacking together something out of scipy and 3-4 weirdly incompatible data science packages. That's fine for me, but your average ME/EE would rather just have a tool that does what they need.
- It also has SIMULINK, which can help engineers get from spec to simulation very quickly, and seems deeply embedded in certain areas (eg: it's pretty common for a job ad with control systems work to want SIMULINK experience).
Is Python gradually displacing it? Probably.
(Honestly, I wish it would happen faster. I've written one large program in Matlab, and I have absolutely no desire to write another.)
[0]: https://docs.julialang.org/en/v1/manual/interfaces/#man-inte...
I'd suggest Python (via the various memoryview related PEPs, plus numpy) does provide most of the required information (vs. e.g. Go or Rust or Ruby), so I'm not sure that proving much other than if the value of using a library to handle multidimensional arrays is enough such that the combined value of a more general language plus library beats Matlab or Fortran (likely due to other ecosystem or licensing effects), people will switch.
Nit: this failed to be respective; the years are the other way around
But that was a great nit to find.
As the article mentions, NumPy can handle both and do all the bookkeeping. So can Eigen in C++.
[1] https://www.math.utah.edu/software/lapack/lapack-blas/dgemm....
(The article is basically about not having to give up in some cases where you can tweak the input parameters and make things work without rewriting the function; but it's not always possible)
Because that's the kind of mental model capacity scientific programmers needed to have back in the days. People still do similarly brain-twisting things today. In this context, converting your logic between C and Fortran to avoid shuffling data is trivial.
Namely:
1. Should the "default" (syntactically-sugar'd) arrays in the language (e.g. `int[]`) be restricted to simple contiguous storage, or should they also hold (potentially negative) stride, etc?
2. Should I have multidimensional arrays? (to be clear, I'm talking about contiguous blocks of data, not arrays-holding-pointers-to-arrays)
3. Should slicing result in a view, or a copy? (naturally, some of these options conflict)
I'm honestly erring towards:
1. "fat" arrays;
2. multidimensional arrays;
3. slicing being views.
This allows operations like reversing, "skipping slices" (e.g. `foo[::2]` in Python), even broadcasts (`stride=0`) to be represented by the same datastructure. It's ostensibly even possible to transpose matrices and such, all without actual copying of the data.
But there are good arguments for the reverse, too. E.g. "fat" arrays need 1 more `size_t` of storage per dimension (to store the stride), compared to "thin" arrays. And a compiler can optimize thin ones much better due to more guarantees (for example, a copy can use vector instructions instead of having to check stride first). Plus "thin" arrays are definitely the more common scenario and it simplifies the language.
So, yeah, still not quite sure despite liking the concept. One obvious option would be for `int[]` to be a thin array, but offering `DataView<int>` [1]. The problem with that is that people implementing APIs will default to the easy thing (`int[]`) instead of the more-appropriate-but-more-letters `DataView<int>`. Ah, dilemmas.
[1] (not actual syntax, but I figured I'd use familiar syntax to get the point across)
Or... you could have sent a ~25 line pull request to opencv to fix this performance problem not just for you, but for thousands of other developers and millions of users.
I think your fix would go here:
https://github.com/opencv/opencv/blob/ba65d2eb0d83e6c9567a25...
And you could have tracked down that slow code easily by running your slow code in gdb, hitting Ctrl+C while it's doing the slow thing, and then "bt" to get a stack trace of what it's doing and you'd see it constructing this new image copy because the format isn't correct.
And what about the other examples I measured, including pure numpy code? More patches to submit?
Or just special case resize, since there the channel order doesn't matter. If you interpret R as A you'll still get the right answer.
If you really want a solution which is a patch rather than a function in your own code, I guess the way to go would be to persuade the maintainers of pygame and separately, the maintainers of its fork, pygame-ce, to add a pixels4d_maybe_rgba_and_maybe_bgra() function returning a WxHx4 array. Whether they would want to add something like that I don't know. But in any case, I think that since closed source projects, open source projects, and code by sister teams exist that will not accept patches, or will not accept them in a timely manner, it's interesting what you can do without changing someone else's code.
i2 = np.ascontiguousarray(pg.surfarray.pixels3d(isurf))
Which does the 100x speedup too and is a "safe" way to adjust memory access to numpy strides.
Whether the output is correct or not is left as an exercise to the author, since the provided benchmark only use np.zeros() it's kind of hard to verify
TFA links to a GitHub repo with code resizing non-zero images, which you could use both to easily benchmark your suggested change, and to check whether the output is still correct.
Is it because in much of the maths domain it turns out you manipulate columns more often than rows?
(not a mathematician but I could believe if the columns represent "dimensions" or "qualities" of some dataset, and you want to apply a function to a given dimension, having data in column-natural order makes that faster.)
Obviously you want to believe naievely there is no difference to the X and Y in the X,Y plane, but machines are not naieve and sometimes there IS a difference to doing things to the set of verticals, and the set of horizontals.
So, x and y must be columns. So, it is nice if our language has column-order as the default, that way a vector is also just a good old 1d array.
If we did y=xA, x and y would be rows, the actual math would be the same but… I dunno, isn’t it just hideous? Ax is like a sentence, it should start with an upper case!
It also fits well with how we tend to learn math. First we learn about functions, f(x). Then we learn that a function can be an operator. And a matrix can represent a linear operator. So, we tend to have a long history of stuff sitting to the left of x getting ready to transform it.
For the row-major layout the multiplication Ax ends up being more cache-efficient than for the column major layout, as for calculating each component of y you need to scan x and a row of A.
I don't know if tiling is done for matvec. I don't think it makes sense there, but I didn't think about it too hard.
Is how you'd write it in NumPy and most (/all?) deep learning libraries, with x.shape=[batch, ndim]
I'd personally prefer that to be the convention, for consistency.
Also, numpy is built on some fortran libraries, like BLAS and LAPACK that assume a column major order. If it took row major input, it would need to transpose matrices in some cases change the order, or rewrite those libraries to use row major form.
Actually let me include a little figure
original reshape swapaxes
abc ab ad
def cd be
ef cf
On a different note, I'm surprised no one has mentioned Cython in this context. This is distinct from, but broadly related to things like @cython.boundscheck(False). Cython is _really_ handy, and it's a shame that it's kinda fallen out of favor in some ways lately.
What does Cython do better than numbs except static compilation? Honest q, I know little about both
By "numbs" here, I'm assuming you mean numba. If that's not correct, I apologize in advance!
There are several things where Cython is a better fit than numba (and to be fair, several cases where the opposite is true). There are two that stick out in my mind:
First off, the optimizations for cython are explicit and a matter of how you implement the actual code. It's a separate language (technically a superset of python). There's no "magic". That's often a significant advantage in and of itself. Personally, I often find larger speedups with Cython, but then again, I'm more familiar with it, and understand what settings to turn on/off in different situations. Numba is much more of a black box given than it's a JIT compiler. With that said, it can also do things that Cython can't _because_ it's a JIT compiler.
If you want to run python code as-is, then yeah, numba is the better choice. You won't see a speedup at all with Cython unless you change the way you've written things.
The second key thing is one that's likely the most overlooked. Cython is arguably the best way to write a python wrapper around C code where you want to expose things as numpy arrays. It takes a _huge_ amount of the boilerplate out. That alone makes it worth learning, i.m.o.
That's not strictly possible, but "circular" references are with "stride tricks". Those can accomplish similar things in some circumstances. But with that said, I don't think that would work in this case.
well, half of why. for some reason i keep doing everything directly on /dev/fb0
Is there a better way to do it?
Theres the announcement where they open sourced the modules in their standard lib: https://www.modular.com/blog/the-next-big-step-in-mojo-open-...
There's the standard lib: https://github.com/modularml/mojo/blob/nightly/stdlib/README...
I suspect when you say "unsafe by design" you may be referring to the dynamic type checking aspect of Python, although Python has supported for a while type annotations, that can be statically checked by linter-like tools like PyRight.
--
1: https://en.wikipedia.org/wiki/Type_safety
2: https://en.wikipedia.org/wiki/Memory_safety
But you can perform unsafe operations e.g.,
import ctypes
ctypes.string_at(0) # access 0 address in memory -> segfault
Python doesn't have an unsafe keyword like Rustlang.
In Python, you can dereference a null pointer and cause a Segmentation fault given a ctypes import.
lancedb/lance and/or pandas' dtype_backend="pyarrow" might work with pygame
eval((lambda:0).__code__.replace(co_consts=()))
cpython is not memory safe.Containers are considered nearly sufficient to sandbox Python, which cannot be effectively sandboxed using Python itself. Isn't that actually true for all languages though?
There's a RustPython, but it does support CFFI and __builtins__.eval, so
def f(): pass
f.__code__ = f.__code__.replace(co_consts=())
f()
Have a look at the linked article to see what meaning of 'unsafe' the author has in mind.
Fun lang though, just not one that ever comes to mind when I hear 'safe'.
"Safety" is an overloaded term, but that happens a lot in software. You'll probably get best results if you try to understand what people are talking about, rather than just assuming everyone else is an idiot.
The generic snide about Python that has nothing to do with the article ain't all that informative.
You could make the same remark you just made about all Python being unsafe about Rust: 'Isn't all Rust, by design, unsafe?'. And compared to eg Agda or Idris that would be true, but it wouldn't be a very useful nor interesting comment when talking about specifically 'unsafe' Rust vs normal Rust.
Fine and good to advocate rigor, whether or not it’s specifically relevant to this post, but maybe be careful with the interpretation and commentary lest people decide the misconception is on your part?
So it doesn't really make sense to pretend there's some single meaning of 'safe' vs 'unsafe' that's appropriate in all contexts.
Python is a great language, but just as the generation of kids who got out of computer science programs in the 2000s were clueless about anything that wasn't Java, it seems this generation is clueless about anything that isn't Python.
There was life before Python and there will be life after Python!
You could also benefit from testing blocks of code with safety enabled to have more confidence when safety is removed.
I tend to prototype in Python and then just rewrite the whole thing in C with classes if I need the 10000x speedup.
It was because of white spaces not lining up. I was like, really? Using white spaces as a way to denote the body of a function? Really?
- Amiga E - Assembler (on Amiga) - perl - tcl/tk - javascript - vbscript - java - vba - c - c++ - python
rock. Granted at some of those rocks, I just took a quick peek (more like a glance), but a rock with whitespaces being important was, at that moment, new to me!
I just meant that for someone coming from a language(s) where different kinds of delimiters are used to denote function bodies, a language that puts such a huge emphasis on whitespace was kinda throwing me off to get an error message slapped at me even though I was replicating the example letter by letter (although, obviously, I missed the importance of whitespace).
Even Raymond's article which was one of the first whined about it [1]. I definitely remember reading that one and thinking I have to check out that language later.
Back then it might have been java, then python, till the next language comes around.
The java thing was so apparent, that when you looked at c++ code, you were able to spot the former java user pretty easily. :)
About python, sometime ago, I was looking into arudino programming in c and found someone offering a library (I think) in c (or c++, can't remember). What i remember is, that this person stopped working on it, because they got many requests and questions regarding to be able to use their library in python.
Well, Java was designed based on C++ GUI development patterns typical in the 10 years of C++ GUI frameworks that predated Java.
Yet, people keep using these kind of remarks.
>1 billion people exist. Each has a unique opinion / viewpoint.
C and C++ 'type safety' is barely there (compared to more sane languages like OCaml or Rust or even Java etc). As to why would anyone do that? The question is 'why not?' It's fun.