Is there a good book that is focused on advanced OTP? On the internals of the BEAM and the hairier parts of the Erlang ecosystem? I can only find basic to intermediate introductory material. Perhaps I should just spend time reading the erlang website back-to-front...
---
1: I'm taking a small break from writing a parallel, distributed website crawler with homegrown HTTP client. Quite fun, very fast, to replace my first, rough prototype written in Rust. The hard part is accepting that most HTTP servers on the open Internet are hopelessly broken in a million ways.
20 years ago, picking up OCaml, as an initial project I decided to write a password cracker to run against my WiFi router. The webserver on my WRT-54G was sending back errors, but Yahoo's webserver (more popular than Google at the time) was perfectly happy replying correctly to my HTTP client's requests.
I had to break out Ethereal (since renamed Wireshark) to debug why Yahoo was fine with my HTTP client but my WRT-54G wasn't. It turns out I didn't realize that OCaml character escapes that began with zero were still decimal, not octal (unlike C). So, I was sending a line terminator of vertical-tab shift-in instead of carriage-return line-feed. I guess kudos to whomever wrote Yahoo's webserver to make proper sense of such wonky requests, though I think it was far too liberal in what it accepted.
Depending on exactly when, you may have been hitting Yahoo's custom http server originally written by David Filo or yapache, a patched version of Apache 1.3.
1) The BEAM book - https://blog.stenmans.org/theBeamBook/
2) Erlang and OTP in Action by Merritt, Logan and Carlson
* In Erlang/OTP 27 we're getting triple-quoted strings, so this patch adds a warning for "accidental" triple-quoted strings.
* In Erlang/OTP 27 exact comparison of 0.0 and -0.0 will change so there is a warning for that as well to help detect issues earlier.
* FIPS support in crypto module for OpenSSL 3.x
Resources that target young people or beginner (we already have some, but more, and more widespread).
Advertising outside the Elixir circle (Chris McCord asked the other day what podcasts would be a good fit for that).
Python has a major edge in France, being defined as the reference language (teached at high school and engineering schools), and as such it sets the frame.
For now, when I mention Elixir around me, although there is interest, the feedback is that it seems like a great language for connoisseurs and people with special needs (e.g. very high concurrency), which most people have no need for.
I personally think it's a great language overall, including from a maintenance point of view, and I try to advertise it as such.
Source: a dad with a 15-yo son who is totally into programming, and quickly saw that Python is everywhere together with Javascript (from a YouTube / blog posts / discord point of view). He doesn't understand why I "bother" with Elixir :-)
1. Lack of strong typing. Dynamic typing is great on 10,000 line projects with two developers. Once you reach 50,000 lines and have a little developer turnover, it stops being fun. And dynamic typing also usually means far less IDE support. I'm spoiled; I want auto-complete.
2. Elixir is amazing for UIs like Discord, where your users interact in real time. If each of your users lives in their own little world, Elixir's greatest strengths have less chance to shine.
Right now, TypeScript is a no-brainer, because React is the dominant choice for UIs, and using the same language on both client and server is a win. And frankly, TypeScript's lovely, even if the ecosystem is a constantly-churning trash fire. So for a variety of commercial projects, TypeScript is a responsible choice. Typed Python, C#, Java, Kotlin, Swift or Rust all have compelling use cases, too.
But given stronger typing and an project that needed Elixir's strong support for distributed systems, I'd definitely consider Elixir.
So I want to put on my consultant/lead engineer hat for this discussion. :-) I've worked for years in Ruby, Lisp, Scheme and other untyped languages, and shipped some fairly large projects.
Any project where you can rightfully say "I've built it" is small enough that dynamic typing can be a pretty pleasant strategy. (I'd miss good auto-completion, though.)
But when I walk onto a project with 5-20 engineers and 5 years of code, dynamic typing typically makes everything slow and painful. "We need to update two major Rails versions and deal with 40 supporting libraries. What's the plan for making sure everything still works? We have OK test coverage, but can we pull half the team for 2 weeks to work through the breakage?"
Or perhaps you reach a scale where you need to onboard a bunch of junior developers or data scientists. Here, static typing helps people get up to speed, it encourages them to refactor aggressively, and it limits breakage caused by miscommunication.
50,000 to 250,000 lines of statically-typed code maintained by a 5 person team with normal turnover can make for a very pleasant project. But in that size range, I really don't enjoy dynamic typing. It's doable, but I can really feel the difference.
I am extremely optimistic that Elixir could support excellent static types. And if it does, I'd definitely consider it for a wider range of projects.
> The power of a type system, the expressiveness of functional programming, and the reliability of the highly concurrent, fault tolerant Erlang runtime, with a familiar and modern syntax.
Haven't used it, but I'd like to hear about others experience.
https://elixir-lang.org/blog/2023/09/20/strong-arrows-gradua...
- Ericsson (they designed Erlang) is using it for its mobile products/networks. Approx 40% of all mobile traffic worldwide goes through Ericsson's infra.
- WhatsApp used/uses Erlang - again millions (billions?) of users
- Bet365 / Klarna use Erlang - millions of users
- from OSS, RabbitMQ, is written in Erlang
I could go on... Erlang has great adoption, but it's probably smaller than, e.g. Python, due to Erlang's specific applications.
in the programming community as a whole erlang/elixir are not very common.
it's important to ack this distinction becuase we still have a hill to climb with spreading awareness for OTP.
1. Not enough companies sharing their experience - most companies want to lead and not follow. There has been a lot of success with Elixir but too often I've heard companies not wanting to indicate they're using Elixir out of concern that it will tip off their competition (I've heard this from two companies now, big ones) or they just don't have a culture of sharing. The most impactful case study on Elixir was one of the first, the Bleacher Report case study https://www.erlang-solutions.com/case-studies/bleacher-repor.... They did an excellent job of articulating the direct business value of transitioning from Rails to Phoenix. We need more of these stories shared
2. Lack of talent - this is a chicken or egg problem but one thing that comes up quite often. If there were more companies actively hiring in Elixir there would be more devs learning Elixir but companies are reluctant to hire for Elixir because there aren't enough devs. Most of the courses/guides are aimed at transitioning Senior level talent. At DockYard through our Academy https://academy.dockyard.com/ we decided to tackle this from the opposite end and commit to producing high quality junior Elixir engineers.
3. Higher price point for talent - the recent Stack Overflow surveys regularly rank Elixir at the top for cost. Personally I think this stat is being influenced by the fact that Elixir has a higher % of more Senior level talent so the average salary will be skewed higher. But that nuance doesn't come through in the cost rankings. If true then as the ecosystem's talent pool starts to normalize we should see this cost average out.
4. Lack of off-the-shelf solutions - Objectively Elixir is faster to build and less costly to maintain than nearly every other tech stack I've ever used. However, because it is so easy to build something we have developed a culture of "just make it from scratch" too often. When newbies ask "how do I do X" questions like auth or something else in another tech stack that they can simply add a node module or a Ruby gem in Elixir most times we lack those pre-built solutions. So the actual build cost is higher because we can't simply drop that solution in and accelerate the dev process.
I've been selling Elixir services for almost a decade at this point and I'm still very bullish on Elixir. We, along with many others, have been trying to close these gaps. It's happening but the most impact anyone reading this can make that is using Elixir is to share your experiences. Write blog posts. Publish case studies. Indicate the reduction in cost, faster times to market, etc... these are the data points that are going to be most compelling and important in the coming decade.
Elixir is sort of a gateway drug to Erlang, sooner or later you want the real thing.
I cannot agree. Erlang is a pita to with. From the lackluster test libraries, the lack of module aliasing, the atrocious UX of parse transform, and so much more.
Erlang bring nearly nothing that Elixir cannot do and the UX of elixir is by far better.
And I am not talking of syntax here.
Erlang was made by and for that generation, with the tools that were there at the time and so it 'speaks to me' in a way that I'm familiar with. Going on the assumption that you've learned to program a little later, by the time that eco systems were more fully featured, IDEs were commonplace and things like test libraries and namespaces were common I can totally understand why something like Erlang would leave you frustrated.
But to me it is a luxury and always will be. So long term, Elixir probably has the future, but that battle hardened foundation that it builds on stems from a different era, where reading the documentation was a prerequisite for doing anything with a system and all systems had their little quirks. Erlang's roots in turn are partially in Prolog, which is another one of those beasts of times long gone, it too is quirky and some of that shows through in Erlang and the way it approaches things.
In my experience it pays off to embrace such things rather than to fight them, then you can spend your time and energy far more productive rather than to let it be in the front of your focus while you're trying to get some work done. In the end it is all tools, and just like in the trades there are no bad tools.
I did not talk of reading documentation because i am literally one of the person you should come to if you need something in the doc. I learned how to write elixir and erlang by reading their doc.
I also do not use an IDE and am really relucant of them because i don't find them useful.
And I am a big defender of prolog. And i am not denying the strong architecture and infrastructure of erlang. I literally contribute to it.
But what you have not addressed is ... all I mentioned. The broken testing libraries. The inability to do the right thing nicely because the parse transform are a pita. The fact that modules names cannot be aliased, generating painful to navigate file structure and names. Etc etc.
And look. It can be made better. We have worked on documentation and there is more coming. We are getting multiline strings. Rebar3 mostly work. Hex can be used for erlang projects.
But in order to do things better we need to acknowledge the problems. Not just say that the kids don't understand. Especially when the person you say that to is actually understanding it pretty well thank you.
Anyway, I'm kind of done with the personal attacks so have a nice day.
That said, Elixir is nicer to read but I was spoiled by Ruby. Erlang is nicer than most of the stuff of the 80s though.
However I wish the Elixir team had more courage and did without all the boilerplate of those handle casts and calls in servers. They used macros to do all sort of things, so I guess they could use them to bury that complexity into the depths of the language and let us use real function names to call servers. It would start to open a gap between Elixir and Erlang but it would be worth it.
I've found that reading any source code is made orders of magnitude difficult by the use of metaprogramming. Sure, it makes things much easier for people who are just picking up the language, but at the cost of imposing a literal wall in front of them when they try to go in deeper.
Case in point, is metaprogramming in Ruby on Rails. It's just not worth it if used that extensively. I'm very thankful that the Elixir source is super approachable for anyone who wants to read it.
So for me at least, both have their strength but I wouldn't say you need Erlang, Elixir can work fine on its own.
OTP is a fantastic example of this sort of thing: it is arcane, fairly complex but surprisingly well built and the quality of the architecture is something most modern frameworks can only dream of. The only reason it is still around is for the same reason that crocodiles are still around: it is extremely well adapted to the niches where it is present, all the other stuff has died out. You can re-invent all (or usually part) of OTP in any language and there have been many attempts at doing just that. But why would you? It already exists and it is far more battle hardened and time tested than anything new likely will ever be. Ok, you'll have to bite that bullet...
I think many web developers do just fine without ever having to touch Erlang, which is one of the nice things I like about Elixir: it's making a platform that is almost tailor made to web development really accessible to web developers. There is an idea outside the ecosystem that Elixir is "overkill" for simple apps but that's just not true!
For those that find themselves needing to dive into Erlang, I too have seen several people describe it as "Biting the bullet," and I'm a bit in that camp myself. For me it's not Erlang the language that's a problem, it's the documentation. This speaks to what you said about being from another era. I just do not find it enjoyable to read.
You otherwise seem to be saying that modern tooling is bad because it doesn't force you to be as diligent a programmer. I neither agree nor disagree with this but it's a whole other convo.
1. Erlang / BEAM itself as the runtim with lots of powerful capabilities (what you probably are talking about) - you need to know this stuff at some point to fully leverage the technology
2. Elixir the language (+ excellent modern tooling) ontop, which really helps to significantly speed up productivity/average code quality - I don't like to over-reductionist approach here, DX is also an important thing to consider instead of the raw foundation
Are you in the REPL all day? Any major interop issues?
Without it, to run RabbitMQ in a Windows containers, you have to run the enormous Server Core images.
A niche issue, I know, but a constant thorn in my side!