FreeBSD/EC2 boot performance over time
109 points
3 days ago
| 10 comments
| daemonology.net
| HN
nijave
2 days ago
[-]
Presumably seconds since I'm not sure what else would make sense here. It'd also be helpful if the y axis had consistent scale between each graph and horizontal lines are set y axis intervals

Seems zfs is quite a bit faster than ufs

reply
cperciva
2 days ago
[-]
The different X axes are because I figured there's no point having a bunch of graphs which are 90% empty. For a long time we only had base/UFS images. (Even arm64 came a few years later than amd64.)
reply
mkesper
2 days ago
[-]
But you aren't able to compare them then.
reply
bas
2 days ago
[-]
Your FreeBSD on AWS work is appreciated, @cperciva.
reply
defrost
2 days ago
[-]
That's an impressive drop from 30 minutes in 2019 to under 10 minutes today.

No, wait .. maybe that's seconds? milliseconds?

reply
cperciva
2 days ago
[-]
Seconds, yes. Sorry I figured that was obvious, I'll add it to the graphs I generate next week.
reply
amiga386
2 days ago
[-]
If you're taking requests, then can the Y axis start at zero, please?

https://handsondataviz.org/images/14-detect/gdp-baseline-mer...

And what is going on with that Y axis? There is no consistent spacing between the marks!

https://imgur.com/HaJmTSU

reply
cperciva
2 days ago
[-]
It's a bit hard for a log axis to start at zero.
reply
amiga386
2 days ago
[-]
Well start it at 0.00001 then. There's no need for a log axis, as all the values are within 2 orders of magnitude of each other - between 1 and 100. They will fit better, and will no longer be distorted, on a linear Y axis starting from zero.
reply
defrost
2 days ago
[-]
Good to hear.

Seconds was my best guess .. but that doesn't make it certain for the viewer.

No real drama, but FWiW I've looked at a lot of data across engineering, math, geophysics, mining, energy, etc and unlabeled data and graphs are a major annoyance.

reply
pushupentry1219
2 days ago
[-]
Thank you! Was about to ask for axis labels as well. I assumed seconds but I had some doubts because there were no labels!
reply
wright08
2 days ago
[-]
I wouldn't bother. It is super obvious.
reply
spease
2 days ago
[-]
Microts!
reply
alexellisuk
2 days ago
[-]
Nice improvements in boot speed. Perhaps a little blurb / intro / summary would be helpful to the post to help with understanding the achievements made.

Did the patches ever make it into Firecracker for booting FreeBSD as a guest? I looked back at the paper trail and it seemed like it may have stalled.

Does anyone know?

reply
cperciva
2 days ago
[-]
Things stalled; there were a couple outstanding bugs (one I know has now been fixed, not sure about the other) and I had to take over FreeBSD release engineering and didn't have time to follow up on the Firecracker bits.

I understand that NetBSD can boot in Firecracker using the same patches so I'm hoping they can resolve any remaining issues and prod the Firecracker developers into merging things.

reply
nostrebored
2 days ago
[-]
Fixing instance types was probably wrong.

You’re getting progressively legacy (and more likely to be degraded) hardware. This impacts how tightly packed the instance type as a whole is, which impacts launch instance performance

reply
cperciva
2 days ago
[-]
Fixing instance types was probably wrong.

Depends what you're trying to measure, I think? My goal as a FreeBSD developer is to look at FreeBSD performance -- this is both to show the improvements which have been made over the years and to alert me to any performance regressions (I generate these graphs automatically when I build the weekly snapshots).

If you want to compare EC2 to other clouds you would definitely want to use the latest instance generation, of course.

reply
silisili
2 days ago
[-]
Would be nice to see how this compares to Linux, I think, for perspective.
reply
cperciva
2 days ago
[-]
I did that comparison a few years ago: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-be...

Re-running the comparison with Linux AMIs from 2024 is on my to-do list.

reply
andai
2 days ago
[-]
Dang, Clear Linux boots in one second? How is it ten times faster?
reply
transpute
2 days ago
[-]
Optimized for automotive use case (max 2s to enable rear camera), https://www.phoronix.com/news/Clear-Linux-Kernel-3s-to-300ms

> They started out with around a three second kernel boot time but cut it down to just 300 ms. Among the optimizations carried out to really speed-up their boot time were ensuring more asynchronous driver probing, only initializing a small amount of RAM at start and then after booted hot-plug the rest of it in parallel via systemd, optimized root file-system mounting, disabling unnecessary kernel modules, and similar approaches. Moving forward they are still looking at optimizations for the boot process around in-kernel deferred memory initialization, SMP initialization changes, ACPI tweaking, and user-space/systemd optimizations.

reply
homebrewer
2 days ago
[-]
It's been possible for a long time:

https://lwn.net/Articles/299483

reply
AdieuToLogic
2 days ago
[-]
Is there an architectural change and/or approach to which the boot performance increase(s) can be attributed?

If not, which is understandable, is there something specific to stable/14 for interested parties to familiarize themselves with?

reply
cbhl
2 days ago
[-]
My impression is that cperciva has been working on FreeBSD/EC2 boot times for the last six years, not just specifically some changes in 14.

The most accessible summary of his work is probably on his Patreon: https://www.patreon.com/c/cperciva/posts. If you go digging there are also some anecdotes on his Twitter too; most recent I can find is https://x.com/cperciva/status/1833735559614988526

reply
cperciva
2 days ago
[-]
I gave a talk at BSDCan 2022 about this ("speeding up the FreeBSD boot") -- I'm on my phone right now so I can't provide a link but it's on YouTube.

There's also a somewhat out of date summary of most of the work on the FreeBSD wiki: https://wiki.freebsd.org/BootTime

reply
homebrewer
2 days ago
[-]
reply
cperciva
2 days ago
[-]
The best summary I can provide is "profiling and lots of small wins".
reply
Temporary_31337
2 days ago
[-]
That suits our deployments where for HA we simply terminate and replace unresponsive EC2 nodes with new ones. I’ll have to talk to other developers to see how much work would it be to compile to freebsd instead of Linux
reply
wilted-iris
2 days ago
[-]
Linux probably boots faster fwiw. As a starting point, see this bench: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-be...
reply
wadefletch
2 days ago
[-]
What are the y-axis units?
reply
NekkoDroid
2 days ago
[-]
I would guess hamburgers eaten :)

On a more serious note, the only thing that really makes sense would be seconds.

reply
pluto_modadic
2 days ago
[-]
wow they make it impressively hard to contact them.
reply
cperciva
2 days ago
[-]
me @FreeBSD.org. I figured most people reading this would know who I am.
reply
tpxl
2 days ago
[-]
I have no idea who you are, nor do I check who submits what (the merit of the argument, not who said it and all that), nor do I have a reason to know (I have no reason to follow FreeBSD at this point).

I understand you do important work, but some people, me included, do other stuff. And there's a lot of people here.

reply
Terretta
1 day ago
[-]
cperciva is the HN comeback of all time:

https://news.ycombinator.com/item?id=35079

(And folks have had since 2007 to top it.)

Funny enough, that started with a "I don't know you" as well:

https://news.ycombinator.com/item?id=35071

reply