Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use
https://github.com/jart/cosmopolitan
to make a binary that runs in many places including bare metal.
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
> the linux kernel configuration menu, a wonderful text menu system with a thousand options which has been baffling new users for about 30 years now.
So true.
In addition to C in the article and Rust linked to in the article, Go fans can use the similar https://gokrazy.org/ project.
Cool article! I'm working on a tangentially related issue requiring microVMs inside isolated infrastructure environments. Latency isn't really my main priority, but I am always tempted by any option to minimize attack surface. I wonder what it would take to replace the host block mount in this configuration with `vsock` for all communications between the host and guest microVM? Then you could avoid any files being mounted on the host at all while still enabling, e.g., one-way egress to a pre-signed S3 URL via a private VPC endpoint. Very cool article!
Amazon Nitro Enclave does pretty much this, the guest has one method of communication, via vsock, and it's up to you to build the pipes on either side.
It's a huge PITA in practice because whatever you want to run inside some enclave usually ends up being a 'normal program' that needs to talk TCP/IP over sockets... so your vsock I/O becomes a weird mix between a TUN proxy or a SOCKS5 local listener inside the VM that tunnels through vsock.
For example, I have the Windows NT 3.50 kernel compiling from scratch with virtio-net drivers, it's fairly straightforward for me to add a bus driver that runs over vsock inside Nitro Enclave that exposes itself (o,o) as a NIC then handle the tunneling logic in a usermode process in the host - but I don't uderstand the point of why you would do that when you already have sufficient attestation methods that don't require you to do vsock isolation.
> I found “Building a tiny Linux from scratch” which does most of what I do here but in Rust and a year ago
Linux from scratch seems to still be doing fine at: https://www.linuxfromscratch.org. It's going on 27 years now.
> yeah, I know, proper C code needs to be scattered with return value checks and sensible reports of errno. I’ve left these out for clarity.
Somewhere, a LLM is trained on this code as we speak :)
But anyway, it's great that people are still interested in learning this stuff for fun.
It's be cool to dual boot with a Linux that has a ~1s boot time, drops you into neovim and lets you save text files to a shared partition.
It's possible to do a similar thing with any old smartphone. :)
https://xnux.eu/p-boot/
I... fail to see the point of running just one process.
If it's just a PoC, then:
1) I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
2) To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
3) It takes 1s to boot linux, but an ordinary PC takes 10s to get to that linux. Even U-boot on ARM takes some seconds to load a kernel.
BTW, if anyone knows any current platform that can XiP a linux kernel, please share.
> I remember seeing a linux firewall/gateway set up to run with just the kernel, without any userspace at all. Completely unhackable.
Do you remember any details that would let me search for it? Because that does sound cool, and even maybe useful; the thought has certainly crossed my mind that a router or VPN box doesn't really get a lot of use out of userspace... Although maybe it's worth keeping for control/configuration/debugging.
> To print some text or run a simple program, I belive DOS without a memory manager would be even faster.
Or just make your code boot directly. It's not hard to make a .efi, or use https://github.com/jart/cosmopolitan to make a binary that runs in many places including bare metal.
> control/configuration/debugging
This is one of several major arguments made against unikernels in that famous Triton rant from a decade ago:
https://tritondatacenter.com/blog/unikernels-are-unfit-for-p...
Basically, even if your application _can_ run as the kernel, and it's desirable for it to run with kernel-level permissions, do you really want production to be a world without strace and iotop and the like?
IIRC, it ran a script as init process that set up the network cards, set up iptables, etc. and then just exited. Kernel would panic (the "init was killed" panic), but the network would still be functional. Automatic reboot on panic was disabled.
To reconfigure, the admin would simply reset it and start the system with "init=/something/else" as kernel parameter that booted to a normal userspace.
Oh, clever; I didn't know you could make Linux panic but keep running.
Wouldn't this be useful for embedded types of applications where you have a very specific task you want to do and you want to do it now.... like that firewall example?
Yeah, but there's a problem.
Do do something actually useful, the program would have to access some data: network, disk, some sensors, etc. Network alone means scanning PCIe for the network card and configuring it, disk access needs controller also on PCIe, then scanning the ports for the drive, reading partition table, mounting the partition, etc.
All that takes a lot more than 1s. The speedup might not even be significant compared to a kernel optimized for that system (all modules built-in, nothing redundant), but full-featured, plus busybox or sysvinit alone.
Embedded devices or other SoC situations, certain limited scope situations where docker is undesirable/unnecessary, etc.
> I... fail to see the point of running just one process.
It makes sense if you got some legacy piece of hardware that has extremely limited resources, both in terms of RAM and storage. Write your code in Go and you don't even need libc any more.
At that point, it would seem that an RTOS would be even more efficient - and if multithreading is not necessary, then just run it on bare metal.
I do like this as an exploration.
It's possible to boot a VM noticeably faster still, though I'm unclear on whether any of that applies to hardware:
https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-l...
You may also want to build and run busybox for your tiny userspace.
Other things you may want to experiment with is gen_init_cpio.c from linux kernel tree. It makes creating initramfs file structure easier from scripts.
And finally if sys/isolinux is also fun to use for minimal boot images.
"Butt Naked Linux" is how I read it.
I know it's off topic. I accept my downvotes.
I could end up stealing that for my own tinkering distro that never ends up releasing because it's awful.