I have noticed that every distributed app I build, as it gets more concurrent, I end up reinventing a lot of operating system work. I end up rebuilding scheduling, different caching techniques, heuristics to try and preemptively add things to cache, etc.
I really should learn more about kernel design. I would probably still be a better distributed systems engineer if I lift more concepts from OS design.
...and a little further on, the OS/app starts to look like a distributed database.
I think it's a good case for including lattice types in the OS ie. From the ground up. Bear in mind that an OS has an API ie. A DSL/language for it, Micropython is a good example:
> Hey! This is an old, abandoned project, with both technical and design issues. Please have fun with this tutorial but do look for more modern and authoritative sources if you want to learn about OS design.
On a related topic - has anyone tried to move the build process of a very old Linux version, say Linux 0.92/0.12 to modern toolchains on modern computers? I believe the original build process requires gcc 1.4.0 and other programs such as `as` which are not available on modern systems.
The target is to be able to build Linux 0.12 using modern gcc and run it in QEMU, or preferable on a true 80386 machine. AFAIK, modern gcc still supports this architecture, so in concept this should be possible. There might be a LOT of changes in source code to be made, though.
The idea behind is to obtain a very old Linux that can be built easily on modern systems, with modern C support, and gradually add my own stuffs using the same tools, so that I don't limit myself to very old toolchains.
Edit: The reason to pick Linux 0.12/0.92 is because 1) It is kinda complete (without network I believe) but not too complex (I believe the loc is under 20K), and 2) It is not a toy OS and can run on real hardware (a 80386 machine), and 3) we have an excellent book for it: https://download.oldlinux.org/CLK-5.0-WithCover.pdf
"NCommander" on YouTube could probably be a good starting point to get a feel for what is required and to ask some questions. Features very interesting/insane archaeology builds like bootstrapping Linux from Scratch on WinXP SFU which ships with an incomplete toolchain from 1993 (?). 8)
The program in the boot sector is run similar to other software: the code is loaded into memory, and then the processor is jumped to the first instruction. This loading and jumping is done by the firmware, which is included in your hardware, separate from your disks.
Let's back up to the start. When you switch on a computer, the power rails on a bunch of the chips come up. As this happens, the chips enter a "reset" state with initial data that is baked into the circuitry. This is the power-on reset (PoR) circuitry. When the power is up and stable, the "reset" is over and the processor starts executing. The initial value of program counter / instruction pointer is called the reset vector, and this is where software execution begins. On an x86 PC, this is something like 0xFFFFFFF0. The memory controller on the system is configured for reads from this address to go NOT to the main RAM, but to a flash memory chip on the board that holds the firmware software. From there the firmware will find your bootable disks, load the bootloader, and pass control to it.
CPU starts executing instructions from firmware (BIOS or UEFI). It performs some complicated computer-specific initialisation, discovers RAM, somewhat initialises PCI-E, USB, other peripherals and finally transfers control to the user-specified code. It's either instructions located at specific address of configured disk (old BIOS computers with MBR) or just a file in FAT32 partition (UEFI).
While I agree this might be a fun resource and useful example code for various aspects of legacy x86 interfacing, I would urge anyone who hopes to actually get into OS development to ignore this (and in fact every other tutorial I’ve ever seen, including those hosted on the popular sites).
For all the reasons stated in the link from the README [1] and agreed by the author, this project should not be followed if one wants to gain an understanding of the design and implementation of operating systems for modern systems. Following it will likely lead only to another abandoned “hello world plus shell” that runs only in emulation of decades old hardware.
My advice is get the datasheets and programmers’ manuals (which are largely free) and use those to find ways to implement your own ideas.
People interested in a "read the manual and code it up on real hardware"-type guide should take a look at Stanford's CS140E[1] repo! Students write a bare metal OS for a Raspberry Pi Zero W (ARMv6) in a series of labs, and we open source each year's code.
There aren't any 1hr+ lectures, just some readings (selected from the manuals in `docs/`) and a bit of exposition from the professor before diving into the lab. Lots of "as needed" assistance
not really my cup of tea, but a random feedback: take a look at nand2tetris as it's really "user friendly" and, if I'm not mistaken, was even made into a game on Steam.
It is user friendly, and it's astounding how much they managed to pack into a single semester. Highly recommended!
However, it's arguably too idealized and predetermined. I think you could get all the way through building the computer in https://nandgame.com/ without really learning anything beyond basic logic design as puzzle solving, but computer organization, as they call it in EE classes, is more about designing the puzzles than solving them. Even there, most of what you learn is sort of wrong.
I haven't worked through the software part, but it looks like it suffers from the same kind of problems. IIRC there's nothing about virtual memory, filesystems, race conditions, deadly embraces, interprocess communication, scheduling, or security.
IMO this is kind of the tradeoff. In 140E we do touch on virtual memory (w/ coherency handling on our specific ARM core), FAT32 "from scratch", etc. but it comes at the expense of prerequisites. There is a lot of effort to "minify" the labs to their core lesson, but there is an inevitable amount of complexity that can't (or shouldn't) be erased.
Obviously you'll need to read the manuals to get much done, but these kinds of tutorials are complimentary.
The issue with x86_64 is that you need to understand some of the legacy "warts" to actually use 64-bits. The CPU does not start in "long mode" - you have to gradually enable certain features to get yourself into it. Getting into 32-bit protected mode is prerequisite knowledge to getting into long mode. I recall there was some effort by Intel to resolve some of this friction, breaking backward compatibility, but not sure where that's at.
The reason most hobby OS projects die is more to do with drivers. While it's trivial to support VGA and serial ports, for a modern machine we need USB3+, SATA, PCI-E, GPU drivers, WIFI and so forth. The effort for all these drivers dwarfs getting a basic kernel up and running. The few novel operating systems that support more modern hardware tend to utilize Linux drivers by providing a compatible API over a hardware abstraction layer - which forces certain design constraints on the kernel, such as (at least partial) POSIX compatibility.
Even taking only x86_64 as an example, going from real to long modes is primarily of concern to those writing firmware these days - a modern operating system will take over from UEFI or a bootloader (itself usually a UEFI executable). The details of enabling A20, setting up segmentation and the GDT, loading sectors via BIOS etc are of course historically interesting (which is fine if that’s the goal!) but just aren’t that useful today.
The primary issue with most tutorials that I’ve seen is they don’t, when completed, leave one in a position of understanding “what’s next” in developing a usable system. Sticking with x86_64, those following will of course have set up a basic GDT, and even a bare-bones TSS, but won’t have much understanding of why they’ve done this or what they’ll need to do to next to support syscall, say, or properly layout interrupt stacks for long mode.
By focusing mainly on the minutiae of legacy initialisation (which nobody needs) and racing toward “bang for my buck” interactive features, the tutorials tend to leave those completing it with a patchy, outdated understanding of the basics and a simple baremetal program that is in no way architected as a good base upon which to continue toward building a usable OS kernel.
You can just copy and paste the initialization code. It only runs once and there's very little of value to learn from it (unless you're into retrocomputing).
disclaimer: I really have no idea about OSes, but hey...
Maybe it's a matter of marketing the product and managing expectations, but many of these projects are A ok being "legacy and obsolete" just for the sake of simplicity for introducing basic concepts.
Let's take two random examples.
(1) "let's create 3D graphics from scratch"
It's quite easy to grab "graphics gems" and create a comprehensive tutorial on software renderer. Sure it won't be practical, and sure it will likely end on phong shading, but for those wanting to understand how 3d models are translated into pixels on screen it's way more approachable than studying papers on nanites.
(2) "let's crate a browser from scratch"
It's been widely discussed that creating a new browser today would be complete madness (yet there's Ladybird!), but shaving the scope, even if it wouldn't be able to run most modern websites would be a interesting journey for someone who'd interested in how things work.
PS. Ages ago I've done a Flash webpage that was supposed to mimic a desktop computer for ad campaign for tv show. Webpage acted as a personal computer of main character and people could lurk into it between episodes to read his emails, check his browser history, etc.
I took it as a learning opportunity to learn about OS architecture and spent ungodly amount of unpaid overtime to make it as close to win3.1 running on dos as possible. Was it really an OS? Of course not, but it was a learning opportunity to get a grasp of certain things and it was extremely rewarding to have an easter egg with a command.com you could launch and interact with the system.
Would I ever try to build a real OS. Hell no, I'm not as smart as lcamtuf to invite couple friends for drinks and start Argante. :)
To pick on graphics, since I'm more familiar with that domain, the problem isn't that this tutorial is about software rasterization, it's that the tutorial is a raytracer that doesn't do shading, textures, shadows, or any geometry but spheres, and spends most of its word count talking about implementing the trig functions on fixed-point numbers instead of just using math.h functions on IEEE floats.
Go do the xv6 labs from the MIT 6.828 course, like yesterday. Leave all textbooks aside, even though there are quite a few good ones, forget all GitHub tutorials that have patchy support, blogs that promise you pie in the sky.
The good folks at MIT were gracious enough to make it available for free, free as in free beer.
I did this course over ~3 months and learnt immeasurably more than reading any blog, tutorials or textbook. There’s broad coverage of topics like virtual memory, trap processing, how device drivers work (high-level) etc that are core to any modern OS.
Most of all, you get feedback about your implementations in the form of tests which can help guide you if you have a working or effective solution.
Tanenbaum's textbook is highly readable, comprehensive (surveys every major known solution to each major prpblem), and mostly correct. xv6 may be a smaller, more old-fashioned, and more practical approach. RISC-V makes the usually hairy and convoluted issues of paging and virtual memory seem simple. QEMU's GDB server, OpenOCD, JTAG, and SWD can greatly reduce the amount of time you waste wondering why things won't boot. Sigrok/Pulseview may greatly speed up your device driver debugging. But I haven't written an operating system beyond some simple cooperative task-switching code, so take this with a grain of salt.
Funny question since you bring up JTAG and RISC-V -- do you have a cheapish RISC-V device you'd recommend that actually exposes its JTAG? The Milk-V Duo S, Milk-V Jupiter, and Pine64 Oz64 all seem not to expose one; IIRC, the Jupiter even wires TDO as an input (on the other side of a logic level shifter)...
I don't know what to recommend there. I have no relevant experience, because all my RISC-V hardware leaves unimplemented the privileged ISA, which is the part that RISC-V makes so much simpler. The unprivileged ISA is okay, but it's nothing to write home about, unless you want to implement a CPU instead of an OS.
I have noticed that every distributed app I build, as it gets more concurrent, I end up reinventing a lot of operating system work. I end up rebuilding scheduling, different caching techniques, heuristics to try and preemptively add things to cache, etc.
I really should learn more about kernel design. I would probably still be a better distributed systems engineer if I lift more concepts from OS design.
...and a little further on, the OS/app starts to look like a distributed database.
I think it's a good case for including lattice types in the OS ie. From the ground up. Bear in mind that an OS has an API ie. A DSL/language for it, Micropython is a good example:
https://www.neilconway.org/docs/socc2012_bloom_lattices.pdf
And ...
> Hey! This is an old, abandoned project, with both technical and design issues. Please have fun with this tutorial but do look for more modern and authoritative sources if you want to learn about OS design.
On a related topic - has anyone tried to move the build process of a very old Linux version, say Linux 0.92/0.12 to modern toolchains on modern computers? I believe the original build process requires gcc 1.4.0 and other programs such as `as` which are not available on modern systems.
The target is to be able to build Linux 0.12 using modern gcc and run it in QEMU, or preferable on a true 80386 machine. AFAIK, modern gcc still supports this architecture, so in concept this should be possible. There might be a LOT of changes in source code to be made, though.
The idea behind is to obtain a very old Linux that can be built easily on modern systems, with modern C support, and gradually add my own stuffs using the same tools, so that I don't limit myself to very old toolchains.
Edit: The reason to pick Linux 0.12/0.92 is because 1) It is kinda complete (without network I believe) but not too complex (I believe the loc is under 20K), and 2) It is not a toy OS and can run on real hardware (a 80386 machine), and 3) we have an excellent book for it: https://download.oldlinux.org/CLK-5.0-WithCover.pdf
"NCommander" on YouTube could probably be a good starting point to get a feel for what is required and to ask some questions. Features very interesting/insane archaeology builds like bootstrapping Linux from Scratch on WinXP SFU which ships with an incomplete toolchain from 1993 (?). 8)
https://github.com/mariuz/linux-0.01 might give you some ideas.
`as` the assembler is still available.
in English: https://download.oldlinux.org/ECLK-5.0-WithCover.pdf
The author dismisses this as out of date, but this is one of the most straightforward examples I’ve seen. At least reading 00 and 01.
from the readme:
> College is hard so I don't remember most of it.
Interesting how counter-productive high stress environments can be for ingraining knowledge.
Also see https://wiki.osdev.org/Getting_Started
x86 only, unfortunately. But some parts can be borrowed for other ISAs.
I still don't understand what's "running" the boot sector.
The program in the boot sector is run similar to other software: the code is loaded into memory, and then the processor is jumped to the first instruction. This loading and jumping is done by the firmware, which is included in your hardware, separate from your disks.
Let's back up to the start. When you switch on a computer, the power rails on a bunch of the chips come up. As this happens, the chips enter a "reset" state with initial data that is baked into the circuitry. This is the power-on reset (PoR) circuitry. When the power is up and stable, the "reset" is over and the processor starts executing. The initial value of program counter / instruction pointer is called the reset vector, and this is where software execution begins. On an x86 PC, this is something like 0xFFFFFFF0. The memory controller on the system is configured for reads from this address to go NOT to the main RAM, but to a flash memory chip on the board that holds the firmware software. From there the firmware will find your bootable disks, load the bootloader, and pass control to it.
In practice, systems vary wildly.
CPU starts executing instructions from firmware (BIOS or UEFI). It performs some complicated computer-specific initialisation, discovers RAM, somewhat initialises PCI-E, USB, other peripherals and finally transfers control to the user-specified code. It's either instructions located at specific address of configured disk (old BIOS computers with MBR) or just a file in FAT32 partition (UEFI).
That's why the word "boot" is there
If you wish to create an OS from scratch, you must first invent the universe.
Nice paraphrase of Sagan. :-)
While I agree this might be a fun resource and useful example code for various aspects of legacy x86 interfacing, I would urge anyone who hopes to actually get into OS development to ignore this (and in fact every other tutorial I’ve ever seen, including those hosted on the popular sites).
For all the reasons stated in the link from the README [1] and agreed by the author, this project should not be followed if one wants to gain an understanding of the design and implementation of operating systems for modern systems. Following it will likely lead only to another abandoned “hello world plus shell” that runs only in emulation of decades old hardware.
My advice is get the datasheets and programmers’ manuals (which are largely free) and use those to find ways to implement your own ideas.
[1] https://github.com/cfenollosa/os-tutorial/issues/269
People interested in a "read the manual and code it up on real hardware"-type guide should take a look at Stanford's CS140E[1] repo! Students write a bare metal OS for a Raspberry Pi Zero W (ARMv6) in a series of labs, and we open source each year's code.
Disclaimer: I'm on the teaching team
[1]https://github.com/dddrrreee/cs140e-25win
Do you have the course online? Looks like a bunch of files
There aren't any 1hr+ lectures, just some readings (selected from the manuals in `docs/`) and a bit of exposition from the professor before diving into the lab. Lots of "as needed" assistance
not really my cup of tea, but a random feedback: take a look at nand2tetris as it's really "user friendly" and, if I'm not mistaken, was even made into a game on Steam.
It is user friendly, and it's astounding how much they managed to pack into a single semester. Highly recommended!
However, it's arguably too idealized and predetermined. I think you could get all the way through building the computer in https://nandgame.com/ without really learning anything beyond basic logic design as puzzle solving, but computer organization, as they call it in EE classes, is more about designing the puzzles than solving them. Even there, most of what you learn is sort of wrong.
I haven't worked through the software part, but it looks like it suffers from the same kind of problems. IIRC there's nothing about virtual memory, filesystems, race conditions, deadly embraces, interprocess communication, scheduling, or security.
It's great but it's maybe kind of an appetizer.
IMO this is kind of the tradeoff. In 140E we do touch on virtual memory (w/ coherency handling on our specific ARM core), FAT32 "from scratch", etc. but it comes at the expense of prerequisites. There is a lot of effort to "minify" the labs to their core lesson, but there is an inevitable amount of complexity that can't (or shouldn't) be erased.
Obviously you'll need to read the manuals to get much done, but these kinds of tutorials are complimentary.
The issue with x86_64 is that you need to understand some of the legacy "warts" to actually use 64-bits. The CPU does not start in "long mode" - you have to gradually enable certain features to get yourself into it. Getting into 32-bit protected mode is prerequisite knowledge to getting into long mode. I recall there was some effort by Intel to resolve some of this friction, breaking backward compatibility, but not sure where that's at.
The reason most hobby OS projects die is more to do with drivers. While it's trivial to support VGA and serial ports, for a modern machine we need USB3+, SATA, PCI-E, GPU drivers, WIFI and so forth. The effort for all these drivers dwarfs getting a basic kernel up and running. The few novel operating systems that support more modern hardware tend to utilize Linux drivers by providing a compatible API over a hardware abstraction layer - which forces certain design constraints on the kernel, such as (at least partial) POSIX compatibility.
Even taking only x86_64 as an example, going from real to long modes is primarily of concern to those writing firmware these days - a modern operating system will take over from UEFI or a bootloader (itself usually a UEFI executable). The details of enabling A20, setting up segmentation and the GDT, loading sectors via BIOS etc are of course historically interesting (which is fine if that’s the goal!) but just aren’t that useful today.
The primary issue with most tutorials that I’ve seen is they don’t, when completed, leave one in a position of understanding “what’s next” in developing a usable system. Sticking with x86_64, those following will of course have set up a basic GDT, and even a bare-bones TSS, but won’t have much understanding of why they’ve done this or what they’ll need to do to next to support syscall, say, or properly layout interrupt stacks for long mode.
By focusing mainly on the minutiae of legacy initialisation (which nobody needs) and racing toward “bang for my buck” interactive features, the tutorials tend to leave those completing it with a patchy, outdated understanding of the basics and a simple baremetal program that is in no way architected as a good base upon which to continue toward building a usable OS kernel.
You can just copy and paste the initialization code. It only runs once and there's very little of value to learn from it (unless you're into retrocomputing).
disclaimer: I really have no idea about OSes, but hey...
Maybe it's a matter of marketing the product and managing expectations, but many of these projects are A ok being "legacy and obsolete" just for the sake of simplicity for introducing basic concepts.
Let's take two random examples.
(1) "let's create 3D graphics from scratch" It's quite easy to grab "graphics gems" and create a comprehensive tutorial on software renderer. Sure it won't be practical, and sure it will likely end on phong shading, but for those wanting to understand how 3d models are translated into pixels on screen it's way more approachable than studying papers on nanites.
(2) "let's crate a browser from scratch" It's been widely discussed that creating a new browser today would be complete madness (yet there's Ladybird!), but shaving the scope, even if it wouldn't be able to run most modern websites would be a interesting journey for someone who'd interested in how things work.
PS. Ages ago I've done a Flash webpage that was supposed to mimic a desktop computer for ad campaign for tv show. Webpage acted as a personal computer of main character and people could lurk into it between episodes to read his emails, check his browser history, etc. I took it as a learning opportunity to learn about OS architecture and spent ungodly amount of unpaid overtime to make it as close to win3.1 running on dos as possible. Was it really an OS? Of course not, but it was a learning opportunity to get a grasp of certain things and it was extremely rewarding to have an easter egg with a command.com you could launch and interact with the system.
Would I ever try to build a real OS. Hell no, I'm not as smart as lcamtuf to invite couple friends for drinks and start Argante. :)
To pick on graphics, since I'm more familiar with that domain, the problem isn't that this tutorial is about software rasterization, it's that the tutorial is a raytracer that doesn't do shading, textures, shadows, or any geometry but spheres, and spends most of its word count talking about implementing the trig functions on fixed-point numbers instead of just using math.h functions on IEEE floats.
Well put! This succinctly sums up the crux of my argument in my other comments.
great counterpoint :)
risc-v seems is a clean-sheet design and that should be a good starting point (imho).
fwiw, xv-6, the pedagogical os has migrated to it.
I don't even know where to _begin_ writing an operating system.
If i wanted to learn just so i have a concept of what an os does, what would you recommend?
I'm not trying to write operating systems per se. I'm trying to become a better developer by understanding operating systems.
Go do the xv6 labs from the MIT 6.828 course, like yesterday. Leave all textbooks aside, even though there are quite a few good ones, forget all GitHub tutorials that have patchy support, blogs that promise you pie in the sky.
The good folks at MIT were gracious enough to make it available for free, free as in free beer.
I did this course over ~3 months and learnt immeasurably more than reading any blog, tutorials or textbook. There’s broad coverage of topics like virtual memory, trap processing, how device drivers work (high-level) etc that are core to any modern OS.
Most of all, you get feedback about your implementations in the form of tests which can help guide you if you have a working or effective solution.
10/10 highly recommended.
Tanenbaum's textbook is highly readable, comprehensive (surveys every major known solution to each major prpblem), and mostly correct. xv6 may be a smaller, more old-fashioned, and more practical approach. RISC-V makes the usually hairy and convoluted issues of paging and virtual memory seem simple. QEMU's GDB server, OpenOCD, JTAG, and SWD can greatly reduce the amount of time you waste wondering why things won't boot. Sigrok/Pulseview may greatly speed up your device driver debugging. But I haven't written an operating system beyond some simple cooperative task-switching code, so take this with a grain of salt.
Funny question since you bring up JTAG and RISC-V -- do you have a cheapish RISC-V device you'd recommend that actually exposes its JTAG? The Milk-V Duo S, Milk-V Jupiter, and Pine64 Oz64 all seem not to expose one; IIRC, the Jupiter even wires TDO as an input (on the other side of a logic level shifter)...
That doesn't seem off-topic at all to me!
I don't know what to recommend there. I have no relevant experience, because all my RISC-V hardware leaves unimplemented the privileged ISA, which is the part that RISC-V makes so much simpler. The unprivileged ISA is okay, but it's nothing to write home about, unless you want to implement a CPU instead of an OS.
If you want to practice, try: https://littleosbook.github.io/
I am an occasional uni TA that teaches OS and I use littleosbook as the main reference for my own project guidebook.
It's a decent warm-up project for undergraduates, giving them a first-hand experience programming in a freestanding x86 32-bit environment.
Start with a simple program loader and file system, like DOS.