I think Casey is currently the most informed person to make a series of books or articles summarizing the history of SW Engineering, all the lessons learned and forgotten, and all the good stuff that was published and still hasn't gained traction in the practice
You think there aren't other people equally or more qualified? I mean Casey may be the most visible person with those qualifications - but lack of visibility doesn't imply lack of competence, plenty of brilliant people work in the background..
Feel free to point to anyone doing these deep dives that specifically tackle the lost knowledge of the early decades of computer science and the ideas that have not yet come to fruition despite being extremely old
> Feel free to point to anyone doing these deep dives
Most of this kind of content comes off as relevant/topical but mindless entertainment. This is not a "deep dive" of anything that I could practically apply in my daily work. It feels good to think this content might add value and then to subsequently consume it, but it's effectively junk food.
I used to spend a lot of time watching crap like lex and primagen before it became obvious that these kinds of information channels are not meaningfully enriching despite sounding like they should be to a casual observer.
As much as I like to watch Casey, Prime, etc. on YouTube, they are not going to teach anybody anything. They are just entertainment for software people with a certain viewpoint.
My one data point is that Casey’s handmade hero series, and his immediate mode gui video from way back in 2005, are what introduced me and some friends to an entirely new way to build graphical user interfaces, and I do feel a great appreciation for that. Guys like Ryan Fleury (of RadDbg) and Vjekoslav Krajacic of Filepilot similarly credit Casey with their “radicalization” :)
To your point though, it was less about specific technical knowledge conveyed by these videos and more about him evangelizing a high level approach that many folks otherwise wouldn’t have considered
I want more things like that old hand made hero to exist in the world, but, from an interview with Casey himself, it doesn’t pay well enough to justify. Someone with the chops to put that together is in a position to be doing something much better incentivized.
You are making many assumptions. First of all, he is doing the deep dives by researching all those articles, and of course the resulting presentation will be very superficial. The point is exactly to publish a book to allow the reader to partake in the deep dive.
Secondly, nobody said anything about applying that stuff to your daily work.
I feel like the comments that are giving you pushback haven't actually listened to Casey before. I agree with you, while opinionated his perspective is very pragmatic and misunderstood. He's done such a great job giving his experience out to the broader world too.
He's entertaining and well spoken. Love it. Not sure the book medium would be the best avenue. My own sweet spot is presentations, live interviews and meetings. I can't write for sh*t, even if I tried. I know.
Ehh I love Casey and have learned a ton by watching how he thinks about things in his handmade hero series, but he is fairly narrow minded in his views of dev. Not that theres anything wrong with that, for the kinds of dev he does his approach is very good. But its not generalizable.
How is this relevant? I didn't say he should make a book about his teachings about SWE, I said he could compile a comprehensive review of the foundational literature of compsci
I suppose it is a matter of what one works on, but that has been consistent in my career.
I have seen many be rewarded for choosing what I would deem intentional, gross negligence. But hey, those devs get work finished faster, and that's all that matters to the non-technical folks.
His dismissal of the argument Knuth makes regarding the hot loops could have been explored a bit better. I found it weird he didn’t mention the difference of types of programs of then vs now. Even today, in scientific code it is still absolutely the case a lot of the time that a huge chunk of the runtime comes from a single very very hot loop. It might be hidden in a library, but it’s there.
Instead he focuses only on “program size”.
Knuth samples where very small FORTRAN programs (compared to today’s standards). Today’s program are bigger but the fundamental number crunching primitive of “let’s compute stuff in a loop” remains. It’s just buried under a pile of extra cruft (data loading, parallelism, dispatching etc).
Now we just deal with a lot more programs that are of a whole different class compared to what they where doing with computers in the 70s. We have much more I/O involved. And hot loops don’t like being I/O bound.
Being I/O bound is usually a result of bad engineering practices though.
If you're I/O bound, that either means the problem doesn't require much computation - which is possible but fairly rare, or more likely that your code is so unoptimised that barely any computation gets carried out while your code is waiting on memory/disk/network.
"I can't do anything because my program is I/O-bound" is more of an excuse / mental justification of why your program is slow instead of an honest reason for so.
> If you're I/O bound, that either means the problem doesn't require much computation - which is possible but fairly rare
This is backwards. I bet that by count, many more programs are written in domains where they're necessarily IO bound than the inverse. Anything that uses the network for its core functionality, anything reliant on a datasource whose aggregate contents are O(memory)+ size, or anything reliant on slow peripherals (lots of embedded software) are in this class.
Scientific simulations, HFT algorithms, video games, LLMs, etc.--the stuff in the other class--aren't inconsequential, but they're dwarfed in number by the class of software that spends 99+% of its time waiting for IO. Hell, entire programming languages (node.js) have been created in response to that proportion.
ETL processes are heavily I/O bound, especially when you're trying to shuttle data from one enterprise system to another enterprise system. It's also common when the culture of data exchange from the regulator all the way down to the companies doing the actual work is batch processing where large amounts of data are shared once a day as opposed to real time. Excel spreadsheets are the norm, not the exception. Requests for data to be sent over via XML or JSON are mainly because my employer wanted to make it easier to process the data ourselves, but the regulators actually expect spreadsheets.
Most of the stuff I work on is almost exclusively network I/O bound. I wouldn't say it's a _result_ of bad engineering practices, though. One group decided on a particular system that's also public-facing, and the group I actually support prefers a more internal-facing system. It also doesn't help that the budgets for both projects are completely separate and firewalled from each other by law. Growth opportunities don't apply here because I deal with a captive market with legally-forced customers.
No, I don't think the way you're characterizing this is accurate. I/O is inherently very slow compared to computation. And many programs genuinely don't have any useful computation to do while waiting for I/O - because the result of that I/O operation contains the information needed for the program to even make its next decision.
Such programs are not necessarily impossible to optimize. One common optimization is to use an event loop, allowing just a few threads to handle thousands of concurrent operations. Because while a thread is waiting for I/O in one request or unit of work, in the meantime it moves on to work on processing another request/unit. Another common optimization is batching/grouping of I/O calls.
> I/O is inherently very slow compared to computation.
Not anymore, no. Your SSD, before any caching, does gigabytes per second of sequential reads. For any bytewise processing, except the most trivial of tasks, you’ll struggle to get above a few hundred megabytes per second with scalar (native) code. To actually keep up with a modern SSD, you’ll virtually always have to hand-write SIMD loops, minimize the number of syscalls with tools like io_uring, or possibly be smart about distributing tasks across cores without ruining the access pattern.
For instance, simdjson is famously fast but I don’t believe it can keep up with say a high-end PCIe Gen 4 SSD like a Samsung 990 PRO, let alone the latest-and-greatest (and, literally, hottest) Gen 5 stuff. And I know of no Unicode normalizer that would be able to do a gigabyte per second on general inputs (not ASCII, not Latin-1) simply because the latency for dependent lookup table accesses is absolute murder.
Your analysis is correct if and only if the data is on the same machine as the calculations. If the data comes from another machine, it comes at network speed. If it comes from the internet, it comes at non-local network speed. That's very different from SSD speed.
> I/O is inherently very slow compared to computation
This isn't really true anymore. IO has bad latency, but modern SSD bandwidth is ~5-15GB/s. If your program is IO latency bound and processing less that 5GB/s you aren't IO bound, you aren't hiding your latency well enough.
It’s on YouTube, pull the transcript and format it with an LLM. If there’s no transcript, there’s Whisper. All of this can be done with local models too.
Audio transcription is nowhere near the same as a properly written piece of text. Videos are simply a different format, and thus a lot of the communication is held by the fact that there's visual cues to go by. If one were to write an article about the same, you'd use much denser language as well as references/links around the topic. Maybe even restructure the whole thing to communicate it better through text.
It is something you can maybe whip up with the help of an LLM but then you're left wondering whether it is actually communicating what the video intended to begin with, as you wouldn't be as knowledgeable about the topic, one wouldn't be able to spot the parts where an LLM interpreted something horribly and thus reaching the wrong conclusions.
For all of the above, I do not consider audio transcript to be of any help when one prefers written content. It is useful for translation or people with hearing issues, but both of those receive help of the visual parts to convey the topic.
It's worth the listen if you're even mildly interested in the history of computer science. He's a great presenter. I guess at some point you do have to prioritize how to spend your time, though.
It is quite long, but I thought it was worth it if you can find the time. Short of that I think just reading the Knuth article the quote is from might bring similar insights.
The last hour is a Q&A. I don't know if that changes your perception of how watchable it is.
It's also a little fluffy. It's basically the history of how the book Structured Programming came about, because it's that book that essentially caused Knuth to write an article that contained the quote.
A lot of it is fairly interesting but it also highlights a big problem I do have with Muratori. He generalizes from self. He didn't fully understand or appreciate the quote about premature optimization, so no one did. And even though he's been told what it actually means, and was probably pointed to the full quote, he would stick on his interpretation. It's likely someone he views as "higher", like Jon Blow, challenged him and then he took the time to actually look at it.
"He didn't fully understand or appreciate the quote about premature optimization, so no one did."
He ain't wrong though.... most people definitely don't get the jive for sure.
Knuth was talking about a massively different kind of optimisation than what we do today yet people keep continuing to parrot the 97% figure uncritically like as it was some gospel. [0]
If you want a spoilery TLDR:
It's more about the journey. He tracks down the origin, finds the support, finds the support flawed, and leaves you to your own conclusion rather than make a new flawed one.
The basic idea is that the origin assumes a highly critical inner hot loop, don't assume where it is, and optimize there.
There's some other time spent saying this justifies slower abstractions for maintainability elsewhere.
Another point I liked was that there was, apparently, an influential book called Structured Programming, whose content was so universally agreed upon, that all programming became Structured Programming. Nobody needs the book anymore.
It's not the focus of the talk and so it's hard to tell if Casey understands (the choice to separate the words GO TO in several places suggests he does) but the `goto` keyword you've seen in several modern languages is not the problematic "GO TO statement", it's a de-fanged remnant, the toy poodle to GO TO's wolf pack.
The actual GO TO complained of is, like the jump instruction in machine code, just entirely unbothered by context. Want to go from the middle of this code about employee payroll processing to mid-way through initializing a weather simulation? No problem. Well. No problem for the machine, for a human programmer it's a complete nightmare. Actually that's putting it mildly, nightmares have more structure. You cannot do anything like that with for example C's goto.
You compiler can, and in a few cases (that's what the discussion about the tail-call optimisation is about for example) it will, but the program you wrote doesn't do this and so you don't have to try to keep the whole program in your head.
So in that sense GOTO died out with, maybe BASICs? I think the BASICs tend to have that wolf nature GOTO feature, but nothing modern has it.
So, I have never seen anyone actually use GOTOs, so maybe a lot of the stigma comes from excessive, poor usage or something.
However, having written a good chunk of ASM in my life. I don't think jumps or branches are really that hard to follow. Jumps/Branches and GOTOs specify the next location. It is not as though one has to guess where.
It's not the arrow, it's the archer that is the problem.
Never as in, in BASIC, or you've never seen goto in C? The de-fanged C "goto" is all over the place in Linux and in similar close-to-metal C software. C does not (yet, likely C2Y will fix this) have labelled break, so goto is used to say "I am inside a mess of nested loops, we're done, end the loops" and as a catch-all failure handler in some codebases.
My guess is that your ASM is inflected by structured programming everywhere. Yes, unlike COME FROM we can see where we're going next with GO TO but if you go back 60+ years there is some scary code that even today's optimizers probably wouldn't emit because it's too crazy. Suppose calculate-total-fuel ends with three CPU instructions which copy register F into register H then add register C to it and multiply the sum by four. Over in locate-horizon it so happens we need to add two things together and multiply them by four and we could do that last. So, if those two things were in registers F and C we could just GO TO that last part of calculate-total-fuel.
You're correct that we don't need to "guess where" it goes, but good luck understanding why the program works when it's like this, let alone the ordinary maintenance work of making small modifications.
> My guess is that your ASM is inflected by structured programming everywhere.
I think you're probably right. To expand on this:
In asm, you can have things that are clearly functions. You have a stack discipline going in and out of them. They end with stack cleanup, then a RET or some such, which pops the return address off of the stack and jumps to it. Within that function, you have JMP instructions (or whatever) that move around within the function. You may also call other functions, by pushing variables on the stack, and then calling JSR or whatever to push the program counter on the stack and jump, and when those functions return, you'll be right where you were in this function. That's all sane, and it's "structured assembly".
Non-structured assembly would be like the example in your second paragraph. You're in one function, and you JMP (not JSR) into the interior of a second function. Or, you simply don't have functions, just labels that you jump around do. That's not structured, and not sane.
No, they're not sarcastic. It was an interesting point; if an idea succeeds well enough people just do it and make it "common sense". It's a point in the talk.
I get the auto-transcript with yt-dlp then ask a cheap LLM like DeepSeek to clean it up.
Though lately I've been uploading the audio to AssemblyAI, I somehow still haven't used up my credits after several years lol
At one point I built a system that would summarize the transcript and I'd be able to ask questions about it, but Gemini can do that natively now so I usually just use that.
I took notice when he breaks down the ethics of gen ai as it pertains to online theft, that AI broke the natural order of putting content online where you'd trade exposure for use https://youtu.be/bjO-s4rNPlY?si=uAtBTb6V_Sw1puBU&t=2244
I believe this is referencing a meme on the primeagen's standup podcast, where Casey is referred to as legendary while he feels undeserving of this title.
Titles aside, his talk is really insightful and it is super interesting to do a deep dive on these old computer/programming topics as the modern concepts were being discovered
Hah, I wasn't aware of that. I was sort of referring to what seems to be not such a meme that jblow is always introduced like that. It just seems weird, even if true...
Yeah Ive been meaning to watch that talk - I love listening to pretty much anything Casey says/does. He's extremely thoughtful and fair.
Thank you for bringing a term to a thing that has been bugging me: people who purport to be devs but spend most of their time on podcasts and blogs talking and writing about being devs, rather than just being devs.
Are they? Casey programmed the walk monster for Jon's game, a brute force tester to make sure you could always walk both ways and never get stuck, instead of building the walking system using a nav mesh. There was one place you could walk where you weren't supposed to, despite the walk monster, that would have been obvious if it was a nav mesh.
Doesn't mean you can't learn a lot from the attempt. Technical craftsmanship and project management skill are two completely separate skills. Definitely don't learn project management from Casey.
Sure, but the biggest reason to direct anybody to these videos is that they should actually make that video game they want to make, not just think about it - and so the fact that Handmade Hero just trails off and doesn't end up producing a finished video game is a problem.
The craftsmanship is dubious. I think it's a problem that people assume Casey knows what he's doing when so often he's like "We're doing it live" and Casey's hand coded solution is pretty bad whereas the thing which came in the box is very good. Casey understands Casey's version, and that's an upside if you are Casey but you are not. If the result was a finished game then hey, whatever, the game was finished and that matters. But that part didn't happen either, so then it's just like watching Tsoding or something.
John Carmack is a legendary game programmer. If I can identify a superior algorithmic approach to something in Doom, does that disqualify Carmack's credentials?
First I think I'd want to see you identify a superior algorithm known when Doom shipped and then we can have that discussion about who's credentials should be disqualified.
I never made claims to be any kind of game programmer, but sure.
Fine, as I recall, Doom data assets are searched linearly whenever they need to pull new data. So if you switch to the chainsaw, and need to rev, the game does a linear scan of all graphics, maps, and sounds looking for the vroom noise.
There are many different data structures that could perform this lookup faster. These have been known since the earliest days of computing.
Does it matter? No. Engineering is all about trade-offs. A linear scan was obviously fast enough and simple to implement.
Similarly, if walk monster manually trolls the map, that does not say anything about Casey in isolation. I believe all of Blow's games use a custom game engine, so unless a nav mesh system was already implemented, that was going to take additional work. The walk monster may have been better bang-for-buck.
Casey is professionally best known for his work at RAD game tools (a highly successful middleware provider for game development back in the day), not games he himself developed.
What he is most highly regarded for is his teaching, particularly the Handmade Hero series on YouTube, which various programmers directly attribute as being responsible for their own professional success to a large extent.
He did work on the video games Dungeon Siege and The Witness afaik, but that's not what he is known for.
He also made one of the fastest terminal emulators (refterm), purely out of spite, to show Microsoft they were full of shit in their answer to an issue he posted to the Windows Terminal github. This directly led to performance improvements in Windows Terminal.
He mostly evangelizes non-pessimal software. He advocates that your program should only be a small factor like 3x slower than the hypothetical optimum - instead of 10000x slower as today's software often is. He compared Visual C++ 6's debugger on hardware from the time to current Visual Studio's debugger on current hardware, and found the former much faster when performing the exact same tasks on the exact same project file and code files.
I have recently gotten a 45x on a shitty vibe coded Python project by using mypyc (2x), replacing datastructures with more CPU-friendly alternatives (15x) and using some smarter heuristics rather than just brute force on top of that (45x total).
And I never left Python, the slowest language around.
AI is only going to make the problem worse unless you know what to look for.
He works in the engine/tool side of things. He worked on some widely used libraries, mainly Bink 2 (video codec) and Granny 3D (3D animation) used in a ton of shipped games.
tl;dr the saying is that "premature optimisation is the root of all evil", and Casey burrows into contemporary data to show that really although the claim was 3% of the code takes up 90% of the runtime even then it was more likely 4% takes 50%.
The best thing you could take away from this lecture is something a reasonable person should take away from the original "root of all evil" saying anyway. Measure. Measure. Measure. If you aren't measuring that's not optimization it's masturbation.
Ironically in the process of measuring for a third time yesterday I tripped a bug in Bill's language for which I opened an issue. This is not the goal of measuring three times but merely a happy accident.
Along the way Casey discovers (?) that Structured Programming means just what we today call programming†, that software was a lot smaller in the days when 4096 bytes of RAM was a good entry level option and that loads of these famous people from 1970s computer science knew each other.
† And knowing about this is one reason the Structured Concurrency people want that everywhere. Very possibly there's a future where it seems silly that people once wrote programs which did not use structured concurrency.
Casey generally, across all his material, advocates for non-pessimisation. Measurement takes too long to apply it to your entire program. He advocates for thinking about how much work the computer should actually have to do, then not making it do much more than that, at all times. This means avoiding serial dependency chains on the network, and huge towers of abstractions, and redundant work. He allows for writing lazy slow code as an intentional tradeoff that you may revise later if it becomes a bottleneck. He does not allow for inherently slow architecture.
Sure, but there are performance issues no profiler will catch in a straightforward flamegraph reading.
Some examples:
1. Your hottest loop is spilling registers which only shows up as non-local cache thrashing (i.e. some other random code becomes slow) or randomly slow instructions i.e. "why is this xorps to initialise this float suddenly slow" due to pipeline stalls
2. Your code stops fitting into cache due to the code bloat, there's no one method which is slow, everything is slowed down by a percentage factor
3. A lot of useless work being done like temporary strings being copied everywhere
4. Your code is "I/O bound" because all the data you're accessing is scattered all across memory, leading to completely predictable TLB stalls
It's very easy to make a large program, quite a bit harder to make a small one...
I'm on AMD and uProf isn't that good (it often fails to match PDB to source, it's slow / crashy, stuff like that) so I mostly use Superluminal myself (has godly UX) and only drop down to uProf when I have a suspicion and want to gather hardware stats.
In other videos Casey argues against the profile–fix–repeat workflow (I’m not saying that you necessarily meant this by measuring), instead arguing for estimating the theoretical maximum, then trying to get close enough to it. His argument is that the former might push you towards a local minimum without realising that you could do much better
When "Estimating the theoretical maximum" you're going to bring your assumptions to that estimate and so you're equally likely to get stuck in some local optimal point because you didn't see what else was possible, so that doesn't persuade me much. Worse, now it's a local optimal which might be entirely in your head.
There is some space for this "Estimate the theoretical maximum" as a high level insight, if the software needs to do a thing which your estimate says is impossible that's important to address up front but I think for practical software engineering you're much more likely to need profile-fix-repeat
Actually measuring is crucial and an estimate is not a measurement. I think one of the reasons many in the Handmade Community don't like measuring is that it too often ruins their "estimate" of how the crap they've written is achieving a "theoretical maximum" in favour of a boring fact that it's much worse than alternatives.
The problem is knowing what to measure. There's another saying
"When a measure becomes a target, it ceases to be a good measure."
As an example from memory, there was a game dev company that celebrated they had maxed out the cores on the PS3. That didn't mean anything though, anyone can max out the cores by filing them with bad code. But hey, their "measurement" told them they had maxed out the machine
This can be a problem, but much less so because so often we're doing "easy mode" where we don't need a proxy. The "it ceases to be a good measure" is because you're measuring a proxy. You wanted to deliver happiness, you measured wealth because it was easier to measure but seemed correlated and now you've got rich miserable people, oops. But software engineers can often measure the actual thing they want to improve directly, not a proxy and so it cannot cease to be a good measure.
With a few caveats, though they do tend to get fixed over time. For example, frame rate. Higher frame rate is better, the end, right? Wellll...
Latency and hitching are annoying to measure, so for a long time, they were pretty much ignored. That has improved and hopefully will continue to improve. There is the bufferbloat initiative, gaming magazines take frame time histograms now, input devices and screens commonly have their latency measured. But latency is still under-measured - for GPUs / GPU drivers, for all software, for games and particularly for websites.
I think Casey is currently the most informed person to make a series of books or articles summarizing the history of SW Engineering, all the lessons learned and forgotten, and all the good stuff that was published and still hasn't gained traction in the practice
You think there aren't other people equally or more qualified? I mean Casey may be the most visible person with those qualifications - but lack of visibility doesn't imply lack of competence, plenty of brilliant people work in the background..
Feel free to point to anyone doing these deep dives that specifically tackle the lost knowledge of the early decades of computer science and the ideas that have not yet come to fruition despite being extremely old
> Feel free to point to anyone doing these deep dives
Most of this kind of content comes off as relevant/topical but mindless entertainment. This is not a "deep dive" of anything that I could practically apply in my daily work. It feels good to think this content might add value and then to subsequently consume it, but it's effectively junk food.
I used to spend a lot of time watching crap like lex and primagen before it became obvious that these kinds of information channels are not meaningfully enriching despite sounding like they should be to a casual observer.
As much as I like to watch Casey, Prime, etc. on YouTube, they are not going to teach anybody anything. They are just entertainment for software people with a certain viewpoint.
My one data point is that Casey’s handmade hero series, and his immediate mode gui video from way back in 2005, are what introduced me and some friends to an entirely new way to build graphical user interfaces, and I do feel a great appreciation for that. Guys like Ryan Fleury (of RadDbg) and Vjekoslav Krajacic of Filepilot similarly credit Casey with their “radicalization” :) To your point though, it was less about specific technical knowledge conveyed by these videos and more about him evangelizing a high level approach that many folks otherwise wouldn’t have considered
I want more things like that old hand made hero to exist in the world, but, from an interview with Casey himself, it doesn’t pay well enough to justify. Someone with the chops to put that together is in a position to be doing something much better incentivized.
You don’t know Computer Enhance then
You are making many assumptions. First of all, he is doing the deep dives by researching all those articles, and of course the resulting presentation will be very superficial. The point is exactly to publish a book to allow the reader to partake in the deep dive. Secondly, nobody said anything about applying that stuff to your daily work.
this person, he is very very smart, maybe the smartest.
this person, of course their presentations is super simple.
you - i suspect you are sophist and like other sophists
Thank you for creating an account just for making stuff up nobody has said or even hinted at
> that I could practically apply in my daily work
When has History of any kind ever held practical value?
Who is also a working engineer (and programmer of the highest caliber) who is in touch with the zeitgeist of engineering.
Me, I do them.
Kevlin Henney
I feel like the comments that are giving you pushback haven't actually listened to Casey before. I agree with you, while opinionated his perspective is very pragmatic and misunderstood. He's done such a great job giving his experience out to the broader world too.
He's entertaining and well spoken. Love it. Not sure the book medium would be the best avenue. My own sweet spot is presentations, live interviews and meetings. I can't write for sh*t, even if I tried. I know.
I'd put Kevlin Henney up there with him. His talk on 1968 alone is pretty amazing
Ehh I love Casey and have learned a ton by watching how he thinks about things in his handmade hero series, but he is fairly narrow minded in his views of dev. Not that theres anything wrong with that, for the kinds of dev he does his approach is very good. But its not generalizable.
How is this relevant? I didn't say he should make a book about his teachings about SWE, I said he could compile a comprehensive review of the foundational literature of compsci
Well the comment was about the history of SWE, if you want a comprehensive history you don't focus on the lived memory of one single individual.
Where have I said I want his lived memories?
Absolutely.
Like I agree with most everything he says and I like to optimise my own software, but for my day to day jobs and contracts it's simply not feasible.
That level of performance and rigor is not what is demanded nor paid or appreciated.
E.g. in Enterprise circles it's still OOP from top to bottom.
Mixed with a tad more functional style due to varying adoption of that paradigm in the languages used by enterprise.
But low level optimizations are not interesting to these customers at all, even though they could help them with some problems.
> not what is demanded nor paid or appreciated.
I suppose it is a matter of what one works on, but that has been consistent in my career.
I have seen many be rewarded for choosing what I would deem intentional, gross negligence. But hey, those devs get work finished faster, and that's all that matters to the non-technical folks.
Terrific presentation. But I have a comment:
His dismissal of the argument Knuth makes regarding the hot loops could have been explored a bit better. I found it weird he didn’t mention the difference of types of programs of then vs now. Even today, in scientific code it is still absolutely the case a lot of the time that a huge chunk of the runtime comes from a single very very hot loop. It might be hidden in a library, but it’s there. Instead he focuses only on “program size”. Knuth samples where very small FORTRAN programs (compared to today’s standards). Today’s program are bigger but the fundamental number crunching primitive of “let’s compute stuff in a loop” remains. It’s just buried under a pile of extra cruft (data loading, parallelism, dispatching etc).
Now we just deal with a lot more programs that are of a whole different class compared to what they where doing with computers in the 70s. We have much more I/O involved. And hot loops don’t like being I/O bound.
> I found it weird he didn’t mention the difference of types of programs of then vs now.
iirc, in the talk casey in fact does goes on about how he tried to find examples, but couldn't. in the q&a, he was also asked about this further.
Being I/O bound is usually a result of bad engineering practices though. If you're I/O bound, that either means the problem doesn't require much computation - which is possible but fairly rare, or more likely that your code is so unoptimised that barely any computation gets carried out while your code is waiting on memory/disk/network.
"I can't do anything because my program is I/O-bound" is more of an excuse / mental justification of why your program is slow instead of an honest reason for so.
> If you're I/O bound, that either means the problem doesn't require much computation - which is possible but fairly rare
This is backwards. I bet that by count, many more programs are written in domains where they're necessarily IO bound than the inverse. Anything that uses the network for its core functionality, anything reliant on a datasource whose aggregate contents are O(memory)+ size, or anything reliant on slow peripherals (lots of embedded software) are in this class.
Scientific simulations, HFT algorithms, video games, LLMs, etc.--the stuff in the other class--aren't inconsequential, but they're dwarfed in number by the class of software that spends 99+% of its time waiting for IO. Hell, entire programming languages (node.js) have been created in response to that proportion.
ETL processes are heavily I/O bound, especially when you're trying to shuttle data from one enterprise system to another enterprise system. It's also common when the culture of data exchange from the regulator all the way down to the companies doing the actual work is batch processing where large amounts of data are shared once a day as opposed to real time. Excel spreadsheets are the norm, not the exception. Requests for data to be sent over via XML or JSON are mainly because my employer wanted to make it easier to process the data ourselves, but the regulators actually expect spreadsheets.
Most of the stuff I work on is almost exclusively network I/O bound. I wouldn't say it's a _result_ of bad engineering practices, though. One group decided on a particular system that's also public-facing, and the group I actually support prefers a more internal-facing system. It also doesn't help that the budgets for both projects are completely separate and firewalled from each other by law. Growth opportunities don't apply here because I deal with a captive market with legally-forced customers.
No, I don't think the way you're characterizing this is accurate. I/O is inherently very slow compared to computation. And many programs genuinely don't have any useful computation to do while waiting for I/O - because the result of that I/O operation contains the information needed for the program to even make its next decision.
Such programs are not necessarily impossible to optimize. One common optimization is to use an event loop, allowing just a few threads to handle thousands of concurrent operations. Because while a thread is waiting for I/O in one request or unit of work, in the meantime it moves on to work on processing another request/unit. Another common optimization is batching/grouping of I/O calls.
> I/O is inherently very slow compared to computation.
Not anymore, no. Your SSD, before any caching, does gigabytes per second of sequential reads. For any bytewise processing, except the most trivial of tasks, you’ll struggle to get above a few hundred megabytes per second with scalar (native) code. To actually keep up with a modern SSD, you’ll virtually always have to hand-write SIMD loops, minimize the number of syscalls with tools like io_uring, or possibly be smart about distributing tasks across cores without ruining the access pattern.
For instance, simdjson is famously fast but I don’t believe it can keep up with say a high-end PCIe Gen 4 SSD like a Samsung 990 PRO, let alone the latest-and-greatest (and, literally, hottest) Gen 5 stuff. And I know of no Unicode normalizer that would be able to do a gigabyte per second on general inputs (not ASCII, not Latin-1) simply because the latency for dependent lookup table accesses is absolute murder.
Your analysis is correct if and only if the data is on the same machine as the calculations. If the data comes from another machine, it comes at network speed. If it comes from the internet, it comes at non-local network speed. That's very different from SSD speed.
Classic latency vs throughput problem. 10s of GB/s of disk bandwidth doesn't help when my problem is serialized durable writes.
> I/O is inherently very slow compared to computation
This isn't really true anymore. IO has bad latency, but modern SSD bandwidth is ~5-15GB/s. If your program is IO latency bound and processing less that 5GB/s you aren't IO bound, you aren't hiding your latency well enough.
He has addressed the "hotspot" notion in the past, one example being part of https://youtu.be/x2EOOJg8FkA
Personally I'm quite sure this is super interesting, but I don't really have 3 hours to listen to this, even 1.5h at 2x speed is too much.
I would very much prefer something written down, so I could absorb this at my own pace. I know, gift horse, but still.
Thanks to modern playback technology, you can pause it and resume play later at your convenience.
Maybe go for a long drive? Long walk? Whatever floats your boat.
I used to do manual labor and I would work my way through like eight hours of audiobooks per day.
I've noticed after I've switched to audiobooks that my retention is horrible compared to reading. For actual complex topics, it's even worse.
This is one talk that is worth the 3 hours (the talk is 2 hours, plus a qa with none other than Ginger Bill of Odin fame)!
It’s on YouTube, pull the transcript and format it with an LLM. If there’s no transcript, there’s Whisper. All of this can be done with local models too.
Audio transcription is nowhere near the same as a properly written piece of text. Videos are simply a different format, and thus a lot of the communication is held by the fact that there's visual cues to go by. If one were to write an article about the same, you'd use much denser language as well as references/links around the topic. Maybe even restructure the whole thing to communicate it better through text.
It is something you can maybe whip up with the help of an LLM but then you're left wondering whether it is actually communicating what the video intended to begin with, as you wouldn't be as knowledgeable about the topic, one wouldn't be able to spot the parts where an LLM interpreted something horribly and thus reaching the wrong conclusions.
For all of the above, I do not consider audio transcript to be of any help when one prefers written content. It is useful for translation or people with hearing issues, but both of those receive help of the visual parts to convey the topic.
It's worth the listen if you're even mildly interested in the history of computer science. He's a great presenter. I guess at some point you do have to prioritize how to spend your time, though.
It is quite long, but I thought it was worth it if you can find the time. Short of that I think just reading the Knuth article the quote is from might bring similar insights.
The last hour is a Q&A. I don't know if that changes your perception of how watchable it is.
It's also a little fluffy. It's basically the history of how the book Structured Programming came about, because it's that book that essentially caused Knuth to write an article that contained the quote.
A lot of it is fairly interesting but it also highlights a big problem I do have with Muratori. He generalizes from self. He didn't fully understand or appreciate the quote about premature optimization, so no one did. And even though he's been told what it actually means, and was probably pointed to the full quote, he would stick on his interpretation. It's likely someone he views as "higher", like Jon Blow, challenged him and then he took the time to actually look at it.
"He didn't fully understand or appreciate the quote about premature optimization, so no one did."
He ain't wrong though.... most people definitely don't get the jive for sure. Knuth was talking about a massively different kind of optimisation than what we do today yet people keep continuing to parrot the 97% figure uncritically like as it was some gospel. [0]
[0] http://www.joshbarczak.com/blog/?p=580
If you want a spoilery TLDR: It's more about the journey. He tracks down the origin, finds the support, finds the support flawed, and leaves you to your own conclusion rather than make a new flawed one.
The basic idea is that the origin assumes a highly critical inner hot loop, don't assume where it is, and optimize there.
There's some other time spent saying this justifies slower abstractions for maintainability elsewhere.
Another point I liked was that there was, apparently, an influential book called Structured Programming, whose content was so universally agreed upon, that all programming became Structured Programming. Nobody needs the book anymore.
Hard to tell if sarcastic, but anyway.
I think the GOTOers just died out.
Some day null, statements (rather than expressions) and side-effects will have always been wrong.
It's not the focus of the talk and so it's hard to tell if Casey understands (the choice to separate the words GO TO in several places suggests he does) but the `goto` keyword you've seen in several modern languages is not the problematic "GO TO statement", it's a de-fanged remnant, the toy poodle to GO TO's wolf pack.
The actual GO TO complained of is, like the jump instruction in machine code, just entirely unbothered by context. Want to go from the middle of this code about employee payroll processing to mid-way through initializing a weather simulation? No problem. Well. No problem for the machine, for a human programmer it's a complete nightmare. Actually that's putting it mildly, nightmares have more structure. You cannot do anything like that with for example C's goto.
You compiler can, and in a few cases (that's what the discussion about the tail-call optimisation is about for example) it will, but the program you wrote doesn't do this and so you don't have to try to keep the whole program in your head.
So in that sense GOTO died out with, maybe BASICs? I think the BASICs tend to have that wolf nature GOTO feature, but nothing modern has it.
So, I have never seen anyone actually use GOTOs, so maybe a lot of the stigma comes from excessive, poor usage or something.
However, having written a good chunk of ASM in my life. I don't think jumps or branches are really that hard to follow. Jumps/Branches and GOTOs specify the next location. It is not as though one has to guess where.
It's not the arrow, it's the archer that is the problem.
Never as in, in BASIC, or you've never seen goto in C? The de-fanged C "goto" is all over the place in Linux and in similar close-to-metal C software. C does not (yet, likely C2Y will fix this) have labelled break, so goto is used to say "I am inside a mess of nested loops, we're done, end the loops" and as a catch-all failure handler in some codebases.
My guess is that your ASM is inflected by structured programming everywhere. Yes, unlike COME FROM we can see where we're going next with GO TO but if you go back 60+ years there is some scary code that even today's optimizers probably wouldn't emit because it's too crazy. Suppose calculate-total-fuel ends with three CPU instructions which copy register F into register H then add register C to it and multiply the sum by four. Over in locate-horizon it so happens we need to add two things together and multiply them by four and we could do that last. So, if those two things were in registers F and C we could just GO TO that last part of calculate-total-fuel.
You're correct that we don't need to "guess where" it goes, but good luck understanding why the program works when it's like this, let alone the ordinary maintenance work of making small modifications.
> My guess is that your ASM is inflected by structured programming everywhere.
I think you're probably right. To expand on this:
In asm, you can have things that are clearly functions. You have a stack discipline going in and out of them. They end with stack cleanup, then a RET or some such, which pops the return address off of the stack and jumps to it. Within that function, you have JMP instructions (or whatever) that move around within the function. You may also call other functions, by pushing variables on the stack, and then calling JSR or whatever to push the program counter on the stack and jump, and when those functions return, you'll be right where you were in this function. That's all sane, and it's "structured assembly".
Non-structured assembly would be like the example in your second paragraph. You're in one function, and you JMP (not JSR) into the interior of a second function. Or, you simply don't have functions, just labels that you jump around do. That's not structured, and not sane.
Windows batch, no?
COMMAND.COM in its batch mode? Kinda, sorta. The interpreter can AIUI "call" other files but the "goto" is restricted to the same file.
So we can jump over a variable declaration which is pretty confusing, but we can't jump into unrelated code.
Also while COMMAND.COM can't be as old as BASIC it must be pretty old.
No, they're not sarcastic. It was an interesting point; if an idea succeeds well enough people just do it and make it "common sense". It's a point in the talk.
There is a 45 min version at Primeagen's "The Standup"
Having watched both, they cover completey different topics.
Audio transcription has been around for a while, you could solve this problem for yourself quite easily.
I get the auto-transcript with yt-dlp then ask a cheap LLM like DeepSeek to clean it up.
Though lately I've been uploading the audio to AssemblyAI, I somehow still haven't used up my credits after several years lol
At one point I built a system that would summarize the transcript and I'd be able to ask questions about it, but Gemini can do that natively now so I usually just use that.
I asked my clanker to summarize it for me:
https://rentry.co/2ttr46r9
I took notice when he breaks down the ethics of gen ai as it pertains to online theft, that AI broke the natural order of putting content online where you'd trade exposure for use https://youtu.be/bjO-s4rNPlY?si=uAtBTb6V_Sw1puBU&t=2244
THE LEGENDARY GAME PROGRAMMER
Isn't that Jonathan blow?
(to be clear, I'm a big fan of Casey)
I believe this is referencing a meme on the primeagen's standup podcast, where Casey is referred to as legendary while he feels undeserving of this title.
Titles aside, his talk is really insightful and it is super interesting to do a deep dive on these old computer/programming topics as the modern concepts were being discovered
Hah, I wasn't aware of that. I was sort of referring to what seems to be not such a meme that jblow is always introduced like that. It just seems weird, even if true...
Yeah Ive been meaning to watch that talk - I love listening to pretty much anything Casey says/does. He's extremely thoughtful and fair.
> I believe this is referencing a meme on the primeagen's standup podcast
Exactly!
I see it 3-4 times already in this comment section, so must conclude that modern programmer culture = parroting YouTube/X devfluencer catchphrases
Yes, programmers are exactly like everyone else in this way.
> devfluencer
Thank you for bringing a term to a thing that has been bugging me: people who purport to be devs but spend most of their time on podcasts and blogs talking and writing about being devs, rather than just being devs.
They are both legendary game programmers
Are they? Casey programmed the walk monster for Jon's game, a brute force tester to make sure you could always walk both ways and never get stuck, instead of building the walking system using a nav mesh. There was one place you could walk where you weren't supposed to, despite the walk monster, that would have been obvious if it was a nav mesh.
He’s known for his Handmade hero series (unfortunately archived now) where he developed his game over 500+ episodes: https://hero.handmade.network/
I’m sure you can find bugs in the work of any game dev you would consider legendary, game implementation is generally very messy
660+ and the game was never finished
Doesn't mean you can't learn a lot from the attempt. Technical craftsmanship and project management skill are two completely separate skills. Definitely don't learn project management from Casey.
Sure, but the biggest reason to direct anybody to these videos is that they should actually make that video game they want to make, not just think about it - and so the fact that Handmade Hero just trails off and doesn't end up producing a finished video game is a problem.
The craftsmanship is dubious. I think it's a problem that people assume Casey knows what he's doing when so often he's like "We're doing it live" and Casey's hand coded solution is pretty bad whereas the thing which came in the box is very good. Casey understands Casey's version, and that's an upside if you are Casey but you are not. If the result was a finished game then hey, whatever, the game was finished and that matters. But that part didn't happen either, so then it's just like watching Tsoding or something.
John Carmack is a legendary game programmer. If I can identify a superior algorithmic approach to something in Doom, does that disqualify Carmack's credentials?
First I think I'd want to see you identify a superior algorithm known when Doom shipped and then we can have that discussion about who's credentials should be disqualified.
I never made claims to be any kind of game programmer, but sure.
Fine, as I recall, Doom data assets are searched linearly whenever they need to pull new data. So if you switch to the chainsaw, and need to rev, the game does a linear scan of all graphics, maps, and sounds looking for the vroom noise.
There are many different data structures that could perform this lookup faster. These have been known since the earliest days of computing.
Does it matter? No. Engineering is all about trade-offs. A linear scan was obviously fast enough and simple to implement.
Similarly, if walk monster manually trolls the map, that does not say anything about Casey in isolation. I believe all of Blow's games use a custom game engine, so unless a nav mesh system was already implemented, that was going to take additional work. The walk monster may have been better bang-for-buck.
I think you’re both saying the same thing
No, that would be John Carmack
Death-frightening scion capable of seeing beyond the illusionary world before our eyes John Carmack?
He is just legendary when it comes to game programming
What games has he shipped?
The legendary games programmer thing is a meme.
Casey is professionally best known for his work at RAD game tools (a highly successful middleware provider for game development back in the day), not games he himself developed.
What he is most highly regarded for is his teaching, particularly the Handmade Hero series on YouTube, which various programmers directly attribute as being responsible for their own professional success to a large extent.
He did work on the video games Dungeon Siege and The Witness afaik, but that's not what he is known for.
He also made one of the fastest terminal emulators (refterm), purely out of spite, to show Microsoft they were full of shit in their answer to an issue he posted to the Windows Terminal github. This directly led to performance improvements in Windows Terminal.
He mostly evangelizes non-pessimal software. He advocates that your program should only be a small factor like 3x slower than the hypothetical optimum - instead of 10000x slower as today's software often is. He compared Visual C++ 6's debugger on hardware from the time to current Visual Studio's debugger on current hardware, and found the former much faster when performing the exact same tasks on the exact same project file and code files.
I have recently gotten a 45x on a shitty vibe coded Python project by using mypyc (2x), replacing datastructures with more CPU-friendly alternatives (15x) and using some smarter heuristics rather than just brute force on top of that (45x total).
And I never left Python, the slowest language around.
AI is only going to make the problem worse unless you know what to look for.
Very interesting that the Christmas Disk still has not been released.
He works in the engine/tool side of things. He worked on some widely used libraries, mainly Bink 2 (video codec) and Granny 3D (3D animation) used in a ton of shipped games.
He did middleware at RAD, home of a lot of good stuff, and worked directly on at least The Witness
It's a common situation for many quotes of such kind. Taken out of context they loose or completely change their initial meaning.
I enjoyed this talk. It’s long, but it’s interesting and goes into a lot of “lost” history.
Will have to give this a watch after the kids go to bed. I like a lot of Casey’s views even if I don’t agree with them.
Do not guess. Measure, but only measure the bottlenecks that threaten the business
It's premature optimization, according to the video description.
Legend
tl;dr the saying is that "premature optimisation is the root of all evil", and Casey burrows into contemporary data to show that really although the claim was 3% of the code takes up 90% of the runtime even then it was more likely 4% takes 50%.
The best thing you could take away from this lecture is something a reasonable person should take away from the original "root of all evil" saying anyway. Measure. Measure. Measure. If you aren't measuring that's not optimization it's masturbation.
Ironically in the process of measuring for a third time yesterday I tripped a bug in Bill's language for which I opened an issue. This is not the goal of measuring three times but merely a happy accident.
Along the way Casey discovers (?) that Structured Programming means just what we today call programming†, that software was a lot smaller in the days when 4096 bytes of RAM was a good entry level option and that loads of these famous people from 1970s computer science knew each other.
† And knowing about this is one reason the Structured Concurrency people want that everywhere. Very possibly there's a future where it seems silly that people once wrote programs which did not use structured concurrency.
Casey generally, across all his material, advocates for non-pessimisation. Measurement takes too long to apply it to your entire program. He advocates for thinking about how much work the computer should actually have to do, then not making it do much more than that, at all times. This means avoiding serial dependency chains on the network, and huge towers of abstractions, and redundant work. He allows for writing lazy slow code as an intentional tradeoff that you may revise later if it becomes a bottleneck. He does not allow for inherently slow architecture.
Sure, but there are performance issues no profiler will catch in a straightforward flamegraph reading. Some examples:
1. Your hottest loop is spilling registers which only shows up as non-local cache thrashing (i.e. some other random code becomes slow) or randomly slow instructions i.e. "why is this xorps to initialise this float suddenly slow" due to pipeline stalls
2. Your code stops fitting into cache due to the code bloat, there's no one method which is slow, everything is slowed down by a percentage factor
3. A lot of useless work being done like temporary strings being copied everywhere
4. Your code is "I/O bound" because all the data you're accessing is scattered all across memory, leading to completely predictable TLB stalls
It's very easy to make a large program, quite a bit harder to make a small one...
How do you even measure those? Will that show up under something like VTune?
Yes, VTune is good.
I'm on AMD and uProf isn't that good (it often fails to match PDB to source, it's slow / crashy, stuff like that) so I mostly use Superluminal myself (has godly UX) and only drop down to uProf when I have a suspicion and want to gather hardware stats.
You can do all that with perf on Linux, but everything is manual.
In other videos Casey argues against the profile–fix–repeat workflow (I’m not saying that you necessarily meant this by measuring), instead arguing for estimating the theoretical maximum, then trying to get close enough to it. His argument is that the former might push you towards a local minimum without realising that you could do much better
When "Estimating the theoretical maximum" you're going to bring your assumptions to that estimate and so you're equally likely to get stuck in some local optimal point because you didn't see what else was possible, so that doesn't persuade me much. Worse, now it's a local optimal which might be entirely in your head.
There is some space for this "Estimate the theoretical maximum" as a high level insight, if the software needs to do a thing which your estimate says is impossible that's important to address up front but I think for practical software engineering you're much more likely to need profile-fix-repeat
Actually measuring is crucial and an estimate is not a measurement. I think one of the reasons many in the Handmade Community don't like measuring is that it too often ruins their "estimate" of how the crap they've written is achieving a "theoretical maximum" in favour of a boring fact that it's much worse than alternatives.
> Measure. Measure. Measure.
The problem is knowing what to measure. There's another saying
"When a measure becomes a target, it ceases to be a good measure."
As an example from memory, there was a game dev company that celebrated they had maxed out the cores on the PS3. That didn't mean anything though, anyone can max out the cores by filing them with bad code. But hey, their "measurement" told them they had maxed out the machine
> The problem is knowing what to measure.
This can be a problem, but much less so because so often we're doing "easy mode" where we don't need a proxy. The "it ceases to be a good measure" is because you're measuring a proxy. You wanted to deliver happiness, you measured wealth because it was easier to measure but seemed correlated and now you've got rich miserable people, oops. But software engineers can often measure the actual thing they want to improve directly, not a proxy and so it cannot cease to be a good measure.
With a few caveats, though they do tend to get fixed over time. For example, frame rate. Higher frame rate is better, the end, right? Wellll...
Latency and hitching are annoying to measure, so for a long time, they were pretty much ignored. That has improved and hopefully will continue to improve. There is the bufferbloat initiative, gaming magazines take frame time histograms now, input devices and screens commonly have their latency measured. But latency is still under-measured - for GPUs / GPU drivers, for all software, for games and particularly for websites.