I've heard this is part of why major infrastructure projects in America can be so expensive. A city builds one subway line, and everyone working on the project has no experience, so it takes a long time and is expensive. The expense convinces people to oppose any more projects, so the city doesn't build any public transit for a decade(s). By the time they're ready to build another line, all the experience has evaporated, so the new line takes a long time and is expensive. Repeat forever.
There's an example of this in railway electrification: if you scroll down to the graph in https://publications.parliament.uk/pa/cm5801/cmselect/cmtran... it shows that the UK tends to do electrification as occasional big projects, whereas Germany has consistently done about the same mileage every year for decades, presumably with the same institutions maintaining their expertise and just moving on to the next bit of track. Their costs are a quarter of the UK's...
Not Just Bikes did a YouTube on Seoul South Korea that brought this point up. They’ve got costs down because they’re working on it continuously.
As a tech writer people have a lot of experience but they never turn it into institutional knowledge because it’s never written down.
Ay best it’s tribal knowledge passed by word of mouth.
I know some people refuse to document things because they are hoping for job security but that never happens. Or sometimes for revenge for getting rid of them. But many companies survive despite those efforts.
Thank you for bringing this up. This is profoundly true for big projects (toll roads/transport) and small infra projects (e.g. community solar). The length of time that it takes to develop things like this, combined with the turnover and the sheer amount of context that single developer has to have to be successful with it, is one of the driving forces in why development is such a difficult/risky business.
It's one of the most consequential problems imaginable to solve, particularly as the US begins to realize that we need to compete with decades of China's subsidized energy and industrialization/manufacturing capacity.
Taking it a level deeper, what most don't realize is that infrastructure is an asset class: before someone funds the construction of $100M of solar technology, a developer will spend 2-5 years developing 15 or so major commercial agreements that enable a lender/financier to take comfort that when they deploy such a large amount of cash, they'll achieve a target yield over 20+ years. Orchestrating these negotiations (with multiple "adversaries") into a single, successfully bankable project is remarkably difficult and compared to the talent needed, very few have the slightest clue how to do this successfully.
Our bet at Phosphor is that this is actually solvable by combining natural language interfaces with really sophisticated version control and programming languages that read like english for financial models and legal agreements, which enables program verification. This is a really hard technical challenge because version control like Git really doesn't work: you need to be able to synchronize multiple lenses of change sets where each lens/branch is a dynamic document that gets negotiated. Dynamically composable change sets all the way down.
We are definitely solving this at Phosphor (phosphor.co) and we're actively hiring for whoever is interested in working at the intersection of HCI, program verification, natural language interfaces and distributed systems.
There’s strategic bidding as well. Specifications cannot cover every conceivable occurrence over the course of a 4 year construction project, so contractors can structure their bid to be low upfront with big pick ups later for change orders when issues arise.
That makes sense. It seems like during the continuous "building up America" period of the late 40s through mid 70s there was no problem of getting shit done at reasonable cost, because of continuously available institutional knowledge.
Once large infrastructure projects become sporadic in nature, you begin to run into issues.
The solution has to be continuous stimulus, but that also runs into problems of corruption and capture by special interests (the longer the stimulus, the more incentive there is for 3rd parties to appropriate funds).
Somehow, other nations have managed to figure this out. Of the developed world, seemingly only Americans are resigned to the belief that such things are sadly impossible.
This is often made worse as a result of hiring outside consultants. Firstly they don't have the institutional knowledge you have when starting a project, but they also aren't incentivised to properly document and hand over their knowledge at the end since that means less future work.
This is why a lot of government projects take so long, they don't see the value in keeping an in-house team of trained experts (see the difference in train line contruction costs in the UK compared to Spain), until you realised how good they were but you can't hire them back.
> What happened next, you may not be surprised to hear, comes in different versions. The person who spotted that there might be a problem may have been a member of Her Majesty’s Constabulary…
>> While they were away, a passing policeman noticed an extraordinary whirlpool in the normally placid canal. He also noticed that the water level was falling. He rushed off to find the dredging gang. By the time they all returned, the canal had disappeared. It was then that realisation dawned. Jack and his men had pulled out the plug of the canal. One-and-a-half miles of waterway had gone down the drain.
> It may have been three anglers who raised the alarm, and given that they have names — Howard Poucher, Graham Boon and Pete Moxon — maybe that version’s true. Another telling says it wasn’t until the evening that
>> local police contacted Stuart Robinson, the British Waterways section inspector.
Other relevant context: sections of UK canals being unintentional drained isn't particularly unusual, although the culprit is usually a paddle left open on a lock gate or a leaky culvert rather than a plug being pulled. Whether that inconveniences anyone for any length of time depends mostly on how full the reservoirs at the top end of the canal are...
Wouldn't have been that unusual in 1972 when nearly all the canals including that one had ceased commercial operations and many of them had been intentionally drained either. I suspect the transition from the canal being infrastructure maintained by locally-stationed full time professionals to a pleasure cruiseway which the new waterways board was willing to devote a bit of time to maintaining only after the previous one had spent several years trying to get it shut down probably had as much impact as the Blitz on the work crew having no idea about plugholes...
Too much emphasis on documentation. It's people that matter.
If you build the sort of culture where people hang around, they will occasionally have time to tell each other the internal folklore. "When I started, an old guy told me about the plug under the canal".
People who work with software know this. Yeah, there are documents describing the system. No, reading them does not mean you understand the system.
Alas, it's an intangible, and doesn't get counted with the rest of the beans.
I suspect this is why it's good for the USA to be constantly at war. If you're only at war occasionally, you forget how to make war and can lose. If you're at war constantly, you'll remember how to do it.
Perhaps tangentially related Re: “Chesterfield’s plug", Chesterton’s fence came to mind today while mulling over this sort of “forgetfulness” (which tends toward outright negligence) in my own circles.
It can be documents and it can be people, but it's not essentially either one. It can take many forms, including being lost when none of those forms has it on offer, as every business is different. An institution with excellent documentation, mature processes, and adept hiring could retain its "memory" without a single human member remaining from the past. Oral history and other humanistic forms of memory make everyone feel warm and fuzzy, but they're not to be idealized as the only real memory simply because they were underappreciated for a some time.
Apologies for bring in "AI" to a non-AI thread, but I really do think that things will be a game changer for institutional memory, both at recording it and recovering it. I don't personally use them but I have many coworkers that use AI tools to join meetings and get summaries/transcriptions aftward that they can read or query (also using AI). As people get more used to it, I would imagine that sort of thing becomes standard practice (regardless of whether or not it should, but that's a different topic)
I've heard this is part of why major infrastructure projects in America can be so expensive. A city builds one subway line, and everyone working on the project has no experience, so it takes a long time and is expensive. The expense convinces people to oppose any more projects, so the city doesn't build any public transit for a decade(s). By the time they're ready to build another line, all the experience has evaporated, so the new line takes a long time and is expensive. Repeat forever.
There's an example of this in railway electrification: if you scroll down to the graph in https://publications.parliament.uk/pa/cm5801/cmselect/cmtran... it shows that the UK tends to do electrification as occasional big projects, whereas Germany has consistently done about the same mileage every year for decades, presumably with the same institutions maintaining their expertise and just moving on to the next bit of track. Their costs are a quarter of the UK's...
Not Just Bikes did a YouTube on Seoul South Korea that brought this point up. They’ve got costs down because they’re working on it continuously.
As a tech writer people have a lot of experience but they never turn it into institutional knowledge because it’s never written down. Ay best it’s tribal knowledge passed by word of mouth.
I know some people refuse to document things because they are hoping for job security but that never happens. Or sometimes for revenge for getting rid of them. But many companies survive despite those efforts.
Thank you for bringing this up. This is profoundly true for big projects (toll roads/transport) and small infra projects (e.g. community solar). The length of time that it takes to develop things like this, combined with the turnover and the sheer amount of context that single developer has to have to be successful with it, is one of the driving forces in why development is such a difficult/risky business.
It's one of the most consequential problems imaginable to solve, particularly as the US begins to realize that we need to compete with decades of China's subsidized energy and industrialization/manufacturing capacity.
Taking it a level deeper, what most don't realize is that infrastructure is an asset class: before someone funds the construction of $100M of solar technology, a developer will spend 2-5 years developing 15 or so major commercial agreements that enable a lender/financier to take comfort that when they deploy such a large amount of cash, they'll achieve a target yield over 20+ years. Orchestrating these negotiations (with multiple "adversaries") into a single, successfully bankable project is remarkably difficult and compared to the talent needed, very few have the slightest clue how to do this successfully.
Our bet at Phosphor is that this is actually solvable by combining natural language interfaces with really sophisticated version control and programming languages that read like english for financial models and legal agreements, which enables program verification. This is a really hard technical challenge because version control like Git really doesn't work: you need to be able to synchronize multiple lenses of change sets where each lens/branch is a dynamic document that gets negotiated. Dynamically composable change sets all the way down.
We are definitely solving this at Phosphor (phosphor.co) and we're actively hiring for whoever is interested in working at the intersection of HCI, program verification, natural language interfaces and distributed systems.
There’s strategic bidding as well. Specifications cannot cover every conceivable occurrence over the course of a 4 year construction project, so contractors can structure their bid to be low upfront with big pick ups later for change orders when issues arise.
Such tricks, however, are known. The further trick is that those looking at bids can flag gaps or not depending on their connections to the bidders.
That makes sense. It seems like during the continuous "building up America" period of the late 40s through mid 70s there was no problem of getting shit done at reasonable cost, because of continuously available institutional knowledge.
Once large infrastructure projects become sporadic in nature, you begin to run into issues.
The solution has to be continuous stimulus, but that also runs into problems of corruption and capture by special interests (the longer the stimulus, the more incentive there is for 3rd parties to appropriate funds).
Somehow, other nations have managed to figure this out. Of the developed world, seemingly only Americans are resigned to the belief that such things are sadly impossible.
the important part of the American system you're not addressing is that it makes sure no one accidentally gets something they don't really deserve.
tl;dr Economies of scale
This is often made worse as a result of hiring outside consultants. Firstly they don't have the institutional knowledge you have when starting a project, but they also aren't incentivised to properly document and hand over their knowledge at the end since that means less future work.
This is why a lot of government projects take so long, they don't see the value in keeping an in-house team of trained experts (see the difference in train line contruction costs in the UK compared to Spain), until you realised how good they were but you can't hire them back.
Via "Coates" on Bluesky https://bsky.app/profile/oddthisday.bsky.social/post/3lvzzmj... at at Medum https://mulberryhall.medium.com/odd-this-day-5b1cfd1fdb32 who provides some other information:
> What happened next, you may not be surprised to hear, comes in different versions. The person who spotted that there might be a problem may have been a member of Her Majesty’s Constabulary…
>> While they were away, a passing policeman noticed an extraordinary whirlpool in the normally placid canal. He also noticed that the water level was falling. He rushed off to find the dredging gang. By the time they all returned, the canal had disappeared. It was then that realisation dawned. Jack and his men had pulled out the plug of the canal. One-and-a-half miles of waterway had gone down the drain.
> It may have been three anglers who raised the alarm, and given that they have names — Howard Poucher, Graham Boon and Pete Moxon — maybe that version’s true. Another telling says it wasn’t until the evening that
>> local police contacted Stuart Robinson, the British Waterways section inspector.
Other relevant context: sections of UK canals being unintentional drained isn't particularly unusual, although the culprit is usually a paddle left open on a lock gate or a leaky culvert rather than a plug being pulled. Whether that inconveniences anyone for any length of time depends mostly on how full the reservoirs at the top end of the canal are...
Wouldn't have been that unusual in 1972 when nearly all the canals including that one had ceased commercial operations and many of them had been intentionally drained either. I suspect the transition from the canal being infrastructure maintained by locally-stationed full time professionals to a pleasure cruiseway which the new waterways board was willing to devote a bit of time to maintaining only after the previous one had spent several years trying to get it shut down probably had as much impact as the Blitz on the work crew having no idea about plugholes...
Too much emphasis on documentation. It's people that matter.
If you build the sort of culture where people hang around, they will occasionally have time to tell each other the internal folklore. "When I started, an old guy told me about the plug under the canal".
People who work with software know this. Yeah, there are documents describing the system. No, reading them does not mean you understand the system.
Alas, it's an intangible, and doesn't get counted with the rest of the beans.
I suspect this is why it's good for the USA to be constantly at war. If you're only at war occasionally, you forget how to make war and can lose. If you're at war constantly, you'll remember how to do it.
This is one reason why what ServiceNow does is so important.
Perhaps tangentially related Re: “Chesterfield’s plug", Chesterton’s fence came to mind today while mulling over this sort of “forgetfulness” (which tends toward outright negligence) in my own circles.
https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
Solid writing.
This misses something very important.
Institutional memory is not information or documents - it's people.
Every single real-world process has implicit knowledge. And you can't always capture that knowledge of paper.
But, many corporations seem to want to get rid of their most experienced people to save money and have better quarterly results for the stock market.
Yes, I think people create more internal documentation then they read.
It can be documents and it can be people, but it's not essentially either one. It can take many forms, including being lost when none of those forms has it on offer, as every business is different. An institution with excellent documentation, mature processes, and adept hiring could retain its "memory" without a single human member remaining from the past. Oral history and other humanistic forms of memory make everyone feel warm and fuzzy, but they're not to be idealized as the only real memory simply because they were underappreciated for a some time.
Apologies for bring in "AI" to a non-AI thread, but I really do think that things will be a game changer for institutional memory, both at recording it and recovering it. I don't personally use them but I have many coworkers that use AI tools to join meetings and get summaries/transcriptions aftward that they can read or query (also using AI). As people get more used to it, I would imagine that sort of thing becomes standard practice (regardless of whether or not it should, but that's a different topic)