>Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
>The specification was not so much finished as abandoned after FHS 3.0 was released...
OK.
>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
>Meanwhile, in the absence of a current standard, systemd has spun off its file-hierarchy documentation to the Linux Userspace API (UAPI) Group as a specification. LWN covered that development in August, related to Fedora's search for an FHS successor.
Ah. Systemd/Fedora want a standard that they can directly control without interference from others.
Author here: It was abandoned. I linked to one of the former maintainers who said as much. The current effort is by a few people who asked the LF to take out over, and have (so far) done little after an initial flurry of activity. That, too, is covered in the other article I wrote about the FHS recently.
Prior to the group who started an update effort, it had not been touched in about a decade. That’s not slow-moving: that’s abandoned.
The FHS ultimately belongs to the users collectively, not those maintaining it. I am old enough to remember the horror that existed before the influence of the FHS. It exists in the fact that it is to some extent respected, not because there is a file somewhere that says it is the FHS standard. If you want changes, then sure, do the politics required to develop support for those changes. You can't just declare a new standard and then do whatever you want.
Developers have this thing where they will think of a standard as a specification. Instead it is a statement of political will. Saying that a standard is "abandoned" due to lack of "maintenance" seems like an example of thinking of a standard as the instantation of a specification; an actual program.
I agree--given your contraints of law and 10 years. But what about a law that hasn't been updated for 150 years? There's plenty of those that we regularly ignore.
There is no automatic, fixed timeframe after which a law simply stops being followed because it hasn't been updated or looked at; and remember, we're still applying the FHS, it's in active use even if it's not updated.
Laws remain in force until they are formally:
* Repealed (abolished) by the relevant legislative body (Parliament, Congress, etc.).
* Struck down by a court as unconstitutional or otherwise invalid.
A 150 year "delete" timer would genuinely undermine the foundation of the legal system. Lawyers, judges, and businesses rely on the continuity of core laws (e.g., contract, property, and tax law). If a 150-year-old property law suddenly lapsed, it could instantly void millions of land titles and commercial contracts...
Much of the US/UK legal system is based on common-law rules that are several hundred years old. In some cases those old laws have been codified, in some cases not, but either way there's no need to drop them just because they're old. On the contrary, laws that have stood that long without needing to be changed have demonstrated that they are extraordinarily good ideas.
The US constitution is still in force after 236 years, and even older laws are still enforced. US courts will sometimes look at precedent from England before the colonies existed.
Meanwhile some laws that are months old are ignored by law enforcement because nothing forces them to read it. It’s that effect which is why so many old laws are ignored rather than formally repealed. When nobody is ridding a horse nobody cares how you need to tie one up when visiting a store etc.
> The US constitution is still in force after 236 years
True, but it's been updated a lot more recently than that.
The last update was still much longer ago than 10 years, of course. The most recently ratified amendment to the Constitution - the Twenty-Seventh Amendment, ratified 1992 - was, incredibly enough, proposed in 1789 along with the ten we know as the Bill of Rights and another one which was never ratified. And of the twenty-seven amendments ratified so far, the one most recently proposed by Congress, the Twenty-Sixth Amendment, was both proposed and ratified in 1971.
Are you suggesting that appending the constitution in 1992 with: No law, varying the compensation for the services of the Senators and Representatives, shall take effect, until an election of Representatives shall have intervened.
Somehow has an impact on anything else? Because by that standard every change to any law updates all existing laws that were not changed. Or I’m just completely misunderstanding your point here.
Obviously the intent was "Thou shall not murder anyone"... Interpreting it otherwise doesn't make sense, and is inconsistent with the rest of the Bible.
Counterpoint: Modern distro’s needs have evolved past the FHS in some cases, and everybody deviates from it slightly but incompatibly.
A standard does no good if it does not reflect reality. I think it is a worthwhile effort to try to bring it back in line with actual real world usage.
OS X doesn't have a merged /usr. On my Mac I see /bin as a separate directory, not a symlink into `/usr`. Does Linux have a compelling reason that OS X lacks?
The reason for having a separate /usr has long slid into obsolescence. Our storage is no longer constrained to require us to mount a remote partition which holds most of the binaries and other sundry required to boot a distribution.
I think the current strongest argument against it is that systemd complains when /usr is on a separate partition[1], and what its devs have weighed in on the matter[2].
sysvinit had no problem being told to mount /usr as soon as network was available, and if you set up an init script to run before /usr was available, but the script needed /usr, that was your own fault.
systemd relies on things in /usr being available, including to decide which scripts to run, and mounting /usr would be one of those scripts, so it has a chicken-and-egg problem.
But ah, it doesn't! Instead the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Personally, I don't mind /usr/bin merging with /bin, the benefit I can see is no more squabbling over whether something should be in /bin or not (i.e. is this tool needed to boot the system, or not?)
The difference being that the authors of sysvinit didn't advertise obnoxious messages at boot time (https://systemd.io/SEPARATE_USR_IS_BROKEN/) and try to get the filesystem standards changed.
One is like "I'll run some scripts in order, everything else is on you", the other is like "I'll take care of everything, I'll do that, WHAT YOU DIDN'T MOUNT /USR ? SHAME ON YOU I DON'T WANT TO DEAL WITH THAT CORNER-CASE"
In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are suitably flexible to match any system (jack of all trades, master of none).
From the creators of systemd we also have GNOME, PulseAudio, and Wayland. They have some design properties in common.
no, it is that it adds complexity which is no longer needed
especially for image based stuff it's a pain
which includes OCI images for things like docker
but also image based distros like e.g. ostree (as used through rpm-ostree by Atomic Fedora desktops like Fedora Silverblue, but also in similar but different forms something Ubuntu has been experimenting with)
macOS didn't merge /usr, but it did do something sorta related.
One of the purposes of usrmerge is to cleanly separate the read-only and read-write parts of the system. This helps with image-based distros, where /usr can be on its own read-only filesystem, and related use cases such as [1]. Usrmerge is not required for image-based distros to work [2], but it makes things cleaner.
macOS, starting in 2019, is also an 'image-based distro', in that it has a read-only filesystem for system files and a separate read-write filesystem for user data. However, the read-only filesystem is mounted at / instead of /usr. Several different paths under the root need to be writable [3], which is implemented by having a single read-write filesystem (/System/Volumes/Data) plus a number of "firmlinks" from paths in the read-only filesystem to corresponding paths in the read-write filesystem. Firmlinks are a bespoke kernel feature invented for this purpose.
Both approaches have their advantages and disadvantages. The macOS approach is nice in that the system filesystem contains _all_ read-only files/directories, whereas under "distro in /usr" scheme, you need a separate tmpfs at / to contain the mount points and the symlinks into /usr. But "distro in /usr" has the advantage of making the separation between read-only and read-write files simpler and more visible to the user. Relatedly, macOS's scheme has the disadvantage that every writable file has two separate paths, one with /System/Volumes/Data and one without. But "distro in /usr" has the opposite disadvantage, in that a lot of read-only files have two separate paths, one with /usr and one without. Finally, macOS's scheme has the disadvantage that it required inventing and using firmlinks. Linux can already achieve similar effects using bind mounts or overlayfs, but those have minor disadvantages (bind mounts are more annoying to set up and tear down; overlayfs has a bit of performance overhead). Actual firmlinks are not necessarily any better, though, since they don't have a clear story for being shared between containers (which macOS does not support). It is nice that "distro in /usr" doesn't require any such complexity.
Ultimately, the constraints and motivations on both sides are quite different. macOS couldn't have gotten everything read-only under one directory as easily because it has /System in addition to /usr. macOS doesn't have containers. macOS doesn't have different distros with different filesystem layouts and deployment mechanisms. And philosophically, for all that people accuse systemd of departing from Unix design principles, systemd seems to see itself as evolving the Unix design, whereas macOS tends to treat Unix like some legacy thing. It's no surprise that systemd would try to improve on Unix with things like "/bin points to /usr/bin" while macOS would leave the Unix bits as-is.
> What ongoing maintenance would a file system standard require?
adaption to _a lot_ of subtle changes to requirements
- very different security related requirements today
- very different performance related requirements/characteristics
- very different need for various edge cases
and lastly adapt based on what turned out to work well and what didn't
so some examples not already mentioned in the article
- /boot -- dead or at least differently used if you use efistub booting
- /etc/X11 -- half dead on wayland
- /etc/xml, /etc/sgml -- dead, should IMHO never have existed
- also why was /etc/{X11,xml,sgml} every explicit part of the standard when the spec for `/etc` already implies them as long as e.g X11 is used ??
- `/media` -- dead/half dead depending on distro, replaced by `/run/media/{username}/{mount}`
- `/sbin` -- "controversial"; frequent reoccurring discussions that it isn't needed anymore, didn't work out as intended etc. It was useful for very old style thin clients as `/sbin` was in storage but `/bin` was mounted. And there are still some edge cases where it can makes sense today but most fall under "workaround for a different kind of problem which is better fixed properly".
- `/tmp` -- "controversial", long history of security issues, `/tmp` dir per program fixes the security issues (e.g. systemd service PrivateTmp option) but requires having a concept of "programs" instead of just "running processes" (e.g. by systemd services or flatpack programs). Also `tmpfiles.d` can help here.
- `/usr/libexec` -- dead, nice idea but introduces unneeded complexity and can be very misleading in combination swith suid and similar
- `/usr/sbin` see `/sbin`
- `/usr/share/{color,dict,man,misc,ppd,sgml,xml}` -- should never have been in the standard they are implied by the definition of `/usr/share`; at least sqml,xml are dead. dict was for spell check/auto completion, except that neither works anymore like dict expects
- `/var/account` -- to specific to some subset of partially dead programs, shouldn't be in the standard
- `/var/crash` -- distro specific mess
- `/var/games` -- basically dead/security mess, I mean 99% of games today are user per-user installed (e.g. Steam) and even for such which are packed any variable download data is per user, making it shared creates a permission/security mess
- `/var/lock` -- as mentioned there are better technical solutions by now, e.g. using `flock` instead of "presence of file" and some other techniques. Tend to also avoid issues of crashed programs not cleaning up "lock files" leading to dead locks and needing manual intervention.
- `/var/mail` assumes a quite outdated form of managing mail which is quite specific to the mailing program, as it's very program specific it IMHO shouldn't be in the standard
- various legacy program specific, non "generic" file system requirements e.g. that `/usr/lib/sendmail` must exist and be a link to a sendmail compatible program and similar.
also missing parts:
- `/run/user/{uid}`
- `/var/run/user/{uid}`
- `/proc`
- `/sys`
- user side versions (e.g. from the XDG spec which is also somewhat in a zombie state from my personal experience with it , e.g. .config, .local/{bin,share})
- references to light weight sandboxing, e.g. per-program /temp etc.
- factory reset stuff (`/usr/share/factory`) needed for having a uniform way for devices sold with Linux and device specific distro customization(e.g. steam deck)
Well, it probably depends on which software's concern will be implementing a policy to prevent users from having permission to fill critical directories and prevent the system from operating normally, which is discussed in the article. Which is also a coordination problem because the most common user of disk is software itself, I think.
FHS seems to specifically imbue the user with the responsibility and consequences of filling up the disk.
>> Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
> What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
It's 2025, anything that wants to be considered modern (and everything should want that), needs to be undergoing constant change and delivering regular "improvements."
>>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
> So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
The FHS people to get off their butts. There's no excuse for that pace now that we have such well-developed AI assistants. They should be pushing quarterly updates at a minimum, and a breaking change at least every year or two. It's been obvious for decades that "etc" is in urgent need of renaming to "config", "home" to "user", and "usr" to "Program Files" to keep up with modern UX trends.
I thought it would have been obvious by the "Program Files" at the end :).
Anyway, Linux community as a whole has an antiquated development process, and needs to modernize and follow the best practices of an industry-leading trend-setter, like MS Teams.
> The FHS 3.0 is clearly reaching the end of its useful life, if not actually expired.
Interesting take.
I think that the FHS is still extremely helpful for packagers, sysadmins and others so they won't stomp on each other's feet constantly. It helps set expectations and prevents unnecessary surprises.
Just the fact that one particular FHS rule might be outdated or even harmful doesn't mean that the FHS as a whole has outlived its usefulness.
Standards are a double edged sword though. They are great for getting everyone to agree to the "most correct" answer. But they also freeze evolution in place. What happens when your standard doesn't support contemporary use cases? What if it's at direct odds with, say, modern security practices?
FHS hasn't changed in years. Since then, sandboxing, containers, novel package schemes, and more are the zeitgeist. What does the FHS say about them?
Looking at this specific use case, someone is saying /var/lock being world-writable is an unacceptable security risk, but that's very dependent on what your world/users look like. If anything it sounds to me like the maintainer is trying to make the FHS smaller and remove support for a lot of use cases. (Use cases that sound pretty valid to me, without digging in.)
Nothing keeps you from following the FHS inside your container or sandbox.
Are you referring to the location where container images live? Then `/var/lib/containers/` and `/var/lib/containers/storage/` would be perfectly FHS compliant.
The idea though is when you don't want to follow the FHS anymore, like systemd is doing.
Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy. At the same time, change needed to happen and change is gonna hurt. And big changes can't wait until they're just as stable as the old system: does anyone develop software like that in their own careers? I try not to ship complete crap but "just as stable as v1" is never a goal.
> Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy
Poettering is a Microsoft employee. It is normal that he follows the direction of the mothership. What is not normal is, that he has so many blind followers.
every distro has defined their own new file system layout standard
sure they all started out with the common ancestor of FHS 3.0, but diverged since then in various degrees
and some modern competing standards try to fix it (mainly UAPI Group)
(And yes some people will go one and one about how UAPI is just a way for systemd to force their ideas on others, but if you don't update a standard for 10+[1] years and aren't okay with others taking over this work either, idk. how you can complain for them making their own standard).
[1]: It's more like 20 years, but 10 years ago the Linux Fundation took over it's ownership.
Debian systemd maintainer Luca Boccassi has recently pushed through and dismissed several problematic and undesired breakages as "niche cases" in a way I personally find antithetical to what I expect from Debian.
I hope they have a change of mind in their approach.
I don't know of anyone that's been doing this for a while that hasn't been touched by systemd stupidity in some way. I still loathe the default behavior around the stub-resolver with unqualified names that "just worked" before Lennart decided he knew best.
Depends on what you mean by this in "been doing this"?
While work now mandates "If you want to use Linux, it has to be Ubuntu" (and I complied). On personal front - about a decade ago I've moved from "vanilla" Gentoo to Calculate Linux - which was and still is 100% Gentoo.
These days difference is even smaller, but already 10+ years ago Calculate had sane profiles as well as all software packages as pre compiled binaries matching those profiles.
And although systemd is one of configurable USE keywords on Calculate/Gentoo - it's still not the default.
So there probably are some folks that haven't been touched by systemd at all... For now.
Yes, the guy is legendary in terms of maximum arrogance. He did impressive work early on designing a complex system, but gets defensive when that overengineered moloch runs into real-world problems. Systemd has accumulated lots of small hacks to make it more versatile, let's hope a better solution will be available one day.
Apparently yes, since the parent to your comment has been flagged.
Personally I find an interesting observation, and microsoft contributing to linux in any way should be met with skepticism based on the entire last 30 years.
People are so quick to wipe away any wrongdoing from Microsoft as soon as they get thrown a bone, there's some interesting psychology here.
It's complicated. Microsoft devotes resources to it and they can afford to do so but they only have that luxury from being a massive user trampling megacorp.
If we could trust them to be devoting resources to it without any risk of abusing their access and power in the future, that would be sort of okay, but we can't.
Like, should Lockheed intentionally hire North Korean programmers at cheap rates because North Korea can afford to devote resources to helping Lockheed? The issue here is not primarily that North Korea is a massive citizen-trampling megastate. It's that Lockheed's interests are misaligned with North Korea's.
To be fair, he does know how globbing works: ".*" should include "." and ".." under normal globbing rules. The 'rm' command (presumably) has a special case in it to avoid traversing those in recursive mode because doing so would be a footgun.
There is Devuan, if you want to Debian but without systemd. I suspect though that "natively" non-systemd distros will be more consistent, personally I've found happiness with Guix.
In this case, I think the upstream maintainer's response -- "Upstream systemd will do X, distros who want to are free to do Y" -- is legitimate. Consider the reverse: If systemd requires a writable /run/lock, then distros who want to be more safe won't really be able to (or will have to implement a much more intrusive patch).
Looking from the outside, it looks more that this is a failure of the Debian systemd package maintainer to follow Debian's rules. (Though since I'm not a part of that community, I recognize that there may be cultural expectations I'm not aware of.)
> "Upstream systemd will do X, distros who want to are free to do Y"
Yes this is a good response from upstream. I can work with that, but in that case, even this response didn't get reflected to mailing list discussion, or drowned out instantly.
My question was more general though, questioning systemd developers' behavior collectively (hence the projects' behavior) through time.
The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
Speak for yourself, then. I’ve been using Linux since 2004, and the systemd components finally made system management easy. No more arcane init scripts. Handling of service dependencies. Proper timers. Simple configuration files. Administration knowledge that immediately carries over between all systems equally.
As a user, systemd has improved my productivity tremendously.
The kind of bad mouthing developers that work on solutions for complex problems, code that runs on billions of machines, reflects more of your own fragile ego than them.
> The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
> As a user, systemd has improved my productivity tremendously.
Both can be true at the same time. Particularly in the beginning, there was a long string of really important things that used to Just Work that were broken by systemd. Things like:
1. Having home directories in automounted NFS. Under sysv, autofs waited until the network was up to start running. Originally under systemd, "the network" was counted as being up when localhost was up.
2. Being able to type "exit" from an ssh session and have the connection close. Under systemd, closing the login shell would kill -9 all processes with that userid, including the sshd process handling the connection -- before that process could close the socket for the connection. Meaning you type "exit" in an interactive terminal and it hang.
It's been a while since I encountered any major issues with systemd, but for the first few years there were loads of issues with important things that used to Just Work and then broke and took forever to fix because they didn't happen to affect the systemd maintainers. If you didn't encounter any of these, it's probably because your use cases happened to overlap theirs.
Yes, systemd and journalctl have massively simplified my life. But I think it could have been done with far less disruption.
There's no need to be rude. While I'm not anti-systemd; it didn't change my life tremendously, either.
People tend to bash init scripts, but when they are written well, they both work and port well between systems. At least this is my experience with the fleet I manage.
Dependencies worked pretty well in Parallel-SysV, too, again from my experience. Also, systemd is not faster than Parallel-SysV.
It's not that "I had to learn everything from scratch!" woe either. I'm a kind of developer/sysadmin who never whines and just reads the documentation.
I wrote tons of service files and init scripts during Debian's migration. I was a tech-lead of a Debian derivative at that time (albeit working literally underground), too.
systemd and its developers went through a lot phases, remade a lot of mistakes despite being warned about them, and took at least a couple of wrong turns and booed for all the right reasons.
The anger they pull on themselves are not unfounded, yet I don't believe they should be on the receiving end of a flame-war.
From my perspective, systemd developers can benefit tremendously by stepping down from their thrones and look eye to eye with their users. Being kind towards each other never harms anyone, incl. you.
Linux is 34 years old, and some of the Unix-ism borrowed are even older. There is genuine cruft that has downsides. Different relative priorities of backwards comparability, maintainability and the various issues the legacy issues cause are reasonable.
Systemd basically arose out of a frustration at the legacy issues so the whole project exists as a modernizing effort. No wonder they consider backwards compatibility low priority.
Because that's what they've always done, and it continues to work for them?
Systemd doesn't work for me, but it has taken over most Linux distributions, so clearly it's got something people want that I don't understand. That was the case for PulseAudio too.
* Their is an option for the old behavior.
* It is a security issue and better solutions to replace exist.
* FHS isn't maintained.
I think everyone involved would prefer updates to the applications, which fix the issue. Debian opted - for now - for reliability for its users, which fits in their mission statement. On Arch /run/lock is only writeable for the superusers, which improves security. As user I value reliability and security and that legacy tools remain usable (sometimes by default, sometimes by a switch).
The "security issue" expressed is that someone creates 4 billion lock files. The entire reason an application would have a path to create these lock files is because it's dealing with a shared resource. It's pretty likely that lock files wouldn't be the only route for an application to kill a system. Which is a reason why this "security issue" isn't something anyone has taken seriously.
The reason is much more transparent if you read between the lines. Systemd wants to own the "/run" folder and they don't like the idea of user space applications being able to play in their pool. Notice they don't have the same security concerns for /var/tmp, for example.
> On Arch /run/lock is only writeable for the superusers, which improves security.
Does it? That means anyone who needs a lock gets superuser, which seems like overkill. Having a group with write permissions would seem to improve security more?
a global /run/lock dir is an outdated mechanism not needed anymore
when the standard was written (20 years ago) it standardized a common way programs used to work around not having something like flock. This is also reflected in the specific details of FHS 3.0 which requires lock files to be named as `LCK..{device_name}` and must contain the process id in a specific encoding. Now the funny part. Flock was added to Linux in ~1996, so even when the standard was written it was already on the way of being outdated and it was just a matter of time until most programs start using flock.
This brings is to two ways how this being a issues makes IMHO little sense:
- a lot of use cases for /var/lock have been replaced with flock
- having a global writable dire used across users has a really bad history (including security vulnerabilities) so there have been ongoing affords to create alternatives for anything like that. E.g. /run/user/{uid}, ~/.local/{bin,share,state,etc.}, systemd PrivateTemp etc.
- so any program running as user not wanting to use flock should place their lock file in `/run/user/{uid}` like e.g. pipewire, wayland, docker and similar do (specifically $XDG_RUNTIME_DIR which happens to be `/un/user/{uid}`)
So the only programs affected by it are programs which:
- don't run as root
- don't use flock
- and don't really follow best practices introduced with the XDG standard either
- ignore that it was quite predictable that /var/lock will get limited or outright removed due to long standing efforts to remove global writable dirs everywhere
i.e. software stuck in the last century, or in this case more like 2 centuries ago in the 2000th
But that is a common theme with Debian Stable, you have to fight even to just remove something which we know since 20 years to be a bad design. If it weren't for Debians reputation I think the systemd devs might have been more surprised by this being an issue then the Debian maintainers about some niche tools using outdated mechanisms breaking.
No, I didn't read the whole article. I follow debian-devel directly. Watched all of it unravel, step by step. I know the resolution since the day it posted to debian-devel.
This was a general question to begin with.
> Their is an option for the old behavior.
The discussion never centered on an option for keeping old behavior for any legitimate reason. The general tone was "systemd wants it this way, so Debian shall oblige". It was a borderline flame-war between more reasonable people and another party which yelled "we say so!"
> It is a security issue and modern solutions to replace exist.
I'm a Linux newbie. Using Linux for 23 years and managing them professionally for 20+ years. I have yet to see an attack involving /var/lock folder being world-writeable. /dev/shm is a much bigger attack surface from my experience.
Migration to flock(2) is not a bad idea, but acting like Nero and setting mailing lists ablaze is not the way to do this. People can cooperate, yet some people love to rain on others and make their life miserable because they think their demands require immediate obedience.
> FHS isn't maintained.
Isn't maintained or not improved fast enough to please systemd devs? IDK. There are standards and RFCs which underpin a ton of things which are not updated.
We tend to call them mature, not unmaintained/abandoned.
> On Arch /run/lock is only writeable for the superusers. As user I value reliability and the legacy tools are usable.
I also value the reliability and agree that legacy tools shall continue working. This is why I use Debian primarily, for the same last 20+ years.
In this case I assume their "fear" is that unprivileged users can exhaust resources (inodes, filesystem space) in an important tmpfs breaking the system. The proper solution for backward compatibility would probably be something like make /run/lock its own mountpoint, but they fixed it in their system (Fedora) so now it's no longer their problem. Just be thankful their software is portable to such strange niche operating systems like Debian. /s
It's been a very long time since I heard about uucico (Unix-to-Unix Copy-In Copy-Out program, part of the UUCP suite). Glad to see it's still being shipped! I wonder if any network uses it.
The "somebody might mailicuously exhaust memory via /run” argument seems silly IMHO... you can trivially put a hard limit on its size via the tmpfs mount option. I guess if you bend over backwards that's still a DOS in that new files can't be created in /run once it is full, but come on...
There is support for quotas in tmpfs! /me runs and hides under desk to avoid fruit being thrown at me
Why is that non-clickbait? Honestly "Debian Technical Committee overrides systemd /run/lock permission change" might be a better title than either, I don't know whether the thing or the actors are more interesting here. But you can only say so much in a title.
> Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
I don't think that's an accurate paraphrase of "Consider this more a passing of the baton from upstream systemd to downstreams: if your distro wants this kind of legacy interface, then just add this via a distro-specific tmpfiles drop-in. But there's no point really in forcing anyone who has a more forward-looking view of the world to still carry that dir."
That kind of drop-in is pretty routine, so I don't know why this became a big thing we're all discussing now.
"everyone else" in this case is pretty much only the debian ecosystem because they insist on enforcing a serial lock policy from the 1980s. It's fine if Debian wants to move at the speed of a Soviet committee but I don't think it should be expected (or would be healthy) for systemd to move at the same pace.
A software developer's primary job is to develop software for their users, not to comply with a third party distributor that repackages their software.
The beef isn't with systemd upstream which already has a very simple/boring workaround for this, it's with the debian package maintainer (some people here are wearing multiple hats).
Really the whole raison d'etre of debian is move at this pace to prioritize stability/compatibility. If you don't like that philosophy there are other distros but a package maintainer's primary job is to repackage software for that distro (which presumably users have chosen for a reason), not comply with upstream.
Letting upstream systemd single-handedly define what directories exist with what modes in your distro has never been the intended Modus Operandi.
Debian has a huge selection of packages available for it and clearly is going to have more headaches when it comes to preserving compatibility with all that software.
This is a trivial matter for Debian to handle appropriately, while systemd stays focused on its current priorities. I'm surprised this is being talked about at all outside the appropriate mailing lists... slow week for linux news?
Somehow I feel that if all the time that has been invested in debating and discussing this had been spent on patching the affected apps, the problem would be properly solved.
I mean, yeah, I get it, systemd bad, democracy good, but these world-writable lock folders are actually a huge pain, and adding some shim code to upgrade to a more secure solution seems achievable?
Genuinely curious - why would world-writeable directory be bad for security? Assuming of course, it's on a separate filesystem mounted with sensible options. Here's what I see from "grep /run/lock /proc/mounts" in sid:
The classic is say you know a root process will write a file called foo.lock in /run/lock, and you (a bad person) have write access to that directory. Then you make foo.lock a symlink to some file (/bin/init or /bin/sh or ld.so for example would be very inconvenient choices) and when the root process writes its lock it destroys that file.
Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
Yup to both of you. But all of this is to say, running shellscripts as root (in particular) needs to be done with extreme care, because if people forget those precautions when writing C, they sure as heck don’t trouble themselves to do it when they’re writing shell.
I remember the time (around 2001-2002) when just about every binary was discovered to have some variant on this exact exploit. I happened to be linux sysadmin for a very large, high-profile set of linux boxes at the time. Happy times.
Hmm - I see there's now "lockdev" for managing access to things like serial lines, but what's the preferred method of expressing "only one instance of this program should run at any one time"?
I use GNU screen for that. Looking at cu, it looks to be just as tiny and has ssh-like tilda-escapes, including "~." to disconnect. Nice, gotta try it out, thanks!
What would you suggest? Personally I find cu the closest to ssh in ergonomics to any of the tui serial terminal programs, and it’s easily available across unixes like FreeBSD or Mac OS, which is a bonus.
Considering I never had to reinstall a Debian system because it got bloated or broke one day, I can accept this slow-cooking approach. Even support them on this regard.
Gentoo Linux has quite a different approach than Debian [0] but after the first month or two (once my new-to-Linux ass figured out what it was doing) I've never had to reinstall any Gentoo system. [1]
If you want what Debian provides, it's a poor choice for you... but -IME- it doesn't break on upgrade, unlike some Debian-derived distros I've tried in the past.
[0] Something along the lines of "Always try to package exactly what's provided by upstream, try hard to get distro patches upstreamed, and try to have the latest available upstream release in the 'testing channel'.".
[1] Well, I do have a machine that (aside from "side-loading" kernel updates from time to time) hasn't been updated in four years. While I'll try to update that one in the normal way, I'm probably going to need to reinstall.
This part seems fairly editorial:
>Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
>The specification was not so much finished as abandoned after FHS 3.0 was released...
OK.
>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
>Meanwhile, in the absence of a current standard, systemd has spun off its file-hierarchy documentation to the Linux Userspace API (UAPI) Group as a specification. LWN covered that development in August, related to Fedora's search for an FHS successor.
Ah. Systemd/Fedora want a standard that they can directly control without interference from others.
Author here: It was abandoned. I linked to one of the former maintainers who said as much. The current effort is by a few people who asked the LF to take out over, and have (so far) done little after an initial flurry of activity. That, too, is covered in the other article I wrote about the FHS recently.
Prior to the group who started an update effort, it had not been touched in about a decade. That’s not slow-moving: that’s abandoned.
The FHS ultimately belongs to the users collectively, not those maintaining it. I am old enough to remember the horror that existed before the influence of the FHS. It exists in the fact that it is to some extent respected, not because there is a file somewhere that says it is the FHS standard. If you want changes, then sure, do the politics required to develop support for those changes. You can't just declare a new standard and then do whatever you want.
Developers have this thing where they will think of a standard as a specification. Instead it is a statement of political will. Saying that a standard is "abandoned" due to lack of "maintenance" seems like an example of thinking of a standard as the instantation of a specification; an actual program.
I know it's not the same, but imagine thinking a law is not longer meant to be followed because it hasn't been updated in 10 years.
I agree--given your contraints of law and 10 years. But what about a law that hasn't been updated for 150 years? There's plenty of those that we regularly ignore.
What's the timeline for software?
There is no automatic, fixed timeframe after which a law simply stops being followed because it hasn't been updated or looked at; and remember, we're still applying the FHS, it's in active use even if it's not updated.
Laws remain in force until they are formally:
* Repealed (abolished) by the relevant legislative body (Parliament, Congress, etc.).
* Struck down by a court as unconstitutional or otherwise invalid.
A 150 year "delete" timer would genuinely undermine the foundation of the legal system. Lawyers, judges, and businesses rely on the continuity of core laws (e.g., contract, property, and tax law). If a 150-year-old property law suddenly lapsed, it could instantly void millions of land titles and commercial contracts...
Much of the US/UK legal system is based on common-law rules that are several hundred years old. In some cases those old laws have been codified, in some cases not, but either way there's no need to drop them just because they're old. On the contrary, laws that have stood that long without needing to be changed have demonstrated that they are extraordinarily good ideas.
The US constitution is still in force after 236 years, and even older laws are still enforced. US courts will sometimes look at precedent from England before the colonies existed.
Meanwhile some laws that are months old are ignored by law enforcement because nothing forces them to read it. It’s that effect which is why so many old laws are ignored rather than formally repealed. When nobody is ridding a horse nobody cares how you need to tie one up when visiting a store etc.
> The US constitution is still in force after 236 years
True, but it's been updated a lot more recently than that.
The last update was still much longer ago than 10 years, of course. The most recently ratified amendment to the Constitution - the Twenty-Seventh Amendment, ratified 1992 - was, incredibly enough, proposed in 1789 along with the ten we know as the Bill of Rights and another one which was never ratified. And of the twenty-seven amendments ratified so far, the one most recently proposed by Congress, the Twenty-Sixth Amendment, was both proposed and ratified in 1971.
Are you suggesting that appending the constitution in 1992 with: No law, varying the compensation for the services of the Senators and Representatives, shall take effect, until an election of Representatives shall have intervened.
Somehow has an impact on anything else? Because by that standard every change to any law updates all existing laws that were not changed. Or I’m just completely misunderstanding your point here.
How often does "thou shall not kill" need an update?
Not even in defense of your own life, family, others? You have a lot of people on HN that celebrate killing of then innocent.
Obviously the intent was "Thou shall not murder anyone"... Interpreting it otherwise doesn't make sense, and is inconsistent with the rest of the Bible.
"Obviously the intent was" probably not the same as it obviously was 150 years ago!
There are plenty of 150-year-old laws that we don't ignore, too.
Counterpoint: Modern distro’s needs have evolved past the FHS in some cases, and everybody deviates from it slightly but incompatibly.
A standard does no good if it does not reflect reality. I think it is a worthwhile effort to try to bring it back in line with actual real world usage.
The /usr merge is an example of something that a modern FHS might reflect.
OS X doesn't have a merged /usr. On my Mac I see /bin as a separate directory, not a symlink into `/usr`. Does Linux have a compelling reason that OS X lacks?
it makes things more complicated without any benefits (at least not anymore)
this doesn't matter for OS X which main changes mostly tend to be diverging away from it's roots into a fully proprietary direction
but it does matter if you build image based Linux distros which might be the future of Linux
Merged /usr is one (increasingly accepted?) means of implementing image-based distros. New OS version, new /usr parition.
The reason for having a separate /usr has long slid into obsolescence. Our storage is no longer constrained to require us to mount a remote partition which holds most of the binaries and other sundry required to boot a distribution.
In other words, the strongest argument for usrmerge is that there is no compelling argument against it?
I think the current strongest argument against it is that systemd complains when /usr is on a separate partition[1], and what its devs have weighed in on the matter[2].
[1] https://freedesktop.org/wiki/Software/systemd/separate-usr-i...
[2] https://www.freedesktop.org/wiki/Software/systemd/TheCaseFor...
sysvinit had no problem being told to mount /usr as soon as network was available, and if you set up an init script to run before /usr was available, but the script needed /usr, that was your own fault.
systemd relies on things in /usr being available, including to decide which scripts to run, and mounting /usr would be one of those scripts, so it has a chicken-and-egg problem.
But ah, it doesn't! Instead the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Personally, I don't mind /usr/bin merging with /bin, the benefit I can see is no more squabbling over whether something should be in /bin or not (i.e. is this tool needed to boot the system, or not?)
> sysvinit had no problem being told to mount /usr [..] if you set up an init script to run before /usr was available,
> the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Unironically in the same post despite being, to my untrained eye, the same thing.
The difference being that the authors of sysvinit didn't advertise obnoxious messages at boot time (https://systemd.io/SEPARATE_USR_IS_BROKEN/) and try to get the filesystem standards changed.
One is like "I'll run some scripts in order, everything else is on you", the other is like "I'll take care of everything, I'll do that, WHAT YOU DIDN'T MOUNT /USR ? SHAME ON YOU I DON'T WANT TO DEAL WITH THAT CORNER-CASE"
In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are suitably flexible to match any system (jack of all trades, master of none).
From the creators of systemd we also have GNOME, PulseAudio, and Wayland. They have some design properties in common.
no, it is that it adds complexity which is no longer needed
especially for image based stuff it's a pain
which includes OCI images for things like docker
but also image based distros like e.g. ostree (as used through rpm-ostree by Atomic Fedora desktops like Fedora Silverblue, but also in similar but different forms something Ubuntu has been experimenting with)
macOS didn't merge /usr, but it did do something sorta related.
One of the purposes of usrmerge is to cleanly separate the read-only and read-write parts of the system. This helps with image-based distros, where /usr can be on its own read-only filesystem, and related use cases such as [1]. Usrmerge is not required for image-based distros to work [2], but it makes things cleaner.
macOS, starting in 2019, is also an 'image-based distro', in that it has a read-only filesystem for system files and a separate read-write filesystem for user data. However, the read-only filesystem is mounted at / instead of /usr. Several different paths under the root need to be writable [3], which is implemented by having a single read-write filesystem (/System/Volumes/Data) plus a number of "firmlinks" from paths in the read-only filesystem to corresponding paths in the read-write filesystem. Firmlinks are a bespoke kernel feature invented for this purpose.
Both approaches have their advantages and disadvantages. The macOS approach is nice in that the system filesystem contains _all_ read-only files/directories, whereas under "distro in /usr" scheme, you need a separate tmpfs at / to contain the mount points and the symlinks into /usr. But "distro in /usr" has the advantage of making the separation between read-only and read-write files simpler and more visible to the user. Relatedly, macOS's scheme has the disadvantage that every writable file has two separate paths, one with /System/Volumes/Data and one without. But "distro in /usr" has the opposite disadvantage, in that a lot of read-only files have two separate paths, one with /usr and one without. Finally, macOS's scheme has the disadvantage that it required inventing and using firmlinks. Linux can already achieve similar effects using bind mounts or overlayfs, but those have minor disadvantages (bind mounts are more annoying to set up and tear down; overlayfs has a bit of performance overhead). Actual firmlinks are not necessarily any better, though, since they don't have a clear story for being shared between containers (which macOS does not support). It is nice that "distro in /usr" doesn't require any such complexity.
Ultimately, the constraints and motivations on both sides are quite different. macOS couldn't have gotten everything read-only under one directory as easily because it has /System in addition to /usr. macOS doesn't have containers. macOS doesn't have different distros with different filesystem layouts and deployment mechanisms. And philosophically, for all that people accuse systemd of departing from Unix design principles, systemd seems to see itself as evolving the Unix design, whereas macOS tends to treat Unix like some legacy thing. It's no surprise that systemd would try to improve on Unix with things like "/bin points to /usr/bin" while macOS would leave the Unix bits as-is.
[1] https://lwn.net/Articles/890463/ [2] https://blog.verbum.org/2024/10/22/why-bootc-doesnt-require-... [3] https://eclecticlight.co/2023/07/22/how-macos-depends-on-fir...
I'm very happy that the "independent standard" facade of the UAPI group fell, and its actions are now directly attributed to systemd's interests.
> What ongoing maintenance would a file system standard require?
adaption to _a lot_ of subtle changes to requirements
- very different security related requirements today
- very different performance related requirements/characteristics
- very different need for various edge cases
and lastly adapt based on what turned out to work well and what didn't
so some examples not already mentioned in the article
- /boot -- dead or at least differently used if you use efistub booting
- /etc/X11 -- half dead on wayland
- /etc/xml, /etc/sgml -- dead, should IMHO never have existed
- also why was /etc/{X11,xml,sgml} every explicit part of the standard when the spec for `/etc` already implies them as long as e.g X11 is used ??
- `/media` -- dead/half dead depending on distro, replaced by `/run/media/{username}/{mount}`
- `/sbin` -- "controversial"; frequent reoccurring discussions that it isn't needed anymore, didn't work out as intended etc. It was useful for very old style thin clients as `/sbin` was in storage but `/bin` was mounted. And there are still some edge cases where it can makes sense today but most fall under "workaround for a different kind of problem which is better fixed properly".
- `/tmp` -- "controversial", long history of security issues, `/tmp` dir per program fixes the security issues (e.g. systemd service PrivateTmp option) but requires having a concept of "programs" instead of just "running processes" (e.g. by systemd services or flatpack programs). Also `tmpfiles.d` can help here.
- `/usr/libexec` -- dead, nice idea but introduces unneeded complexity and can be very misleading in combination swith suid and similar
- `/usr/sbin` see `/sbin`
- `/usr/share/{color,dict,man,misc,ppd,sgml,xml}` -- should never have been in the standard they are implied by the definition of `/usr/share`; at least sqml,xml are dead. dict was for spell check/auto completion, except that neither works anymore like dict expects
- `/var/account` -- to specific to some subset of partially dead programs, shouldn't be in the standard
- `/var/crash` -- distro specific mess
- `/var/games` -- basically dead/security mess, I mean 99% of games today are user per-user installed (e.g. Steam) and even for such which are packed any variable download data is per user, making it shared creates a permission/security mess
- `/var/lock` -- as mentioned there are better technical solutions by now, e.g. using `flock` instead of "presence of file" and some other techniques. Tend to also avoid issues of crashed programs not cleaning up "lock files" leading to dead locks and needing manual intervention.
- `/var/mail` assumes a quite outdated form of managing mail which is quite specific to the mailing program, as it's very program specific it IMHO shouldn't be in the standard
- various legacy program specific, non "generic" file system requirements e.g. that `/usr/lib/sendmail` must exist and be a link to a sendmail compatible program and similar.
also missing parts:
- `/run/user/{uid}`
- `/var/run/user/{uid}`
- `/proc`
- `/sys`
- user side versions (e.g. from the XDG spec which is also somewhat in a zombie state from my personal experience with it , e.g. .config, .local/{bin,share})
- references to light weight sandboxing, e.g. per-program /temp etc.
- factory reset stuff (`/usr/share/factory`) needed for having a uniform way for devices sold with Linux and device specific distro customization(e.g. steam deck)
so yes, it's quite outdated
> `/usr/libexec` -- dead,
Definitely not dead, the XDG portals and Polkit agents live here.
Well, it probably depends on which software's concern will be implementing a policy to prevent users from having permission to fill critical directories and prevent the system from operating normally, which is discussed in the article. Which is also a coordination problem because the most common user of disk is software itself, I think.
FHS seems to specifically imbue the user with the responsibility and consequences of filling up the disk.
A more neutral phrasing would be.
> Debian Policy still cites the FHS, and FHS has remained static for over a decade.
>> Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
> What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
It's 2025, anything that wants to be considered modern (and everything should want that), needs to be undergoing constant change and delivering regular "improvements."
>>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
> So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
The FHS people to get off their butts. There's no excuse for that pace now that we have such well-developed AI assistants. They should be pushing quarterly updates at a minimum, and a breaking change at least every year or two. It's been obvious for decades that "etc" is in urgent need of renaming to "config", "home" to "user", and "usr" to "Program Files" to keep up with modern UX trends.
I genuinely can't tell if this is satire.
I thought it would have been obvious by the "Program Files" at the end :).
Anyway, Linux community as a whole has an antiquated development process, and needs to modernize and follow the best practices of an industry-leading trend-setter, like MS Teams.
The X11 people are trying this hard. I'm really curious how Wayland will evolve but, the history of GTK and QT does not give me much hope.
Ok, so it's only half satire, or is this reply also a satire? I mean, MS Teams, really?
> The FHS 3.0 is clearly reaching the end of its useful life, if not actually expired.
Interesting take.
I think that the FHS is still extremely helpful for packagers, sysadmins and others so they won't stomp on each other's feet constantly. It helps set expectations and prevents unnecessary surprises.
Just the fact that one particular FHS rule might be outdated or even harmful doesn't mean that the FHS as a whole has outlived its usefulness.
Standards are a double edged sword though. They are great for getting everyone to agree to the "most correct" answer. But they also freeze evolution in place. What happens when your standard doesn't support contemporary use cases? What if it's at direct odds with, say, modern security practices?
FHS hasn't changed in years. Since then, sandboxing, containers, novel package schemes, and more are the zeitgeist. What does the FHS say about them?
Looking at this specific use case, someone is saying /var/lock being world-writable is an unacceptable security risk, but that's very dependent on what your world/users look like. If anything it sounds to me like the maintainer is trying to make the FHS smaller and remove support for a lot of use cases. (Use cases that sound pretty valid to me, without digging in.)
> What does the FHS say about them?
Nothing keeps you from following the FHS inside your container or sandbox.
Are you referring to the location where container images live? Then `/var/lib/containers/` and `/var/lib/containers/storage/` would be perfectly FHS compliant.
The idea though is when you don't want to follow the FHS anymore, like systemd is doing.
Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy. At the same time, change needed to happen and change is gonna hurt. And big changes can't wait until they're just as stable as the old system: does anyone develop software like that in their own careers? I try not to ship complete crap but "just as stable as v1" is never a goal.
> Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy
Poettering is a Microsoft employee. It is normal that he follows the direction of the mothership. What is not normal is, that he has so many blind followers.
yes but also no
every distro has defined their own new file system layout standard
sure they all started out with the common ancestor of FHS 3.0, but diverged since then in various degrees
and some modern competing standards try to fix it (mainly UAPI Group)
(And yes some people will go one and one about how UAPI is just a way for systemd to force their ideas on others, but if you don't update a standard for 10+[1] years and aren't okay with others taking over this work either, idk. how you can complain for them making their own standard).
[1]: It's more like 20 years, but 10 years ago the Linux Fundation took over it's ownership.
Debian systemd maintainer Luca Boccassi has recently pushed through and dismissed several problematic and undesired breakages as "niche cases" in a way I personally find antithetical to what I expect from Debian.
I hope they have a change of mind in their approach.
This is usual systemd maintainer behavior. Poetterings "I don't consider it much of a problem" is legend: https://github.com/systemd/systemd/issues/5644
I don't know of anyone that's been doing this for a while that hasn't been touched by systemd stupidity in some way. I still loathe the default behavior around the stub-resolver with unqualified names that "just worked" before Lennart decided he knew best.
Depends on what you mean by this in "been doing this"?
While work now mandates "If you want to use Linux, it has to be Ubuntu" (and I complied). On personal front - about a decade ago I've moved from "vanilla" Gentoo to Calculate Linux - which was and still is 100% Gentoo.
These days difference is even smaller, but already 10+ years ago Calculate had sane profiles as well as all software packages as pre compiled binaries matching those profiles.
And although systemd is one of configurable USE keywords on Calculate/Gentoo - it's still not the default.
So there probably are some folks that haven't been touched by systemd at all... For now.
Yes, the guy is legendary in terms of maximum arrogance. He did impressive work early on designing a complex system, but gets defensive when that overengineered moloch runs into real-world problems. Systemd has accumulated lots of small hacks to make it more versatile, let's hope a better solution will be available one day.
Whenever I'm using Guix I enjoy the simplicity of Shepherd [1]
[1] https://shepherding.services/manual/html_node/Introduction.h...
Interesting coincidence that both are @Microsoft.
Do we really want Microsoft employees setting standards for Debian?
Apparently yes, since the parent to your comment has been flagged.
Personally I find an interesting observation, and microsoft contributing to linux in any way should be met with skepticism based on the entire last 30 years.
People are so quick to wipe away any wrongdoing from Microsoft as soon as they get thrown a bone, there's some interesting psychology here.
Also Microsoft is doing all kinds of abusive things to their users in Windows 11.
It's complicated. Microsoft devotes resources to it and they can afford to do so but they only have that luxury from being a massive user trampling megacorp.
If we could trust them to be devoting resources to it without any risk of abusing their access and power in the future, that would be sort of okay, but we can't.
Like, should Lockheed intentionally hire North Korean programmers at cheap rates because North Korea can afford to devote resources to helping Lockheed? The issue here is not primarily that North Korea is a massive citizen-trampling megastate. It's that Lockheed's interests are misaligned with North Korea's.
The fool doesn't know how globbing works, but considers his uninformed guess good enough without testing it or reading (and understanding) the spec.
To be fair, he does know how globbing works: ".*" should include "." and ".." under normal globbing rules. The 'rm' command (presumably) has a special case in it to avoid traversing those in recursive mode because doing so would be a footgun.
To be fair, POSIX-compliant shells are allowed to exclude `.` and `..`, and some do, and POSIX says this may be required in the future.
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V...
Link to such an dismissal: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1112535#64
There is Devuan, if you want to Debian but without systemd. I suspect though that "natively" non-systemd distros will be more consistent, personally I've found happiness with Guix.
Debian works fine without systemd (at least for now) though Devuan does indeed make like easier.
Can anyone tell why systemd developers run fast and loose with what they believe and bully everyone with a stick made out of their ideas?
In this case, I think the upstream maintainer's response -- "Upstream systemd will do X, distros who want to are free to do Y" -- is legitimate. Consider the reverse: If systemd requires a writable /run/lock, then distros who want to be more safe won't really be able to (or will have to implement a much more intrusive patch).
Looking from the outside, it looks more that this is a failure of the Debian systemd package maintainer to follow Debian's rules. (Though since I'm not a part of that community, I recognize that there may be cultural expectations I'm not aware of.)
> "Upstream systemd will do X, distros who want to are free to do Y"
Yes this is a good response from upstream. I can work with that, but in that case, even this response didn't get reflected to mailing list discussion, or drowned out instantly.
My question was more general though, questioning systemd developers' behavior collectively (hence the projects' behavior) through time.
The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
You think systemd could be a psyop? Gain influence by paying devs to do the ugly but necessary work, but then sow loads of dissent at the same time...
Speak for yourself, then. I’ve been using Linux since 2004, and the systemd components finally made system management easy. No more arcane init scripts. Handling of service dependencies. Proper timers. Simple configuration files. Administration knowledge that immediately carries over between all systems equally.
As a user, systemd has improved my productivity tremendously.
The kind of bad mouthing developers that work on solutions for complex problems, code that runs on billions of machines, reflects more of your own fragile ego than them.
> The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
> As a user, systemd has improved my productivity tremendously.
Both can be true at the same time. Particularly in the beginning, there was a long string of really important things that used to Just Work that were broken by systemd. Things like:
1. Having home directories in automounted NFS. Under sysv, autofs waited until the network was up to start running. Originally under systemd, "the network" was counted as being up when localhost was up.
2. Being able to type "exit" from an ssh session and have the connection close. Under systemd, closing the login shell would kill -9 all processes with that userid, including the sshd process handling the connection -- before that process could close the socket for the connection. Meaning you type "exit" in an interactive terminal and it hang.
It's been a while since I encountered any major issues with systemd, but for the first few years there were loads of issues with important things that used to Just Work and then broke and took forever to fix because they didn't happen to affect the systemd maintainers. If you didn't encounter any of these, it's probably because your use cases happened to overlap theirs.
Yes, systemd and journalctl have massively simplified my life. But I think it could have been done with far less disruption.
Obligatory XKCD: https://xkcd.com/438
There's no need to be rude. While I'm not anti-systemd; it didn't change my life tremendously, either.
People tend to bash init scripts, but when they are written well, they both work and port well between systems. At least this is my experience with the fleet I manage.
Dependencies worked pretty well in Parallel-SysV, too, again from my experience. Also, systemd is not faster than Parallel-SysV.
It's not that "I had to learn everything from scratch!" woe either. I'm a kind of developer/sysadmin who never whines and just reads the documentation.
I wrote tons of service files and init scripts during Debian's migration. I was a tech-lead of a Debian derivative at that time (albeit working literally underground), too.
systemd and its developers went through a lot phases, remade a lot of mistakes despite being warned about them, and took at least a couple of wrong turns and booed for all the right reasons.
The anger they pull on themselves are not unfounded, yet I don't believe they should be on the receiving end of a flame-war.
From my perspective, systemd developers can benefit tremendously by stepping down from their thrones and look eye to eye with their users. Being kind towards each other never harms anyone, incl. you.
Linux is 34 years old, and some of the Unix-ism borrowed are even older. There is genuine cruft that has downsides. Different relative priorities of backwards comparability, maintainability and the various issues the legacy issues cause are reasonable.
Systemd basically arose out of a frustration at the legacy issues so the whole project exists as a modernizing effort. No wonder they consider backwards compatibility low priority.
That has been their method since the beginning; why would they change from a tactic which works for them?
The central problem with systemd is that they don't want to let you go about your business, they want you to conform to their rule.
Because that's what they've always done, and it continues to work for them?
Systemd doesn't work for me, but it has taken over most Linux distributions, so clearly it's got something people want that I don't understand. That was the case for PulseAudio too.
Did you read the same article?
I think everyone involved would prefer updates to the applications, which fix the issue. Debian opted - for now - for reliability for its users, which fits in their mission statement. On Arch /run/lock is only writeable for the superusers, which improves security. As user I value reliability and security and that legacy tools remain usable (sometimes by default, sometimes by a switch).> It is a security issue
The "security issue" expressed is that someone creates 4 billion lock files. The entire reason an application would have a path to create these lock files is because it's dealing with a shared resource. It's pretty likely that lock files wouldn't be the only route for an application to kill a system. Which is a reason why this "security issue" isn't something anyone has taken seriously.
The reason is much more transparent if you read between the lines. Systemd wants to own the "/run" folder and they don't like the idea of user space applications being able to play in their pool. Notice they don't have the same security concerns for /var/tmp, for example.
they don't like the idea of user space applications being able to play in their pool
i think that is somewhat reasonable. but then systemd should have its own space, independent of a shared space: /var/systemd/run or /run/systemd/ ?
> then systemd should have its own space, independent of a shared space
This would go contrary to an unstated goal: making everyone else to dance to systemd's tune, for their own good.
Unstated, but obvious. From a long while back: [0][1]
[0] <https://lore.kernel.org/all/20140402144219.4cafbe37@gandalf....>
[1] <https://lore.kernel.org/all/CA+55aFzCGQ-jk8ar4tiQEHCUoOPQzr-...>
> On Arch /run/lock is only writeable for the superusers, which improves security.
Does it? That means anyone who needs a lock gets superuser, which seems like overkill. Having a group with write permissions would seem to improve security more?
no that isn't what it means at all
a global /run/lock dir is an outdated mechanism not needed anymore
when the standard was written (20 years ago) it standardized a common way programs used to work around not having something like flock. This is also reflected in the specific details of FHS 3.0 which requires lock files to be named as `LCK..{device_name}` and must contain the process id in a specific encoding. Now the funny part. Flock was added to Linux in ~1996, so even when the standard was written it was already on the way of being outdated and it was just a matter of time until most programs start using flock.
This brings is to two ways how this being a issues makes IMHO little sense:
- a lot of use cases for /var/lock have been replaced with flock
- having a global writable dire used across users has a really bad history (including security vulnerabilities) so there have been ongoing affords to create alternatives for anything like that. E.g. /run/user/{uid}, ~/.local/{bin,share,state,etc.}, systemd PrivateTemp etc.
- so any program running as user not wanting to use flock should place their lock file in `/run/user/{uid}` like e.g. pipewire, wayland, docker and similar do (specifically $XDG_RUNTIME_DIR which happens to be `/un/user/{uid}`)
So the only programs affected by it are programs which:
- don't run as root
- don't use flock
- and don't really follow best practices introduced with the XDG standard either
- ignore that it was quite predictable that /var/lock will get limited or outright removed due to long standing efforts to remove global writable dirs everywhere
i.e. software stuck in the last century, or in this case more like 2 centuries ago in the 2000th
But that is a common theme with Debian Stable, you have to fight even to just remove something which we know since 20 years to be a bad design. If it weren't for Debians reputation I think the systemd devs might have been more surprised by this being an issue then the Debian maintainers about some niche tools using outdated mechanisms breaking.
No, I didn't read the whole article. I follow debian-devel directly. Watched all of it unravel, step by step. I know the resolution since the day it posted to debian-devel.
This was a general question to begin with.
> Their is an option for the old behavior.
The discussion never centered on an option for keeping old behavior for any legitimate reason. The general tone was "systemd wants it this way, so Debian shall oblige". It was a borderline flame-war between more reasonable people and another party which yelled "we say so!"
> It is a security issue and modern solutions to replace exist.
I'm a Linux newbie. Using Linux for 23 years and managing them professionally for 20+ years. I have yet to see an attack involving /var/lock folder being world-writeable. /dev/shm is a much bigger attack surface from my experience.
Migration to flock(2) is not a bad idea, but acting like Nero and setting mailing lists ablaze is not the way to do this. People can cooperate, yet some people love to rain on others and make their life miserable because they think their demands require immediate obedience.
> FHS isn't maintained.
Isn't maintained or not improved fast enough to please systemd devs? IDK. There are standards and RFCs which underpin a ton of things which are not updated.
We tend to call them mature, not unmaintained/abandoned.
> On Arch /run/lock is only writeable for the superusers. As user I value reliability and the legacy tools are usable.
I also value the reliability and agree that legacy tools shall continue working. This is why I use Debian primarily, for the same last 20+ years.
In this case I assume their "fear" is that unprivileged users can exhaust resources (inodes, filesystem space) in an important tmpfs breaking the system. The proper solution for backward compatibility would probably be something like make /run/lock its own mountpoint, but they fixed it in their system (Fedora) so now it's no longer their problem. Just be thankful their software is portable to such strange niche operating systems like Debian. /s
Funny part is that it used to be a separate file system, before Luca decided to kill it. https://lwn.net/Articles/1041948/
It's been a very long time since I heard about uucico (Unix-to-Unix Copy-In Copy-Out program, part of the UUCP suite). Glad to see it's still being shipped! I wonder if any network uses it.
The "somebody might mailicuously exhaust memory via /run” argument seems silly IMHO... you can trivially put a hard limit on its size via the tmpfs mount option. I guess if you bend over backwards that's still a DOS in that new files can't be created in /run once it is full, but come on...
There is support for quotas in tmpfs! /me runs and hides under desk to avoid fruit being thrown at me
Non-clickbait title -- Debian Technical Committee overrides /run/lock permission change
More-clickbaity title -- Debian Technical Committee tells its doofus maintainers to stop worshipping Poettering so much
"DTC serves crow to 2 systemd maintainers with a history of accepting money from Microsoft"
Why is that non-clickbait? Honestly "Debian Technical Committee overrides systemd /run/lock permission change" might be a better title than either, I don't know whether the thing or the actors are more interesting here. But you can only say so much in a title.
Yeah. I expected better from LWN.
IMO the interesting bit here isn't the specific technical change but the interpersonal one of overriding the maintainer(s) decision.
Thus the title reflects the most interesting bit of the story.
Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
> Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
I don't think that's an accurate paraphrase of "Consider this more a passing of the baton from upstream systemd to downstreams: if your distro wants this kind of legacy interface, then just add this via a distro-specific tmpfiles drop-in. But there's no point really in forcing anyone who has a more forward-looking view of the world to still carry that dir."
That kind of drop-in is pretty routine, so I don't know why this became a big thing we're all discussing now.
Well yes, but the complication is that Luca is both systemd developer and debian developer, so the passing of the baton did not really happen here.
"everyone else" in this case is pretty much only the debian ecosystem because they insist on enforcing a serial lock policy from the 1980s. It's fine if Debian wants to move at the speed of a Soviet committee but I don't think it should be expected (or would be healthy) for systemd to move at the same pace.
A software developer's primary job is to develop software for their users, not to comply with a third party distributor that repackages their software.
The beef isn't with systemd upstream which already has a very simple/boring workaround for this, it's with the debian package maintainer (some people here are wearing multiple hats).
Really the whole raison d'etre of debian is move at this pace to prioritize stability/compatibility. If you don't like that philosophy there are other distros but a package maintainer's primary job is to repackage software for that distro (which presumably users have chosen for a reason), not comply with upstream.
Wow, talk about making a mountain of a mole hill.
Letting upstream systemd single-handedly define what directories exist with what modes in your distro has never been the intended Modus Operandi.
Debian has a huge selection of packages available for it and clearly is going to have more headaches when it comes to preserving compatibility with all that software.
This is a trivial matter for Debian to handle appropriately, while systemd stays focused on its current priorities. I'm surprised this is being talked about at all outside the appropriate mailing lists... slow week for linux news?
Somehow I feel that if all the time that has been invested in debating and discussing this had been spent on patching the affected apps, the problem would be properly solved.
I mean, yeah, I get it, systemd bad, democracy good, but these world-writable lock folders are actually a huge pain, and adding some shim code to upgrade to a more secure solution seems achievable?
Genuinely curious - why would world-writeable directory be bad for security? Assuming of course, it's on a separate filesystem mounted with sensible options. Here's what I see from "grep /run/lock /proc/mounts" in sid:
The classic is say you know a root process will write a file called foo.lock in /run/lock, and you (a bad person) have write access to that directory. Then you make foo.lock a symlink to some file (/bin/init or /bin/sh or ld.so for example would be very inconvenient choices) and when the root process writes its lock it destroys that file.
Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
> and when the root process writes its lock it destroys that file.
Unless you do open("/run/lock/foo.lock", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW)
Yep. And for good measure, first open with O_CREAT as tempfile with random name, then rename() it to predictable "foo.lock".
Yup to both of you. But all of this is to say, running shellscripts as root (in particular) needs to be done with extreme care, because if people forget those precautions when writing C, they sure as heck don’t trouble themselves to do it when they’re writing shell.
I remember the time (around 2001-2002) when just about every binary was discovered to have some variant on this exact exploit. I happened to be linux sysadmin for a very large, high-profile set of linux boxes at the time. Happy times.
> Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
Annoying side effect: now you gotta guess which process created the darn lockfile.
A more sensible approach is to do sanity checking on the lockfile and its contents (i.e. does the contained PID match one's own binary).
The argument is also that you could effectively DoS the system by exhausting space or inodes.
Hmm - I see there's now "lockdev" for managing access to things like serial lines, but what's the preferred method of expressing "only one instance of this program should run at any one time"?
I don't know what the preferred method is. But so far, flocking on my own executable works for me.
Debian discussions make political discussions seem quick and fast acting by comparison
> He said that he uses cu ""almost constantly for interacting with embedded serial consoles on devices a USB connection away from my laptop""
Whyyyyyyyyyyyyyyy
There are a million better ways of doing this.
I use GNU screen for that. Looking at cu, it looks to be just as tiny and has ssh-like tilda-escapes, including "~." to disconnect. Nice, gotta try it out, thanks!
I got used to using minicom way way back. Has a nice TUI and can do stuff.
I only moved to using screen about 2 years ago after over 2 decades of minicom
cu might be good for anything that you have a fixed config, otherwise I'll just go with minicom
What would you suggest? Personally I find cu the closest to ssh in ergonomics to any of the tui serial terminal programs, and it’s easily available across unixes like FreeBSD or Mac OS, which is a bonus.
cu is the regular way of doing it
I don't see the problem. Minicom and even picocom are bloated compared to cu
One comment [0] highlights a point in favor of the current implementation:
> create a lock file for every dial-in line to prevent its use by programs looking for a dial-out line.
[0]: https://lwn.net/Articles/1042594/
As always, it will just take another decade until debian has figured it out.
Considering I never had to reinstall a Debian system because it got bloated or broke one day, I can accept this slow-cooking approach. Even support them on this regard.
Gentoo Linux has quite a different approach than Debian [0] but after the first month or two (once my new-to-Linux ass figured out what it was doing) I've never had to reinstall any Gentoo system. [1]
If you want what Debian provides, it's a poor choice for you... but -IME- it doesn't break on upgrade, unlike some Debian-derived distros I've tried in the past.
[0] Something along the lines of "Always try to package exactly what's provided by upstream, try hard to get distro patches upstreamed, and try to have the latest available upstream release in the 'testing channel'.".
[1] Well, I do have a machine that (aside from "side-loading" kernel updates from time to time) hasn't been updated in four years. While I'll try to update that one in the normal way, I'm probably going to need to reinstall.
Yes, the slow-cooking approach is one of the main reasons why I prefer Debian.
once it's ready they'll push it, that's how it should be
I am in no rush. I like something being stable.