Debian 13, Postgres, and the US time zones

(rachelbythebay.com)

195 points | by move-on-by 12 hours ago ago

81 comments

  • JdeBP 6 hours ago

    This does not trace things back far enough. The root is where IANA has long since segregated out a set of timezone file names into a "backward" collection:

    * https://data.iana.org/time-zones/tzdb/backward

    If one traces references, one finds this connected bug on Launchpad. Amusingly to anyone who has ever seen these sweeping timezone database changes over the years, Launchpad marked it as "This bug affects 1 person.".

    * https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/200807...

    The rules for the "backward" file are here:

    * https://data.iana.org/time-zones/theory.html#naming

    All of the US/* timezone names, such as US/Pacific here, have been backwards compatibility measures in place for the whole of the 21st century and some of the late 20th. The Olson database in the 1980s (mod.sources v08i085, comp.sources.unix v14i030) used these names. But the naming scheme changed somewhen in the 1990s to a continent/city and ocean/city form and backwards compatibility with the old names has been preserved ever since by the "backward" file.

    * https://groups.google.com/g/comp.unix.solaris/c/ON_MPZxVdv0/...

    Debian has moved into a non-depended-upon package a backwards compatibility measure that is as old as Debian itself.

    • AnssiH 6 hours ago

      > Launchpad marked it as "This bug affects 1 person.".

      That just means no one has clicked "affects me too" button yet (after logging in).

    • ForHackernews 5 hours ago

      It's good Debian is making this change now to be prepared when the legacy United States are downed at the end of 2052.

      • tempodox 2 hours ago

        I can’t tell whether you’re joking. Seems like a distinct possibility now.

        • maybewhenthesun 24 minutes ago

          2052? I hope the last 2 digits aren't swapped...

  • roelschroeven 5 hours ago

    > The worst part about this is that it didn't get so much as a mention in the Debian 13 release notes. I read through that document before going for it and never encountered it. Indeed, even now, you won't find "tzdata" or "zone" in it.

    I had another issue during my upgrade to Debian 13 that also wasn't mentioned in the release notes. I filed a bug report, but I was told that the issue was not important enough to put in the release notes, and I should have instead more closely read NEWS.Debian of the package. So I don't really know anymore what the "Issues to be aware of" chapter in the release notes is for, because apparently it's only a small selection of issues you need to be aware of.

    • styanax an hour ago

      Sharing undocumented gotcha: nginx default config (/etc/nginx/nginx.conf) now has `server_tokens off` set with a comment that it's "common practice" (agreed). This is not in upstream's git version of the file[1], I therefore presume it's a Debian maintainer change.

      The operational upgrade failure is if you have any existing drop-in config that sets `server_tokens off` already a hard fail Nginx error about duplicated keywords will occur, causing the apt process to exit with failure code(s) during the standard upgrade process.

      [1] https://github.com/nginx/nginx/blob/master/conf/nginx.conf

    • msk-lywenn 2 hours ago

      What was your issue ?

      • roelschroeven 2 hours ago

        The version of pdns-recursor in trixie uses a different configuration file syntax then older versions.

        (There is a tool to convert the old configuration syntax to the new, but it requires a working installation. There is a command line option to enable support for the old format, which if nothing else helps to be able to run the conversion tool, but there's no information how to enable that command line option in the way Debian starts pdns-recursor.)

  • yellers 6 hours ago

    Hang on a second, "(...) in 2023. US/* was moved to tzdata-legacy (...)"

    US/* was moved to 'backward' (the file for backward compatibility) in the tz database in 1993(!) and as such was essentially marked as deprecated long enough. https://data.iana.org/time-zones/tzdb/backward

    You're telling me you didn't notice ? It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.

    • mynegation 2 hours ago

      Not only did I not notice, I have never known that country prefixes were a thing, having to deal with tzdata since 1999. I wonder if that timezone was typed in manually? I doubt Postgres 15 file contained it to begin with.

    • zahlman 5 hours ago

      > You're telling me you didn't notice ? It was...

      In a large fraction of cases in the FOSS world, it comes across that the developers really do want to communicate this sort of thing, but there's no clarity on where or how they should do so. See for example various deprecations in Python packaging tools (and standards).

      • JdeBP 5 hours ago

        In this case, they did communicate it, and the aforegiven Vogon reference is a mischaracterization. The naming convention is in the current IANA doco and Eggert copy.

        * https://data.iana.org/time-zones/tz-link.html#tzdb

        * https://web.cs.ucla.edu/~eggert/tz/tz-link.htm#tzdb

        Paul Eggert explained the continent/ocean plus largest city naming convention on a WWW page almost a quarter of a century ago. The WWW page was so well publicized that you can find its URL baked into at least four of the O'Reilly animal-cover books from the early 2000s.

        * https://web.archive.org/web/20011023074744/http://www.twinsu...

        It was explained on Usenet and on mailing lists prior to that.

        • bbarnett 4 hours ago

          You know, the tzdata people are quite haughty. They claim to store all that change, accurately, and yet here we are.

          An example of this falsehood? Well, in the 70s my father convinced most of my hometown, at least the portions between Main St and Wharf, that DST was absurd.

          For almost an entire year, this was observed.

          Do you think they kept record in tzdata? I tried to convince them, but no! I still have some dateplanners my father had printed up, and even a picture of the sign that was out on the road (to alert visitors!).

          But no!

          Do not trust the tzdata people. As you can see, they are not so accurate.

          • macintux 3 hours ago

            Once upon a time I stumbled upon a small book of maps showing the time zones in Indiana each year in the mid-20th century. It was fascinating to see how various towns and cities would jump back and forth between Eastern and Central year to year, with little regard for the surrounding rural areas.

            I frequently kick myself for losing track of that book.

      • rollcat 5 hours ago

        Then you get the reverse. I just upgraded to macOS Sonoma (yes I'm always one major version behind with Apple stuff...), and I was annoyed as heck when I had to click through "Look what's new in Calendar!", "Look what's new in Reminders!", "Look what's new in StripClubs!"... I need to use my software right now, I will not read this. Then I will forget it ever popped up, and will not read it in the future either.

        • thesuitonym 11 minutes ago

          There's a vast difference between Apple showing you all it's useless new AI integrations and developers telling system administrators what they need to know.

    • cpburns2009 35 minutes ago

      Why are time-zones even prefixed by continent? Country-prefixed time-zones make more sense because they're defined politically.

      • eurg 26 minutes ago

        Cities may find themselves in other countries easier than on other continents.

      • hnuser123456 25 minutes ago

        I disagree. Country borders can move. I have not heard of a city moving between continents however.

  • Tepix 5 hours ago

    Pro tip: Use the command

        timedatectl list-timezones
    
    to get a list of all timezones your debian based OS recognises.

    For your convenience, here is a list for a Debian 13 box with 628 entries:

    https://pastes.io/output-of-timedatectl-list-timezones-on-de...

  • nja 7 minutes ago

    I'm still not totally sure why these names were deprecated in the first place...

    I mean, the folks who run the tz db definitely know what they're doing, it just never 100% clicked with my thinking.

    I always prefer `US/Eastern` over `America/New_York` -- it seems more "canonical" to me. New York is _currently_ the anchor city for ET, but will it always be? The place I live (Boston) is currently on ET, but in the future it might be on Atlantic Time. If there was an `America/Boston`, I would use that to be safe, but since there isn't, it just seems better to be to be specific that I mean "Eastern Time" and not "whatever the time is in NYC"... At least then if Boston switches to a different tz, I could intentionally switch to "Atlantic Time" -- doesn't that make more sense? Versus I guess what I'd have to do, which is switch to `America/Puerto_Rico`? (I had to actually search that one, too bad there's no `US/Atlantic`...)

  • acdha 3 hours ago

    I ran into a similar bug in Git Lab where the v18 upgrade didn’t replace now-unsupported time zones so things like scheduled pipelines simply stopped running and were labeled inactive in the UI without explanation.

    https://gitlab.com/gitlab-org/gitlab/-/issues/556779#note_26...

    The fix is simply to change them to the non-backwards linked names but it caused some confusing API errors since data which would no longer pass validation was already in the database and didn’t look wrong.

  • evdubs 9 hours ago

    I also ran into this using:

    - Debian 13

    - Interactive Brokers' Trader Workstation

    - Racket's `gregor` date and time library

    IBKR still sends the old, deprecated US/* timezones. As noted in the article, the solution is to `apt install tzdata-legacy`.

  • op00to an hour ago

    I feel like I’ve been using America/New_York for a decade at least. Am I misremembering?

    • db48x 4 minutes ago

      No, that’s been the standard name for a quarter century.

  • reactordev 2 hours ago

    Ooofff. The two difficult things in software engineering, naming, and timestamps.

    This hit me in the early 2000s and now everything I do is in UTC. All dates, timestamps, everything, UTC. If you want to look at a local window in time, convert the window to a utc start and end date and search. When viewing, use a js function to translate the utc date to a local one to print. The mental gymnasium of local to utc to local again…

    • dotancohen an hour ago

        > The two difficult things in software engineering, naming, and timestamps.
      
      And off-by-one errors. The two most difficult things in software engineering: naming things, timestamps, and off-by-one errors.
  • poemxo 7 hours ago

    I don't get how this is a snag, it's right there in the log.

    • skylurk 4 hours ago

      Agreed.

      > At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

      I get this, but now you get to laugh and dust yourself off. If it had silently started doing the wrong thing, that would be a worthy complaint.

  • burnt-resistor 5 hours ago

    Always run production systems in the Etc/UTC timezone. This eliminates an entire class of problems while only creating minor inconveniences.

    • bityard 41 minutes ago

      I work for a company with servers, employees, and customers in basically every time zone around the world. And yet every server, internal tool, and workflow uses Pacific time. UTC is used precisely nowhere. Setting aside the issues of DST, I imagine it's convenient for the employees and managers in HQ but absolute madness for everyone else.

      • a012 7 minutes ago

        Are you at Google? Because they also keep using US/Pacific timezone in every incidents where it affects everyone over the globe

    • macintux 3 hours ago

      I work with a development team who manages data integration and migration for massive datasets, and they have sensibly set a standard that every date/time value they store in their databases will be UTC.

      But they explicitly or implicitly have also decided not to store the timezone in the strings, so every single value is technically ambiguous. Absolutely drives me crazy.

      Update: since there have been questions, these are strings, not native datetime values.

      • shawnz an hour ago

        Postgres always stores date/time values in UTC (if they are timezone-aware)

        • macintux an hour ago

          These are strings. And not Postgres.

      • ta1243 2 hours ago

        > they have sensibly set a standard that every date/time value they store in their databases will be UTC.

        Not sensible at all for future date/times.

    • Tepix 5 hours ago

      For servers, i agree 100%. For desktop systems, not so much.

      • kawsper 5 hours ago

        How else could you tell what time it is on the servers?

        • raverbashing 4 hours ago

          That's easy, if you're Walmart just use the TZ of your corporate HQ

          (That's what I heard anyway)

          • kstrauser 2 hours ago

            My non-Walmart startup was expanding from California to other regions many years ago, and there was some debate about whether we’d use UTC or America/Los_Angeles globally. I was part of Team Over My Dead Body.

          • madcaptenor 14 minutes ago

            I work for a different company headquartered in the central time zone and we also do this. On some systems.

          • ThePowerOfFuet 2 hours ago

            Google's internal timestamps are in Pacific Time, too.

      • actionfromafar 5 hours ago

        Eastern Standard Tribe, represent!

        • kstrauser 2 hours ago

          I appreciated the reference.

    • bayindirh 3 hours ago

      You can also run in UTC+$YOUR_OFFSET timezone if you don't use DST.

      On the other hand, UTCx timezones are not silver bullets if your fleet is a part of a multi-continent federation.

    • bornfreddy 5 hours ago

      It also makes sense. Timezone is user specific - if you have users from all over the globe, they will need different settings, so this should be set in frontend.

  • loloquwowndueo 2 hours ago

    Imagine my outrage when America/Montreal was deprecated years ago. “It’s the same as America/Toronto, just use that” they said :)

    • ta1243 2 hours ago

      Have Montreal and Toronto ever been in different timezones?

      • loloquwowndueo 2 hours ago

        No they have not. But why not nix Toronto instead? Everybody knows Montreal is better, it should get dibs at naming the time zone.

        I’m just joking all the way, by the way :)

        • ta1243 38 minutes ago

          Toronto's bigger. Same reason Boston and Washington are "New York" and Seatle/SanFrancisco are Los_Angeles

          (Detroit gets its own zone because it was significantly different to new york before 1970, normally it's only difference post 1970 which merit a new zone)

  • ggm 11 hours ago

    Is there a problem with ISO3166 denoted information in general or is there a specific US issue here? I would think ISO code denoted tzdata was a public good in some sense.

    • mananaysiempre 7 hours ago

      This has nothing to do with ISO 3166 at all. The tzdata database is the work of a single person, Arthur Olson, not a committee (or rather it was, from its birth in the 1980s until 2011 when a company decided to sue him for no reason).

      And for most of its life it’s been an explicit policy that timezones are named by continent and representative population center, not by country, to avoid entangling it in territorial disputes and improve naming stability for historical data. The US/* (and Canada/*, etc.) names are deprecated and have been since 1995 (?), but apparently people were still using them because the deprecation wasn’t really apparent unless one was especially into reading release notes.

    • masklinn 8 hours ago

      All the legacy time zones were moved out of the default zoneinfo install. It’s not a us-specific issue but the legacy US/ timezones remain in widespread use, and they stop working on Debian 13 ootb (possibly Ubuntu noble as well?).

      • pixelesque 7 hours ago

        Does anyone know why they are still in widespread use?

        Config defaults somewhere still using them? Man page examples? Tutorials using them? Or just force of habit?

        • zahlman 5 hours ago

          > Does anyone know why they are still in widespread use?

          Because of a lack of things compelling people to change them until it causes a breakage. And then when it does cause a breakage, most people would rather move heaven and earth to complain, research workarounds, etc. rather than just change it. (Institutional structures can also make "just" changing it far harder than that should be.)

        • rtpg 7 hours ago

          Some of this is surely just muscle memory or intertia as well. I remember random config values from when I was trying out linux boxes back in high school that I replicated into files that just don't get touched for decades afterwards.

          When was the last time you rebuilt your company's postgres config from scratch?

          • Symbiote 3 hours ago

            > When was the last time you rebuilt your company's postgres config from scratch?

            Last year, when we upgrade to version 17.

            I looked at the example/template configuration, diffed it with our configuration from PG15, and for every change decided whether to keep our version or the new setting.

            I didn't use it, but Debian/APT has had a tool to do this sort of comparison for any software upgrade for as long as I can remember.

            Do other people just copy the old config and shout "YOLO!"?

          • magicalhippo 2 hours ago

            Still typing "nano -w filename" each and every time since back around y2k when I was working on Linux for the first time I was told that bad things could happen if I didn't...

        • acdha 3 hours ago

          There’s definitely inertia but I think it’s also that the US/ names match official usage: nobody, not even residents, says the time zone is New York because the official name is Eastern time.

          • cpburns2009 41 minutes ago

            Exactly this. I still don't know if there's a technical difference between America/New_York, America/Detroit, or America/Indianapolis.

        • fredoralive 5 hours ago

          I suspect some of it will be because the legacy form is a bit more intuitive than the standard form. You don’t really use continents and cities as a reference to time zones normally, countries and local subdivisions makes more sense, but as other people note, it brings up POLITICS.

          • MartijnBraam 5 hours ago

            You don't use them normally in the US, I've been referring to europe/amsterdam or europe/paris all my life in Linux installers and various equipment. I've never ever encountered netherlands/amsterdam or something like that.

            • Symbiote 2 hours ago

              From the list of deprecated zones, we could have been using "GB", "Poland", "Portugal", "CET", but that's about it. "Netherlands" didn't exist.

              Given the old names were deprecated in 1993, it's hardly surprising that I never before discovered "GB".

        • Macha 4 hours ago

          I wonder how much of an influence it is that US/Eastern is easier to type than America/New_York

      • pdw 7 hours ago

        Are they in widespread use? They were deprecated in 1995.

        • ahoef 7 hours ago

          I don't see conflict in those statements.

    • evdubs 9 hours ago

      You can see the zones that are deprecated at https://packages.debian.org/sid/all/tzdata-legacy/filelist

  • 1oooqooq 3 hours ago

    love the takes on this one blog, but "I'm copying an old conf file for over 20yrs and now got this issue" is kinda weak.

    American exceptionalism time zones aren't used since the 90s. even the cpus from that time are already dropped from kernel support. heck even the text encoding is gone.

    • kstrauser 2 hours ago

      That’s unkind. I want to do the right thing in such cases, but I’m also learning about this today for the very first time. I’ve never, not once, heard that US/Pacific was a bad idea until this post. If not for this, I still wouldn’t know. I thought it and America/Los_Angeles were semantically identically and just kind of symlinks to PST8PDT or whatever.

      If anything, the city TZ always felt off, like I was opting in to that specific city’s strange legal decisions or something.

      • Symbiote 2 hours ago

        For general user interfaces, it's for the user interface to show "US Pacific" or "UK" rather than America/Los_Angeles and Europe/London.

        The timezone selector in KDE shows "Los Angeles | America/United States of America | Pacific" and "London | Europe/United Kingdom", for example.

  • raziel2p 3 hours ago

    > At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

    You're running a database system and you just casually comment out the configuration setting the timezone?

    In what way did everything "seem fine"? SELECT 1 returned something? No further investigation required??

  • donalhunt 8 hours ago

    What sort of monster are you by not defaulting to UTC on systems? (⊙_⊙') /s

  • pfexec 6 hours ago

    Debian is known to have made similar monstrously stupid decisions.

    For example, they patch OpenSSH source code in a way that makes defaults behave differently than upstream. In the name of backwards compatibility of course.

    I assume this will continue until it doesn't anymore, and the only notification you shall receive from the ivory tower is a cryptic one-liner buried in a changelog somewhere.

    • traceroute66 4 hours ago

      > For example, they patch OpenSSH source code in a way that...

      Isn't it the same thing with the RedHat downstreams ? (Not necessarily OpenSSH but other packages)

      IIRC RedHat do all sorts of things to keep their gov / corp customers happy, also usually in the name of backwards compatability, all of which then end up in the downstreams.

    • burnt-resistor 5 hours ago

      Russ Allbery left over bureaucracy and systemd. It sounds like it's chocked full of people who want power and an excuse to patch downstream to create a cottage industry of quirks, busywork, and codependency.

      I prefer real choice and light patches that try to upstream as much as possible, or workaround upstream obstinacy rather than create incompatible idiosyncrasies. One area that isn't well represented in barely a/no distro is init freedom neither married to nor completely divorced from the sprawling octopus.

      • actionfromafar 5 hours ago

        I don't think you can be partly married to Cuthulhu. (Systemd.)

        • styanax 4 hours ago

          [ eldritch noises intensify ]

          As an actual answer, it's not too bad on Debian; we only really use/need: systemd (system and user), -logind, -journald, -udevd. All in all, not too many tentacles but there are a few...