Zig ELF Linker Improvements Devlog

(ziglang.org)

219 points | by kristoff_it a day ago ago

91 comments

  • bpavuk 21 hours ago

    I am so used to thinking that Zig, Rust, and the likes are only viable in niches where C is viable, but no. not anymore at least - once this linker and incremental compilation on other targets land, Zig will become THE C replacement and that will let me iterate at the speed of JS or Python with performance of C or Rust. even Andrew's initial dream - to create a DAW with uncompromising UX - will become much easier to create. once someone creates a Zig-native immediate-mode or reactive UI framework, that is.

    I am still a little salty about `@cImport` removal, though! without it, I can't confidently call it "Kotlin of C" anymore.

    • kllrnohj 15 hours ago

      > that will let me iterate at the speed of JS or Python with performance of C or Rust.

      Didn't Go already do that?

      > I am so used to thinking that Zig, Rust, and the likes are only viable in niches where C is viable, but no. not anymore at least - once this linker and incremental compilation on other targets land, Zig will become THE C replacement

      Yes, and it will still only be useful in the same niche that C is because the entire philosophy of Zig is to essentially be like C. You're never going to interate at JS/Python speeds with Zig because you'll always be wrangling with memory lifecycles, object lifecycles, etc...

      Rust is significantly different.

      • dapperdrake 15 hours ago

        Go is for the cases where GC works for you (many, many cases).

        Zig is for when you need control over the allocator (also many such cases).

      • bpavuk 7 hours ago

        > Didn't Go already do that?

        no. GC pauses turn any serious systems work into hell.

        > Yes, and it will still only be useful [...]

        this does not exclude the possibility of creation of libraries that manage everything for me within their domain of responsibility, such as dvui

        • kllrnohj 4 hours ago

          > no. GC pauses turn any serious systems work into hell.

          did you miss the context I was responding to? They were comparing against JS & Python and rapid iteration speeds. That's obviously not "serious systems work" where GC pauses matter

          > this does not exclude the possibility of creation of libraries that manage everything for me within their domain of responsibility, such as dvui

          Sure, but if you're philosophically okay with hidden allocations and control flow, why not just choose a language that doesn't compromise itself for those ideals in the first place?

          • bpavuk an hour ago

            > They were comparing

            that was me comparing, please kindly read usernames next time :)

            > why not just choose a language that doesn't compromise itself for those ideals in the first place?

            I might want to wrap a timing-sensitive machinery into some nice UI. Rust has Tauri for that, sure, but now we are bringing npm and have zero chances of having GPU-accelerated UI without a crap-ton of fuckery which would be easier in Zig. another path for resolving that same issue is Compose Desktop + Project Panama, but then you are dealing with data marshalling, FFI boundary, and manual resource management in an environment that does not expect this.

            so, here is a genius idea: why not have everything in one language? C++ already does that, much like C, but Zig does that so much better, and incremental compilation time is one of the more practical and immediate developer UX benefits it provides. Rust? good luck having shared mutable state there.

          • wolttam 3 hours ago

            Because they like and are excited about Zig.

      • vips7L 11 hours ago

        Go is a terribly verbose language.

    • akazantsev 18 hours ago

      > Kotlin of C

      That sounds good on paper. But as a guy who tried to learn Kotlin and only it. It comes with baggage to learn Java to use its libraries because... You know... they interact seamlessly and stuff. In the end, for a new learner, it might actually make things harder.

      Nothing about Zig and C here, just a bit salty from my experience with Kotlin.

      • vanviegen 2 hours ago

        Kotlin is a terrible language for learning, as it has a lot syntactic shortcuts that are easy to mistake for magic. I think it's actually easier to learn some Java first, as it's simpler and teaches you the semantics both languages (mostly) share.

      • vips7L 11 hours ago

        Kotlin also has its own features that differ from the way the JVM or Java has decided to develop them. For example: coroutines vs virtual threads or Kotlin “value” classes and Java value classes. The semantics don’t match and Kotlin stops becoming simply a “better Java”.

        • pjmlp 8 hours ago

          That is the curse of guest languages, see C and C++ as well.

          That is why I always say keep with the platform, and why despite my endless rants on C, I keep myself up-to-date in regards to it.

          Eventually they always diverge.

      • bpavuk 17 hours ago

        Kotlin <-> Java interop is a totally optional topic you could have skipped over, telling you as a Kotlin developer (who recently had to switch to .NET because Ukrainian job market is fucked up). besides, Java itself isn't that hard if it's Java 17 or newer, and it's rather good if it's Java 25 or newer

        • eddythompson80 16 hours ago

          > Kotlin <-> Java interop is a totally optional topic you could have skipped over

          This is the same place F# has been stuck in. It’s a great language on its own, but you can’t just use F#. Every F# must also do C# interop. It’s too 100% optional in theory, but never in practice. The best CLR/JVM libraries for anything are Java/C# ones. You need to interop with them to develop practical Kotlin/F# applications. You can limit yourself to the Kotlin/F# ones, but then you’re artificially limiting yourself to experimental libraries at best. You will find yourself needing a charting library, a DNS library, an SMTP library, an AWS SDK or a rabbitmq SDK. The best ones are gonna be Java/C#. Yes, you can always find a random GitHub repo for a “Kotlin-native X”, but the Java X library is a thousand times more mature, stable, performant, feature rich, etc. Same problem with F#. And the “glue” code is so “straight forward”, why would any one bother?

          • demilicious 9 hours ago

            I have quite literally never had to do Java interop in my Kotlin work. Do you have an example?

            In contrast, your statement about F# strikes me as mostly true - albeit my interop was always the other way around (consuming F# code from C#).

            • pjmlp 8 hours ago

              Calling Koltin co-routines directly from Java code, as an example.

              There is a reason Android team has Java friendly guidelines.

      • dapperdrake 15 hours ago

        Clojure also "comes with the baggage" of learning Java libraries.

        And JavaScript comes with the baggage of learning about web browsers and NodeJS's "fs" module.

    • alexboehm 18 hours ago

      It's already sort of possible. https://codeberg.org/fellowtraveler/flux here is my Zig DAW. It has been amazing for the audio engine, but the ui is currently using imgui.

      • lukaslalinsky 8 hours ago

        I've tried building your project, but hit problems due to dependency hash mismatches. Do you have a screenshot somewhere?

    • dnautics 17 hours ago

      idk, making @cImport just "@import" is an improvement imo.

      • bpavuk 17 hours ago

        `@import` that you have to configure in the build system first.

        this makes porting projects gradually, file by file, rather cumbersome. now I have to rewrite quite a lot of Chocolate Doom because my port was halfway there and then @cImport got gutted... or keep going with Zig 0.16.2 until it's either 100% Zig or has little enough files that upgrading won't make my build.zig file implode in lines of code

        • dnautics 16 hours ago

          i would have guessed easier, since it makes c modules and zig modules ~indistinguishable?

    • gliptic 19 hours ago

      > Zig-native immediate-mode

      dvui?

      • bpavuk 17 hours ago

        many thanks, will look into that...

    • Mawr 13 hours ago

      > Zig, Rust, and the likes are only viable in niches where C is viable

      Pretty much correct, yes.

      > Zig will become THE C replacement and that will let me iterate at the speed of JS or Python with performance of C or Rust.

      Fundamentally impossible. C/Zig/Rust have 100% performance as a top goal, which has to be traded off with something else and that's always realistically going to be programmer work/effort/time.

      You can't have a house built 100% fast but also 100% cheap and with 100% quality. At best you could cleverly abuse the law of diminishing returns and aim for ~80% in all three areas. That's basically what Go's trying to do.

      > once this linker and incremental compilation on other targets land,

      In any case, why would a better linker and faster compile times of all things achieve this supposed goal?

      Beyond being low level, Zig is still pretty memory unsafe and has you make choices about each allocation, making it unappealing as an applications language. Zig and Python are completely different worlds.

      • bpavuk 7 hours ago

        > At best you could cleverly abuse the law of diminishing returns and aim for ~80% in all three areas. That's basically what Go's trying to do.

        then how does Zig achieve ~90%?

        > In any case, why would a better linker and faster compile times of all things achieve this supposed goal?

        sub-100ms rebuild is actually more important as the project grows. when you iterate, you think differently. picking different colors or fonts or whatnot becomes much cheaper, so you are more willing to try

      • spease 13 hours ago

        > You can't have a house built 100% fast but also 100% cheap and with 100% quality.

        That’s essentially what technological development does, is make those tradeoffs more favorable.

        • tovej 2 hours ago

          No? The conventional engineering wisdom has always been: pick two out of fast, cheap, or good.

          And technological improvements don't remove these tradeoffs. You still have to balance tradeoffs. The best thing you could do is have a mix of technologies that choose different tradeoffs. A single technology (zig) won't do.

  • setheron 21 minutes ago

    Is there a design dock or explanation about how it can do incremental linking?

    It's evaded other linkers in the past: gcc, llvm, mold etc....

  • onlyrealcuzzo 18 hours ago

    I've been building a memory safe language that transpiles to Zig with a Go-like runtime that can run interpreted (no GC) or compiled - high-level that feels like Ruby but with incremental typing like TypeScript.

    The Zig team between 0.16 and this has really made me glad I chose Zig as the target instead of Rust - which probably would've been a lot easier to target (since it's already memory safe).

    I believed it had the best build system design and was the best transpilation target, and I really believe that 6 months later.

    The main reason I wanted no GC is because I think aliasing is the root of all evil, and I want a language with zero global complexity (but doesn't require a PhD to use).

    • keithasaurus 17 hours ago

      Working on something kinda similar. No GC, Python feel, managed memory, performance approaching C. It's here: https://blorp-lang.org if you want to compare approaches.

      • onlyrealcuzzo 17 hours ago

        It looks pretty cool!

        It's not clear how much concurrency is part of what you're trying to solve.

        All I could find is this: https://blorp-lang.org/docs/concurrency/ - which doesn't give me much as to how you handle shared memory, safety, deadlocks, etc.

        Definitely down to chat more - looks like you've got some traction, which is impressive and awesome!

        I'd love to pick your brain as it appears you're further along than I am.

        • keithasaurus 16 hours ago

          Yeah, concurrency in blorp doesn't allow shared mutable references, so deadlocks aren't really a concern. Otherwise it's meant to be simple-ish -- virtual threads, channels, no async/await. Pure functions allow safe parallelism naturally, so that's fairly straightforward, though the API is still incomplete, for example the "Parallel" section here: https://blorp-lang.org/docs/lists/. It's still under heavy development (working on it right now).

          What are the over-arching goals of your language?

          • onlyrealcuzzo 16 hours ago

            Right on.

            1) I want to minimize global complexity, which by definition maximizes local reasoning.

            2) I want to make the vast majority of bugs simply unrepresentable - taking it past Rust, and even past Pony - WHILE allowing shared mutable memory, but without requiring a PhD to use.

            The goal is in EASY mode, it's barely harder to use than Ruby or Python (just the occasional pedant compiler error that has automatic options to fix itself most of the time). You don't even have to supply types or compile. It has a REPL, etc.

            When you bump to DEFAULT mode and then to STRICT mode, all the annotation is automatic - your code just might look "ugly" if you like having no types anywhere etc.

            But DEFAULT & STRICT mode give people and LLMs everything they need to know to understand the effects of an individual function.

            • keithasaurus 16 hours ago

              I have some similar goals. Have you considered leaning more into inference than gradual typing? One pattern I like is allowing the compiler to develop a more complex mental model, but keeping it straightforward for users -- you can do that with inference, ownership, purity, effect types, etc. What I actually think is really tantalizing is using tooling to fill in some of those gaps -- for instance, the editor could know types, required capabilities etc, without the user ever needing to type anything, but when the user needs it, they can find it, query it, test against it.

              • onlyrealcuzzo 14 hours ago

                Cool - it sounds pretty similar. It's interesting that it looks so different. I'll have to investigate more.

                WRT to inference, yes. I infer everything in EASY mode.

                And the compiler give the user autofix via choice when a type is ambiguous (I don't default to huge union types - I assume no one wants to do that and make them choose a type - may potentially allow AutoUnion to allow that).

                I couldn't tell if you're using affine ownership, but I assume so if you don't have a GC. If they try to create an alias - they get a use after move error, and the compiler tells them they need to either COPY (auto-fix) or create a RefCount (usually auto-fix) - they pick.

                • keithasaurus 9 hours ago

                  No, blorp doesn't use affine types (one or zero uses). In blorp, ownership is not explicitly controlled by users at all, so it's opaque. Under the hood, it's perceus for compile-time ownership and borrowing and automatic reference counting with copy-on-write optimizations for the runtime. This is made reasonably easy for the compiler to reason about in blorp because semantically it doesn't _really_ have in-place mutation -- `var` really means "re-bindable" to a new value; and then under the hood we'll mutate in place where we can.

                  • onlyrealcuzzo 3 hours ago

                    Interesting.

                    How do you prevent data races in concurrent code?

                    Or do you not allow shared mutable memory?

                    • keithasaurus an hour ago

                      Yeah, no shared mutable memory; coordination is done via channels.

                • keithasaurus 8 hours ago

                  How is compiling to zig? I considered doing it but chose C instead because of how much zig is still changing. I've considered using it's C compiler for targeting multiple platforms locally.

                  • onlyrealcuzzo 3 hours ago

                    I would HIGHLY recommend transpiling to Zig.

                    Comptime is very powerful and awesome to use for a transpilation target.

                    Also, Io and how Zig handles allocation and gives you full control of the allocator make it super easy to do things you probably want to do in your language.

  • librasteve 20 hours ago

    There has been some speculation about porting the Raku backend (Meta-Object Aware Runtime Virtual Machine - MOARVM)from C to Zig. For example the wider set of Zig Hash options could be a big optimization.

    Since you ask, the front end is self hosting in NQP and with the ripening RakuAST project increasingly in Raku Grammars. The new AST (6.e.PREVIEW) will bring much better introspection and high level optimization handles. So the potential to refactor/rewrite the VM for substantial speed gains is wide open.

    Anyway those with skills and interest are welcome to join the -Ofun at https://raku.org/community

    • GuB-42 16 hours ago

      I am following Raku and Zig from afar, and they both share similarities in that both languages are "optimized for fun" in a way, so no surprize that they come together.

      Zig focus on compilation speed and give developers control (even more so than C). Raku, as a Perl descendant is a giant toybox and there is no one telling you not to use them. Both have in common that they give a lot of freedom to developers, which is really enjoyable. They also have in common that they are not very mature and have a limited market share and fine with it, and a community that looks genuinely nice.

      The opposite of Rust. The language has the "bondage and discipline" philosophy, kind of like ADA, the idea being that it does everything it can to stop you from making mistakes. There is a lot of value in that, but it is not particularly fun. Its community was similarly defined by rules, its code of conduct was infamous, again, it serves a purpose but to me, putting the rules forward doesn't make the community look like a fun place to be. And there is the evangelism, the Rust community is aggressive "rewrite it in Rust!", "no memory unsafe languages!", etc... I have never seen such attitude from the Zig community. Sure they love their language and will tell you it is the best in the world, but they will not say that you are wrong for not using it. As for Raku, they don't seem to care that no one else use their language, they just hope it will happen eventually if they continue going forward.

      • librasteve 4 hours ago

        lol … fwiw we are as bemused by the situation as anyone (well, I am, I can’t really speak for anyone else). My take is that Raku (née perl6) got the worst of all worlds … divided the perl community, was 15 years in the making, got dragged into the reputational suction as the unreconstructed perl Titanic sank. All the same, it’s a wonderful reimagining of all the great things of perl from all the community RFPs and the genius of Larry and Damian and co and fixes most of the negatives and is now there and steadily improving and what’s not to like!

      • spease 13 hours ago

        > the Rust community is aggressive "rewrite it in Rust!", "no memory unsafe languages!"

        This is more of a meme than reality at this point.

        • pjmlp 8 hours ago

          Does not help the regulator posts on HN about having rewriten known projects into Rust, and since we are on a Zig thread, Bun.

          • surajrmal an hour ago

            Those rewrites happen because someone was interested in making them happen (often the maintainer). The meme is that evangelists will constantly advocate to you that your project should be rewritten.

    • AndyKelley 16 hours ago

      If you decide to try this, feel free to share your progress and struggles on IRC, ziggit, or ZSF zulip. Plenty of people would be interested in helping out.

      Good luck and happy hacking!

      • librasteve 5 hours ago

        thanks! I have done a bit of work on Raku + Polars (ie the Pandas rewrite in Rust). This has been hampered by Rust lack of a binary FFI - so it has ended up running into unsafe keyhole surgery territory. My current feel is that we would need to move down to Apache Arrow and build a bunch of FFI / MOARMVM level scaffolding to keep it fast. Maybe take a look at COW at the same time.

        Is any of that already lying around in Zig?

    • nagaiaida 19 hours ago

      minor correction, moar stands for metamodel on a runtime, not meta-object aware runtime

  • teabee89 a day ago

    This is the promise that blew my mind the first time I heard about Zig years ago. So happy to see this become reality!

  • derefr 21 hours ago

    So, this linker does fast incremental linking, which is great for development iteration speed.

    But I assume that any kind of incremental linking, is mutually exclusive with link-time optimization? I.e. you'd never want to use this option for a release build?

    • dapperdrake 15 hours ago

      Research "cl:define-compiler-macro".

      It has been done before.

      And LTO is when the C people and the C++ people started to agree.

  • up2isomorphism an hour ago

    I don’t think either zig or rust can replace C, no matter how much this kind of work. They will of course improve its own language. The only language that can replace C is a language is as simple as C. None of them are simple language.

  • ksec 20 hours ago

    Are there any other languages that offer similar compilation performance. The only one I know of or remember is Turbo Pascal.

    • brabel 8 hours ago

      Everyone forgets D. It’s probably the fastest to compile, even faster than Go

    • pjmlp 8 hours ago

      Delphi, D, Nim, Go, C# / .NET Native / Native AOT, Oberon (any on the language family), Ada (depends on the compiler, 7 vendors),...

      • flohofwoe 6 hours ago

        FWIW, IME at least Nim isn't particularly fast when building, at least when compared to C projects. E.g. in the sokol-nim bindings I'm seeing build timings like:

            32743 lines; 0.953s
        
        ...building 32k lines of code in a second really isn't fast on an M1 Mac, and that's for a debug build without optimization.
        • pjmlp 5 hours ago

          Depends on which compilation pipeline, I would say.

        • cenamus 5 hours ago

          I mean Nim just compiles to C or C++, so it's bound to be slower

    • mgrandl 19 hours ago

      Isn’t go (with cgo disabled) still at least as fast to compile?

    • ozgrakkurt 16 hours ago

      Compilation speed isn’t that much of a factor of language as far as I can understand. It is more related to how optimization is done and how machine code is generated.

      Also obviously it is about how fast the actual implementation of the compiler/build-system is.

      • mattgrice 15 hours ago

        Definitely not true. Otherwise we would have really fast C++ compilers and no one would ever have implemented hacks like precompiled headers.

        • pjmlp 8 hours ago

          That hack is because of C.

          Definitely true when using VC++ with C++20 modules and MSBuild.

          It also helps not compiling everything from source as many UNIX folks do.

          • flohofwoe 6 hours ago

            > Definitely true when using VC++ with C++20 modules and MSBuild.

            Lol, sorry, but as soon as MSBuild is involved the compiler can be infinitely fast and you'd still need to be waiting for the build. Also the main problem of MSVC is the slow linker, and that isn't fixed by C++ modules. This is also the first time I'm hearing that C++ modules actually help with compilation speed in real world projects - the best I've heard so far is that they're a bit faster than precompiled header but not by much, which simply isn't good enough for typical C++ projects.

            • pjmlp 5 hours ago

              Follow engineering principles and actually test it.

              The company behind Cadifra UML Editor is quite happy with their migration, the owner keeps posting about their modules experience on Reddit C++.

              Microsoft also has CppCon talks on the matter.

              All my C++ hobby projects use modules, as I only care about VC++.

      • dapperdrake 15 hours ago

        This has drastically changed with "recent developments".

        Iteration speed is everything now, if and only if you learn from the additional iterations.

  • androiddrew 17 hours ago

    I really would like a 1.0, then I think it can actually be adopted by business.

  • quikoa 21 hours ago

    These improvements are quite promising and I'm looking forward to giving that a spin once it is released.

    Will the Windows side for 0.17.x get some compiler improvements as well or is this Linux only?

  • c0rruptbytes 19 hours ago

    i'm not cool and hip like hacker news devs, but I've been seeing Zig a lot, is this the new cool thing on the street? no more Rust?

    • epolanski an hour ago

      It's a hard-to-dislike language in the niche of systems programming.

      The author is strongly biased and opinionated on his architectural and design choices, and those choices resonate with many.

      • pjmlp an hour ago

        I don't like the @ all over the place and the approach to slurping source as structs for libraries.

        I would faster get back to Modula-2 or Object Pascal.

    • agluszak 19 hours ago

      Zig's been around for ~10 years. It's more low-level and lightweight than Rust. Different goals, different trade-offs. If Rust is the new C++, Zig is the new C.

      • c0rruptbytes 19 hours ago

        > If Rust is the new C++, Zig is the new C.

        thank you, this helps!

        • donpdonp 10 hours ago

          I used to think Zig was the new C, but its different enough to be its own category, imho. For an actually "new" C, try https://c3-lang.org/.

      • Ygg2 16 hours ago

        Still no 1.0 version though. So technically it's year 0.

        • peesem 14 hours ago

          "technically" usually means something like "strictly", not "by a completely different metric". work takes time. zig has had a decade of work put into it.

          • Ygg2 13 hours ago

            Technically means according to a strict, often legal definition.

            The strict definition being we don't count developments that happened before version 1.

            Like when we talk about Rust, we don't mention the virtual threads or GC or the @ symbol for GC references. Even though those all happened during its development.

  • bcardarella 21 hours ago

    I wonder how much this work being pushed forward right now is a response to the Bun drama.

    • kristoff_it 21 hours ago

      None of it, we've been working on this stuff for a long time already, scroll the devlog backwards, you will find plenty of entries on that topic.

      It's the opposite: people have become more receptive to communication about this work now that there's "drama" attached to it.

      This post I co-authored with Andrew is from 2020. In it we announce the idea of getting rid of LLVM from the debug build pipeline and since then work has been steadily going forward, it's just not trivial to bootstrap a full compiler pipeline for all major targets, but we're finally getting there.

      https://kristoff.it/blog/zig-new-relationship-llvm/

      • bcardarella 21 hours ago

        I'm very glad to see the work, thank you for all of the efforts.

    • cafebabbe 20 hours ago

      There is absolutely no "Bun drama", there are just two projects with different goals and methodologies, mutually incompatible. All this thing is just a small bunch of bored, terminally-twitting people ...

      In any case, I'm super glad for this milestone (and impressed!).

    • mi_lk 21 hours ago

      Some people really can’t operate without stirring unnecessary drama.

      What if that’s true and what if that’s not true?

    • dzbarsky 21 hours ago

      None? All of these things were in flight for a while and given Zigs anti-AI stance i think they wrote off Bun ever since the acquisition

      • dnautics 17 hours ago

        what anti-ai stance? i have multiple projects in zig that are pretty much written by AI, no problem.

        • mihaelm 16 hours ago

          They're probably refering to their strict "No LLM / No AI" policy: https://codeberg.org/ziglang/zig/src/branch/master/README.md...

          which applies to contributing to the Zig project.

          The "contributor poker" blog post should probably be a required reading to understand where it comes from: https://kristoff.it/blog/contributor-poker-and-ai/

          "Anti-AI stance" is painting it with too broad of a brush. You're definitely not breaching any CoC or whatever by using AI for your Zig projects.

          • dnautics 3 minutes ago

            > too broad of a brush

            yes, that was my point. as domeone who uses ai extensively to write zig (and someone who has made very small non-AI cobtributions to zig in the past), rejecting ai is currently a strategically good decision for core zig.

          • epolanski an hour ago

            And the largest Zig projects all make use of AI assistance to build software.

            • dnautics 3 minutes ago

              tigerbeetle does not, but the others do.