The Goose Programming Language

(aardappel.github.io)

202 points | by bobbydigitales 5 hours ago ago

86 comments

  • malkia 2 minutes ago

    For Windows, these N data stacks would probably just RESERVE the address space, and as it grows COMMIT it, right?

    I guess there would be some penalty to be paid here for smaller pages, but being mostly contigous might also help some.

    That to be said, such language seems to require 64-bit virtual address space in order to function reasonably, right?

    Also a .dll written in it might not be the best suited, for example if loaded in the middle of fragmented virtual address space it may need to find itself hard to allocate N contigious data stacks (do they absolutely have to be contiguous, what if they are simply ropes or deques or connected chunks ... obviously if some data structure requires to be contigous that would be a problem, but maybe this can be expressed somehow.... or maybe I'm overthinking here).

    Super excited about this!

  • Aardappel 5 hours ago

    Btw I made the language, AMA. repo: https://github.com/aardappel/goose

    • adastra22 2 hours ago

      I would throw the entire docs out and write it from scratch in your own words. I get that we live in an era where the machine writes most/all of the code, and I can accept that. What I need, to even consider using and extending this project, is some understanding of what part of the design is yours and what part is accident by Claude. I can't get that from reading Claudese.

      • WillAdams 24 minutes ago

        The version of the documentation which I would very much like to see would be a Literate Programming version where all the code is explained from the ground up.

        Apparently, this can actually assist the LLM:

        https://silly.business/blog/we-should-revisit-literate-progr...

        as discussed here previously:

        https://news.ycombinator.com/item?id=47300747

      • felixgallo 2 hours ago

        We as a society have to cut this low effort commenting out. The documentation is perfectly comprehensible. What action would you take differently if a given feature were implemented by Claude, or a human?

        • spuz 43 minutes ago

          If you've ever been tasked with coming up to speed on a system mostly documented with AI writing, you'll quickly realise how hard it is to trust anything you are reading. Even if the Claude documentation is 100% accurate, you'll never shake the worry that some crucial fact you founded your understanding on was wrong. It's a fast recipe for psychosis.

          That's without even mentioning that AI writing can often state the what but not the why. That human perspective is usually the most valuable knowledge to get across in writing.

          • JojoFatsani 16 minutes ago

            There’s no reason to trust a human any more than an LLM for this. Humans are not 100% accurate and very capable of making crucial factual errors in documentation. Ask me how I know.

        • adastra22 an hour ago

          As I said in my message, it would determine whether I use the project or not. I need to match it to my use cases, and if I find a match I need to know: is it matching because my use case falls in the domain of what the human collaborator wants from this project, or is my use case a hallucination that will go unsupported?

          You cannot tell either from the AI written docs. I'd rather OP gave a rambling post about some problem he has in gamedev that led him to think a new language was needed, why the constraints are different, and how this will improve X, Y, or Z in his own words. Then I can infer what this project is and where it is going. Right now, I have no clue.

          • knowsuchagency an hour ago

            The docs are perfectly legible and to the point. I don't see the need to waste the author's time with aesthetics

            • adastra22 26 minutes ago

              Legibility is not the concern. The issue is whether the point the docs are making are the actual point the author is concerned with.

          • felixgallo an hour ago

            With respect, what on earth are you talking about? The documentation is clear about what the language does. You can download it and test it out if you like, and verify if yourself if the documentation matches the feature descriptions. What the human collaborator wants is immaterial. You get the code.

            The author has written that post about why the language came about. He doesn't owe you that, but he did it anyway. He doesn't owe you anything more. You can use it or not. Advising him to throw away all the documentation and rewrite it again by hand from scratch so that adastra22 can deign to examine it further is ridiculous.

            If you have a question about whether or not this further iterations will fit your specific (and so far undescribed) particular use case, the correct avenue for that is to ask the author politely. Or Fable; Fable could probably tell you without having to bother the author. But holy cow, the entitlement.

            • adastra22 25 minutes ago

              Are we now in a situation where all code is disposable, throwaway things with no long-lived projects?

              We're talking about a programming language. Languages, and their compilers, evolve over time. Where is this one headed?

        • condwanaland an hour ago

          Why are low effort comments not worthwhile but low effort docs are fine?

    • koolba 4 hours ago

      What does this mean (from your link):

      > Nested data stays inline. A string, an array of strings, a record with variable-size fields and an array of those records are each one contiguous block with no pointer in it. A record that is 160 bytes and an allocation in C++ is 29 bytes and none in Goose.

      I’m having trouble parsing that last sentence.

      • Aardappel 4 hours ago

        It means that to do variable sized things like vectors and strings, C++ often ends up having to allocate their contents dynamically. Which results in data structures with lots of 64-bit pointers over multiple allocations.

        Even Rust's "enum" feature, which can store different sized things all in the same space, does that at the cost of the smallest variant being just as big as the biggest.

        Goose can do all of that not only inline (no pointers), but with variable size, meaning small items take less space. It's like a serialized data representation, but iterable. It makes for big space savings vs these languages.

        • randfur 3 minutes ago

          I wish it was more clear about the memory layout of nested variable sized data structures. As far as I can work out the max capacity is decided at construction time which makes the whole thing a lot less intriguing.

        • eslaught 4 hours ago

          As far as I can tell, you're essentially creating a new stack for every dynamically-sized variable (because otherwise nothing would be able to grow by bump allocation without moving). But this means you're creating a separate virtual memory allocation for each of these objects, which (I presume?) then rounds the size up to the OS's minimum page size (4KiB or similar).

          Doesn't that waste a lot of space? Or is there something I'm missing?

          Also it would seem to force every allocation to go through the OS kernel to allocate unless there's some trick to avoid that.

          • vanderZwan 3 hours ago

            I think what you might be missing is that you can also do a sort of livenes analysis to determine how many dynamically-sized variables are "awake" at the same time at most. Then you only need to create as many stacks as that number, because you can reuse the stacks of "sleeping" variables (since the "awake" variables are guaranteed to be freed once their block goes out of scope, and threads don't share any memory so no data race worries regarding those stacks).

            For example, suppose we have a function with a growable array that calls another function (without passing said array) that also uses a growable array as some temporary scratch space. Then the first function's array isn't being used when the second is sleeping, so the inner function's array could safely share the stack because it will all be freed with a single pointer restore once the second function exits.

            Now, I actually don't know how the compiler works but as I understand it does things similar to my example to optimize the number of stacks needed (but let's wait for WvO to confirm or deny whether my understanding matches reality).

          • Aardappel 3 hours ago

            Only top level growable arrays get their own "stack", if you create an array of arrays it only uses one stack.

            How you deal with the inner stack growing depends, ideally they don't grow (construction only), or you can make them fixed size, or with a capacity, or in the worst case, as a pool. The language has facilities for all of these.

        • koolba 4 hours ago

          These AI generated docs would make more sense if you instruct it to not include numbers unless they form a complete example.

      • lynndotpy 4 hours ago

        I parsed it as "A record that is ((160 bytes and an allocation) in C++) is ((29 bytes and none) in Goose)."

        So, I read it as, "C++ uses 160 bytes and an allocation, when Goose requires 29 bytes and no allocation".

    • giovannibonetti 31 minutes ago

      Where does the language stand in the array-of-structs vs struct-of-arrays choices? Does it offer an annotation to choose that like in Zig?

    • SwellJoe 25 minutes ago

      Way off topic, but I enjoyed Amiga E. And, I reckon you're one of a handful of people I would take seriously when they say, "I made a new programming language."

    • nextaccountic 4 hours ago

      > Threads that share nothing. A worker is compiled as a separate program with its own memory, and flat values cross typed queues as a memcpy. Data races, locks, atomics and memory orderings do not exist in the language.

      how to perform this copy without atomics?

      do the workers run as threads or processes?

      also you claim it's faster than rust and c++. but for certain kinds of software it's certainly (and substantially) slower, right?

      • Aardappel 4 hours ago

        The copy is made by the runtime, which obviously does use atomics underneath, but you don't ever use these yourself.

        The workers are threads. The separate memory is the concurrency model, but it's still single process, multiple threads.

        The "faster than" is on average, and both in my own benchmarks and independent benchmarks you can find examples of algorithms where it is slower, but they tend to be the exception. "substantially" slower I haven't found yet.

        https://kostya.github.io/LangArena/#/top_tab

        • Jtarii 27 minutes ago

          That site is just benchmarking how well an AI can implement things, it has nothing to do with how "fast" the language is.

    • osigurdson an hour ago

      What are some use cases (if any) that the language is not suitable for?

      • malkia 21 minutes ago

        Probably kernel code, or cases where allocation randomization is important. Seems like with goose you may end up executing more deterministically (but still it can randomize where these N data stacks start), but because it's single threaded workers (in essence), all data construction would follow the same pattern, hence an attacker (something snooping your memory) would know easier where things would be.

    • AnEro 4 hours ago

      You noted AI creating a large amount of this code base with yourself directing it. Do you have a process that keeps you sane while doing this over such a large codebase?(especially for a brand new concept and implementation) [Obviously congrats and can't wait to read through to have deeper questions!]

      • Aardappel 4 hours ago

        A big part of this making it base the code of this new compiler on my previous compiler for the Lobster Programming Language. I had it go thru that codebase, distill an elaborate code style guide from it, and used that to make it write the new code pass by pass in that style. That made me feel "at home" as if it is my code, easier to review etc, and easier to know we're going in the right direction by seeing similar structure to my past compiler work.

        Once the initial compiler was there it became easier, as Fable is rather good of working within the context of what is already there.

        • AnEro 4 hours ago

          Thanks much appreciated!! Feeling silly for not noticing my best experience with AI in larger/complex cases has been languages/frameworks that are opinionated in usage/design(zig/go/elixir/rust)

    • amelius 4 hours ago

      Is this like forcing the escape analysis of the compiler to be perfect, by making it explicit in the programming language?

      I'm wondering what kind of architectural corners this forces the programmer into ... have you found any so far?

      • Aardappel 4 hours ago

        Yes, the compiler guarantees references to data structures can't escape. I believe it does that "perfectly" :)

        It certainly requires the programmer to rethink how they structure their data, as you have to think of where it is going to be stored, and how to store it in a way such that it will have similar lifetimes to what it is stored with, to get the most out of the language.

    • fancyfredbot 4 hours ago

      What happens if the memory is freed in a different order to allocation? Does the language disallow that? Or does it get allowed but mean the stack can't shrink?

      How are you deciding how many stacks to use?

      • Aardappel 4 hours ago

        The compiler automatically optimally allocates all arrays currently in scope that are able to grow to "stacks" (address spaced reserved areas).

        The programmer choose what to store where, so you typically store things in arrays by lifetime. Goose gives you many more tools to do that, it can store entire variable sized trees of an enum type in a single array.

        You try to avoid having to deallocate in mixed/random order, but if you absolutely must, you can turn arrays into pools, basically.

        • fancyfredbot 4 hours ago

          Thanks!

          I'm a bit confused about how you could decide the number of stacks at compile time. I can see you could see how many arrays are lexically in scope. However if the code puts one dynamic array inside another dynamic array I can get to an unbounded number of total stacks without ever needing more than two in lexical scope at once. Do you also add stacks at runtime? Like a stack of stacks?

          • Aardappel 4 hours ago

            The top level arrays all are owned by a local variable. An array of arrays is just allocated to a single stack.

            • fancyfredbot 3 hours ago

              I'm sorry if I'm being dense here but I can't see how that would work?

              The spec on your GitHub says: "Two simultaneously-live resizable values never share a stack."

              So I surely can't have a dynamic array of dynamic arrays share a single stack? That's an unbounded number of resizeable values sharing a stack?

              You could make it work on a single stack but that would involve having non-contiguous memory for each inner array since you can add to any array in any order.

              The spec implies that we need one stack per element of the outer dynamic array which would let you have contiguous memory for each inner array, but you wouldn't know how many stacks you needed at compile time.

              • Aardappel 2 hours ago

                The language has different levels of solutions for the inner arrays being dynamic, for example a) only dynamic upon construction, b) fixed size, c) fixed size with cap, or d) allocated in a pool in a different array.

                Depending on your needs you use one of these solutions.

    • lemming 4 hours ago

      Do you consider Goose to be a replacement for Lobster, or do they fill different niches?

      • Aardappel 4 hours ago

        I think they have slightly different places on the "speed vs ease of use" pareto frontier.

        With Goose, the core design favors speed over everything else. Though not only was I able to make that safe, it is fairly pleasant to use. But it remains a low level systems language that requires you to think hard about data organization.

        Lobster is a waaay more high level language made to be pleasant to use first, fast second. Surprisingly though, it can get pretty close in speed to C++/Rust nowadays too, but still a lot slower than Goose.

        • lemming 4 hours ago

          Do you have plans to continue using AI to improve Lobster too, e.g. working on implementation speed, or perhaps self-hosting?

          How have you found AI at writing Goose or Lobster code, if you give them a skill or something, are they relatively competent at actually writing code in the language?

          Both are really interesting projects, BTW - thank you! I've been meaning to try Lobster out but haven't found time, I think this might be the push I needed.

          • Aardappel 3 hours ago

            I've already been improving Lobster's speed a lot in recent times: https://github.com/aardappel/lobster/blob/master/tests/compa...

            Self-hosting would be cool too, but for both Lobster and Goose I've tried to keep it simple for people to build the language in any environment, which is why its still all in C++.

            I don't use any "skills". From what I can see, the #1 way an AI produces "good code" is to simply work in the context of existing good code. I don't need to hold it by a style guide, it seems to infer that from context just fine. At least Fable does, other models less so.

            • andyferris 2 hours ago

              I suppose something like a skill would help those starting with the language. If nothing else, a well written skill can be read by a human - or you could load the skill and generate a program you are familiar with.

            • vanderZwan 3 hours ago

              > From what I can see, the #1 way an AI produces "good code" is to simply work in the context of existing good code.

              "Just Git Gud First" - Wouter van Oortmerssen

              • perrygeo 36 minutes ago

                That's unnecessarily dismissive. I find the statement quite enlightening and I wish other engineers would realize the wisdom here.

                LLMs are pattern matching machines. Your code is the base template that will determine all patterns moving forward. No amount of context engineering or prompting can save your code from itself.

                Pragmatically, it suggests we need a much greater investment in code quality, and much less hand-wringing over AI tooling.

    • ksec 4 hours ago

      First, Congrats on the release!

      Do you know if Goose is the first time something like this has been done?

      • Aardappel 4 hours ago

        In this particular combination, most certainly.

        There is lots of prior art on the individual parts of the design, e.g. "regions" and various stack languages etc, see: https://github.com/aardappel/goose/blob/master/docs/design/p...

        Part of this is that 64-bit address spaces make this design practical, and almost all languages in use today were designed with the limitations of 32-bit in mind. Once you can assign arrays huge amounts of address space, you don't have to realloc, which gets you stable pointers, and a lot of the other Goose features.

    • groundzeros2015 3 hours ago

      Why are you pretending that John is endorsing your project?

    • jdw64 3 hours ago

      Why was it named Goose?

      • Aardappel 2 hours ago

        I've named many of my past languages with animal names, e.g. Lobster. Goose just popped into my head as cool sounding at the time. It has no special meaning.

        • spacemarine1 2 hours ago

          I feel the need, the need for speed. Talk to me, Goose.

        • jdw64 43 minutes ago

          Thank you.

    • vanderZwan 4 hours ago

      If you had no other obligations and todos getting in the way, what's the first thing you'd rewrite in Goose yourself?

      (And why isn't it Amiga E?)

      • Aardappel 4 hours ago

        Most of my current Goose projects are games, not surprisingly, but I have some language/tooling ideas as well.

        As for Amiga E, I am very proud of having made it way back when obviously, but it doesn't stack up against my more modern languages (Goose and Lobster), so I think I'll leave it be :)

  • malkia an hour ago

    "Threads that share nothing. A worker is compiled as a separate program with its own memory, and flat values cross typed queues as a memcpy. Data races, locks, atomics and memory orderings do not exist in the language."

    That reminds me of Dart Isolates - https://dart.dev/language/isolates

  • ebiederm 4 hours ago

    I seem to remember the ML guys went down this route. I believe they called it regions.

    From what little I read in the literature I seem to recall that there were enough programs with quite a bit more memory usage that they decided this approach was a dead end.

    Does anyone know anymore details?

    • Aardappel 4 hours ago

      Yes, ML Kit region inference is probably the closest prior art to Goose.

      Back then though, they couldn't lean on large 64-bit address spaces like Goose can. They implemented all of this with a chain of blocks of memory instead, and tracked most of it at runtime that Goose now does at compile time.

      https://github.com/aardappel/goose/blob/master/docs/design/p...

  • bobbydigitales 5 hours ago

    "That looks really neat! The automatic analysis and use of multiple data stacks and resizable fields takes it from "lots of work can be done this way" to "basically everything can be".

    The relative references and built in binary structure validation also enable direct memory mapping of data structures and immediate use of network data. "Parsing" file and network data is often 100x slower than it needs to be." - John Carmack

  • pradn 3 hours ago

    Goose is pretty cool. The idea is that you don’t need to make representable all programs: a language for a restricted purpose is ok. As long as you can statically specify the number of growable memory regions, you can use Goose. However, since there’s no garbage collection, you’ll probably need to be careful about memory use while a program is running. Ie: the stacks can get too big if you keep growing them, even as old data is not used any more.

    Still, for a large number of programs - like data-oriented applications - it’s probably a good fit. It might work ok with entity-component game systems as long as you reuse old memory…

    • Aardappel 3 hours ago

      You typically "allocate only" if a data structure is made entirely on construction, or if its growth is bounded.

      If it is potentially completely unbounded, you need different solutions like a pool, or essentially reconstructing etc.

  • cardanome 29 minutes ago

    I hate when I see a new programming language, get my hopes up and it turns out to be AI slop again.

    There might be some decent design ideas there but if you can't be arsed to write them down in your own words but instead make me read AI generated slop I am out.

  • AnEro 5 hours ago

    Such a cool idea! I've been toying around making a few libraries to force languages to do that direct io zero-read mapping to data structures. Ofcourse this looks way better and infinity less hacky, but it is a fun and trippy experience to find someone working in the same direction as you independently.

  • wewewedxfgdf an hour ago

    What are the down sides? That's the key question when a better thing comes along.

    • Aardappel an hour ago
      • wewewedxfgdf an hour ago

        I'd put the pros and cons right up front on the README.

        Selling the positives for a new programming languages should go hand in hand with a realistic thorough analysis of the negatives.

        Negatives of the language as a brief point 18 on the tutorial makes the negatives look hidden.

        • malkia 11 minutes ago

          He is not "selling" anything. I don't see a price, quite the opposite - it's on the user to evaluate it's points, not on the "seller" (that he is not). Obviously if this becomes paid, or paid through support product this would be one of the first points of the table - "what are the limitations, what should we be aware of sorts".

  • calvinmorrison 2 hours ago

    Can someone explain the weird phrasing here "Nothing moves, so references survive growth" that AI does.

    • Aardappel 2 hours ago

      Almost all languages rely on reallocation when a growable array grows, which means you can't store pointers to elements, as they get invalidated upon a reallocation.

      In Goose, things grow in-place only, since they have a large amount of address space reserved.

  • groundzeros2015 3 hours ago

    He didn’t comment on goose. He talked about a concept and the goose guy said “that’s like my idea”. And John said “yeah sounds like it”.

    • Aardappel 3 hours ago

      You can see from his comment that he apparently went to the Goose repo and read it. What he references is neither in his original tweet nor my reply.

      • groundzeros2015 2 hours ago

        The headline is misleading. He hasn’t been commenting about the Goose language and his experience with it. And you are promoting.

  • gigatexal 3 hours ago

    Would be interesting to see Luke’s Zed language from the replies on HN.

  • mi_lk 5 hours ago

    Obviously the guy is replying to something that you can't see without logging in twitter. And without context this means little

  • geoffbp 5 hours ago

    No relation to https://goose-docs.ai/ I guess?