Unsigned Sizes: A Five Year Mistake

(c3-lang.org)

45 points | by lerno 2 hours ago ago

46 comments

  • kevin_thibedeau 2 hours ago

    Systems programmers love to hate on unsigned integers. Generations have been infected with the Java world model that integers have to be pretend number lines centered on zero. Guess what, you still have boundary conditions to deal with. There are times when you really really need to use the full word range without negative values. This happens more often with low level programming and machines with small word sizes, something fewer people are engaged in. It doesn't need to be the default. Ada has them sequestered as modular types but it's available to use when needed.

    • uecker 2 hours ago

      Having them available is not the issue, using them for sizes and indices is what causes a lot of tricky bugs.

      • throwaway894345 an hour ago

        Why does an unsigned type for sizes or indices fare worse than a signed type? When do I want the -247th element in an array? When do I have a block that is -10 bytes in size?

        • kevin_thibedeau an hour ago

          There are (rare) times when you want negative array indices. C lets you index in both directions from a pointer to the middle of an array. That's why array indexing is signed in C. Some libc ctypes lookup tables do this. For sizing there is no strong case for negatives other than to shoehorn them into signed operations.

          • throwaway894345 an hour ago

            That’s interesting but seems pretty dangerous. How do you know you aren’t going to decrement off the front of the array? Keeping the pointer to the first element in the array and using offsets seems safer for humans and I don’t think the computer would care.

            • mmilunic 15 minutes ago

              Kinda a smart alec response, but how do you know you aren’t going to increment off the end of the array when operating normally? I guess it is twice the danger.

        • uecker an hour ago

          the reason is not that you want a negative index or size, but that you want the computation of the index to be correct, and you want to have obvious errors. Both turns out to be easier with signed types.

    • pron an hour ago

      In Java, unsigned arithmetic is available through an API and, as you said, it is pretty much only needed when marshalling to certain wire protocols or for FFI. Built-in unsigned types are useful primarily for bitfields or similar tiny types with up to 6 bits or so.

    • einpoklum an hour ago

      > There are times when you really really need to use the full word range without negative values.

      There are a few of those, but that is the niche case. Certainly when we're talking about 64-bit size types. And if you want to cater to smaller size types, then just just template over the size type. Or, OK, some other trick if it's C rather than C++.

      • pixelesque an hour ago

        Sometimes (and very often in some scenarios/industries, i.e. HPC for graphics and simulation with indices for things like points, vertices, primvars, voxels, etc) you want pretty good efficiency of the size of the datatype as well for memory / cache performance reasons, because you're storing millions of them, and need to be random addressing (so can't really bit-pack to say 36 bytes, at least without overhead away from native types, which are really needed for maximum speed without any branching).

        Losing half the range to make them signed when you only care about positive values 95% of the time (and in the rare case when you do any modulo on top of them you can cast, or write wrappers for that), is just a bad trade-off.

        Yes, you've still then only doubled the range to 2^32, and you'll still hit it at some point, but that extra byte can make a lot of difference from a memory/cache efficiency standpoint without jumping to 64-bit.

        So very often uint32_t is a very good sweet spot for size: int32_t is sometimes too small, and (u)int64_t is generally not needed and too wasteful.

  • deathanatos 17 minutes ago

    > The former is easier to define, but has the downside of essentially “silencing warnings”. Let’s say the code was originally written to cast an u16 to u32, but later the variable type changes from u16 to u64 and the cast is now actually silently truncating things. Here we have casts becoming a sort of “silence all warnings”.

    Well … we even mention Rust in the paragraph right before this. In Rust, you can up a u16 to a u64 this way:

      let bigger: u32 = x.into();
    
    or

      let bigger = u32::from(x);
    
    The conversion `from` is infallible, because a u16 always fits in a u32. There is no `from(u64) -> u32`, because as the article notes, that would truncate, so if we did change the type to u64, the code would now fail to compile. (And we'd be forced to figure out what we want to do here.)

    (There are fallible conversions, too, in the form of try_from, that can do u64 → u32, but will return an error if the conversion fails.)

    Similarly, for,

      for (uint x = 10; x >= 0; x--) // Infinte loop!
    
    This is why I think implicit wrapping is a bad idea in language design. Even Rust went down the wrong path (in my mind) there, and I think has worked back towards something safer in recent years. But Rust provides a decent example here too; this is pseudo-code:

      for (uint x = 10; x.is_some(); x = x.checked_sub(1))
    
    Where `checked_sub` is returns `None` instead of wrapping, providing us a means to detect the stopping point. So, something like that. (Though you'd probably also want to destructure the option into the uint for use inside the loop.) Of course, higher-level stuff always wins out here, I think, and in Rust you wouldn't write the above; instead something like,

      for x in (0..=10).rev()
    
    (And even then, if we need indexes; usually, one would prefer to iterate through a slice or something like that. The higher-level concept of iterators usually dispenses with most or all uses of indexes, and in the rare cases when needed, most languages provide something like `enumerate` to get them from the iterator.)
  • Groxx 2 hours ago

    >If sizes are unsigned, like in C, C++, Rust and Zig – then it follows that anything involving indexing into data will need to either be all unsigned or require casts. With C’s loose semantics, the problem is largely swept under the rug, but for Rust it meant that you’d regularly need to cast back and forth when dealing with sizes.

    TBH I've had very little struggle with this at all. As long as you keep your values and types separate, the unsigned type that you got a number from originally feeds just fine into the unsigned type that you send it to next. Needing casting then becomes a very clear sign that you're mixing sources and there be dragons, back up and fix the types or stop using the wrong variable. It's a low-cost early bug detector.

    Implicitly casting between integer types though... yeah, that's an absolute freaking nightmare.

    • kibwen 39 minutes ago

      > As long as you keep your values and types separate, the unsigned type that you got a number from originally feeds just fine into the unsigned type that you send it to next.

      Part of me feels like direct numeric array indexing is one of the last holdouts of a low-level operation screaming for some standardized higher-level abstraction. I'm not saying to get rid of the ability to index directly, but if the error-resistant design here is to use numeric array indices as though they were opaque handles, maybe we just need to start building support for opaque handles into our languages, rather than just handing out numeric indices and using thoughts and prayers to stop people from doing math on them.

      For analogy, it's like how standardizing on iterators means that Rust code rarely needs to index directly, in contrast with C, where the design of for-loops encourages indexing. But Rust could still benefit from opaque handles to take care of those in-between cases where iterators are too limiting and yet where numeric indices are more powerful than needed.

      • dnmc 9 minutes ago

        Maybe this isn't what you're suggesting, but it's already possible to make an interface that prevents callers from doing math on indices in Rust — just return a struct that has a private member for the index. The caller can pass the value back at which point you can unwrap it and do index arithmetic.

      • Groxx 34 minutes ago

        You do need to store those if they're totally opaque though, e.g. how do you represent a range without holding N tokens? Often I like it, and it allows changing the underlying storage to be e.g. generational with ~no changes, but it kinda can't be enforced for runtime-cost reasons.

        Using a unique type per array instance though, that I quite like, and in index-heavy code I often build that explicitly (e.g. in Go) because it makes writing the code trivially correct in many cases. Indices are only very rarely shared between arrays, and exceptions can and should look different because they need careful scrutiny compared to "this is definitely intended for that array".

  • LegionMammal978 2 hours ago

    > But what about the range? While it’s true that you get twice the range, surprisingly often the code in the range above signed-int max is quite bug-ridden. Any code doing something like (2U * index) / 2U in this range will have quite the surprise coming.

    Alas, (2S * signed_index) / 2S will similarly result in surprises the moment the signed_index hits half the signed-int max. There's no free lunch when trying to cheat the integer ranges.

    • lerno an hour ago

      The difference is that in the unsigned case you get a seemingly plausible value, and in the signed case you get a negative value which you can be sure is wrong. This is the problem.

  • larsnystrom 6 minutes ago

    I don’t understand how dealing with numbers correctly is not a solved problem in computer engineering by now.

  • ok123456 37 minutes ago
    • norir 5 minutes ago

      In my reading, what Stroustroup is saying is that given other problems in c/c++, that singed sizes are less bad than unsigned but both have clear and significant deficiencies. A new language doesn't have to inherit all of these deficiencies.

  • ks2048 2 hours ago

    I know language designers have a lot of trade-offs to consider... But I would say if you know a value will logically always be >= 0, better to have a type that reflects that.

    The potential bugs listed would be prevented by, e.g. "x--" won't compile without explicitly supplying a case for x==0 OR by using some more verbose methods like "decrement_with_wrap".

    The trade-off is lack of C-like concise code, but more safe and explicit.

    • pron an hour ago

      > But I would say if you know a value will logically always be >= 0, better to have a type that reflects that.

      Except that's not quite what unsigned types do. They are not (just) numbers that will always be >= 0, but numbers where the value of `1 - 2` is > 1 and depends on the type. This is not an accident but how these types are intended to behave because what they express is that you want modular arithmetic, not non-negative integers.

      > e.g. "x--" won't compile without explicitly supplying a case for x==0

      If you want non-negative types (which, again, is not what unsigned types are for) you also run into difficulties with `x - y`. It's not so simple.

      There are many useful constraints that you might think it's "better to have a type that reflects that" - what about variables that can only ever be even? - but it's often easier said than done.

      • Groxx an hour ago

        That's true for signed numbers too though? `int_min - 2 > int_min`

        I agree they're a bit more error-prone in practice, but I suspect a huge part of that is because people are so used to signed numbers because they're usually the default (and thus most examples assume signed, if they handle extreme values correctly at all (much example code does not)). And, legitimately, zero is a more commonly-encountered value... but that can push errors to occur sooner, which is generally a desirable thing.

        • oasisaimlessly 27 minutes ago

          > That's true for signed numbers too though? `int_min - 2 > int_min`

          No, that's undefined behavior in C, and if you care about correctness, you run at least your testsuite in CI with -ftrapv so it turns into an abort().

          • Groxx 24 minutes ago

            Which makes them even less safe than unsigned, where it is defined, yes? The optimizations that can lead to are incredibly hard to predict.

            Besides, for safety there are much clearer options, like wrapping_add / saturating_add. Aborting is great as a safety tool though, agreed - it'd be nice if more code used it.

    • mamcx an hour ago

      I think it should be alike in Pascal where you have size ranges as types, and then, you can declare that this collection fall on this range (and very nicely, you can make it at enum):

      https://www.freepascal.org/docs-html/ref/refsu4.html

      • elch 42 minutes ago

        And none of those "amenities" existed anymore in Oberon.

  • Validark 30 minutes ago

    I am personally moving in the opposite direction. I haven't meaningfully used a signed integer in years, and I see signed integers as being for more niche use-cases. I mainly only use a signed types when I want to do a "signed shift right". If there was a >>> operator in Zig I wouldn't even think of signed integers.

    Given your examples, I think you'd have fewer issues if you were working with unsigned integers exclusively. Although I'm curious about what other code you were referencing with this: "But seeing how each change both made the code easier to reason about and more correct, I couldn’t deny the evidence."

    With regards to modulo, in Zig if you try to use it with a signed integer it will tell you to specify whether you want `@mod` or `@rem` semantics. In my case, I'd almost never write `x % 2`, I'd write `x & 1`. I do use unsigned division but I'd pretty much never write code that would emit the `div` instruction.

    I'm not saying you're wrong though! Everyone has a different mind. If you attain higher correctness and understandability through using signed integers, that's great. I'm just saying I'm in the opposite camp.

  • ximm 2 hours ago

    Is the text on this page really #bbbdc3 on #ffffff? How is anyone supposed to be able to read that?

    • sureglymop an hour ago

      Weirdly, you have to turn on javascript for the text color to change...

    • idbehold an hour ago

      For me it's #353841 on #ffffff which meets WCAG AAA standards for accessible text.

  • EdSchouten an hour ago

    > If sizes are unsigned, like in C, C++, Rust and Zig – then it follows that anything involving indexing into data will need to either be all unsigned or require casts.

    I don’t really get this claim. Indexing should just look up the element corresponding to the value provided. It’s easy to come up with semantics that are intuitive and sound, even if signed integers or ones smaller than size_t are used.

    • adrian_b an hour ago

      Indexing does that, but the indices must vary in a certain range, whose limits are frequently determined by using something like "sizeof(array)/sizeof(element)" which is an unsigned number.

      This is especially inconvenient in C, where there exist extremely dangerous legacy implicit casts between signed integers and unsigned integers, which have a great probability of generating incorrect values.

      Because the index is typically a signed integer, comparing it with an unsigned limit without using explicit casts is likely to cause bugs. Using explicit casts of smaller unsigned integers towards bigger signed integers results in correct code, but it is cumbersome.

      These problems are avoided as said in TFA, by making "sizeof" and the like to have 64-bit signed integer values, instead of unsigned values.

      Well chosen implicit conversions are good for a programming language, by reducing unnecessary verbosity, but the implicit integer conversions of C are just wrong and they are by far the worst mistake of C much worse than any other C feature.

      Other C features are criticized because they may be misused by inexperienced or careless programmers, but most of the implicit integer conversions are just incorrect. There is no way of using them correctly. Only the conversions from a smaller signed integer to a bigger signed integer are correct.

      Mixed signedness conversions have always been wrong and the conversions between unsigned integers have been made wrong by the change in the C standard that has decided that the unsigned integers are integer residues modulo 2^N and they are not non-negative integers.

      For modular integers, the only correct conversions are from bigger numbers to smaller numbers, i.e. the opposite of the implicit conversions of C. The implicit conversions of C unsigned numbers would have been correct for non-negative integers, but in the current C standard there are no such numbers.

      The current C standard is inconsistent, because the meaning of sizeof is of a non-negative integer and this is also true for the conversions between unsigned numbers, but all the arithmetic operations with unsigned numbers are defined to be operations with integer residues, not operations with non-negative numbers.

      The hardware of most processors implements at least 3 kinds of arithmetic operations: operations with signed integers, operations with non-negative integers and operations with integer residues.

      Any decent programming language should define distinct types for these kinds of numbers, otherwise the only way to use completely the processor hardware is to use assembly language. Because C does not do this, you have to use at least inline assembly, if not separate assembly source files, for implementing operations with big numbers.

      • uecker 36 minutes ago

        Not sure what change in the C standard you mean. unsingned was always modulo. Otherwise, use -Wsign-conversion.

  • cperciva an hour ago

    I don't get it. Is this a parody of poor design decisions?

    Sure, it's possible to write bugs in C. And if you really want to, you can disable the compiler warnings which flag tautologous comparisons and mixed-sign comparisons (a common reason for doing this is to avoid spurious warnings in generic-type code).

    But, uhh, "people can deliberately write bugs" has got to be the weakest justification I've ever seen for changing a language feature -- especially one as fundamental as "sizes of objects can't be negative".

    • adrian_b 27 minutes ago

      The C language does not have any data type that has the property "can't be negative".

      Signed integers can be negative. The so-called "unsigned" integers of C are integer residues modulo 2^N, which are neither positive nor negative, i.e. these concepts are not applicable to "unsigned" integers.

      An alternative view is that any C "unsigned" is both positive and negative. For example the unsigned short "1" is the same number as "65537" and as "-65535".

      So any sizeof value in C is negative (while also being positive).

      In contradiction with what you say, the change described in TFA, by making sizes 64-bit signed integers, is the only method to guarantee that the sizes are non-negative in a language that does not have dedicated non-negative integers.

      Other programming languages have non-negative integers, but C and C++ and many languages derived from them do not have such integers.

      The arithmetic operations with non-negative integers differ from the arithmetic operations of C. On overflows and underflows, they either generate exceptions or have saturating behavior.

      • cperciva a minute ago

        Leaving aside the fact that, yes, unsigned integer types are definitely not negative -- my point wasn't about types at all. Objects cannot take up a negative number of bytes of memory!

      • alberto-m 8 minutes ago

        > An alternative view is that any C "unsigned" is both positive and negative. For example the unsigned short "1" is the same number as "65537" and as "-65535".

        This can be disproven by the fact that dividing by `unsigned e = 1U` is well defined and always yields the starting number. If the unsigned numbers were really modular numbers as you suggest, division could not be defined.

  • IshKebab an hour ago

    It seems like they've identified common bugs patterns in C that would have been ameliorated by using signed, but come to the wrong conclusion that signed is the correct answer rather than that C is poorly designed for making the broken code the easy option.

    Fix the language. Don't hack around it by using the wrong type.

    • ozgrakkurt 22 minutes ago

      This is already fixed in c via bitint types and disabling implicit integer sign casting.

  • jonstewart an hour ago

    I hate using languages that only have signed integers. Using integers that can’t be negative fits many problems nicely and avoids the edge case of having to check for negative.

    • adrian_b 21 minutes ago

      You are perfectly right, but neither C nor C++ nor many more recent languages derived from them have non-negative integers.

      The so-called "unsigned" integers of C are integer residues, where each value can be interpreted either as both positive and negative or as neither positive nor negative. In any case no "unsigned" value can be said to be non-negative.

      You have to go back to languages not contaminated by C, like Ada, to find true non-negative integers among the primitive data types.

      In C++, it is possible to define a non-negative integer type, which can have good performance if you implement its operations in assembly language.

      However I am not aware of an open-source library including such a type.

    • einpoklum an hour ago

      It's not "can't be negative", it's just that the semantics for negativity is wrapping around.

      And - yes, there are very important use cases for unsigned/modulo-2n/wraparound values. But sizes of data structures are generally _not_ one of those use cases. The fact that the size is non-negative does not mean that the type should be unsigned. You should still be able to, say, subtract sizes and get a signed value which may be negative.

      • throwaway894345 an hour ago

        That’s definitely not true. Unsigned ints have no “negativity” semantic. Wrapping around is what happens when you decrement the minimum value of any integer type, including signed types. Regardless of the type you use to represent an integer value that cannot legally be negative, you will have to take care not to allow your program to return values lower than zero for things like indices or sizes.

        • oasisaimlessly 22 minutes ago

          > Wrapping around is what happens when you decrement the minimum value of any integer type, including signed types.

          No, signed wraparound is undefined behavior in C, whereas unsigneds are defined to wraparound. If you use -ftrapv, signed wraparound is an immediate abort().

          • adrian_b 12 minutes ago

            That is right.

            While C like in many other places fails to define the correct behavior to avoid shaming the processor or compiler makers that fail to provide it, there are only 2 correct behaviors on overflows and underflows, like when incrementing the biggest number or decrementing the smallest number.

            Both for signed integers and for non-negative integers, the 2 alternatives of correct behavior on overflows and underflows is to either generate exceptions or to saturate the result to the biggest or smallest representable number.

            Wraparound is the correct behavior for integer residues, which are a distinct data type from either signed integers or non-negative integers.

            While some people criticize C for making easy for careless programmers to make certain kinds of bugs, like access outside bounds, those are easily mitigated by using appropriate compiler options.

            For me a much more serious defect of C is this confusion promoted by it in the heads of most programmers, who do not understand which are the fundamental integer types and which are the correct conversions between them, because C uses "unsigned" instead of at least 3 distinct types that it does not have, bit strings, integer residues and non-negative integers. More rarely, "unsigned" is used for other 2 types that are missing, binary polynomials and binary polynomial residues. All these 5 types must be primitive types in a programming language because all modern processors implement in hardware distinct operations for all 5 types, which can be accessed only through assembly language when these types are missing.