12
$\begingroup$

I’m trying to decide whether my language should provide signed or unsigned bytes, but I’m struggling to find a good reason to choose either side.

Popular languages vary in their decision. For example, Java uses signed bytes, although they have unsigned shorts. C++ has unsigned bytes (std::byte), but I've seen a lot of code using signed chars as bytes.

Is there any consensus view on whether bytes ought to be signed or unsigned by default? Answers should ideally provide citations to back up their claims.

$\endgroup$
5
  • 2
    $\begingroup$ @RydwolfPrograms the question is meant the other way round. It's type byte := i8 or type byte := u8? The language has both types already irrespective of the byte type. $\endgroup$
    – feldentm
    Commented Dec 10, 2023 at 6:57
  • $\begingroup$ Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Programming Language Design and Implementation Meta, or in Programming Language Design and Implementation Chat. Comments continuing discussion may be removed. $\endgroup$
    – Michael Homer
    Commented Dec 11, 2023 at 20:55
  • 1
    $\begingroup$ C++'s primary byte types isn't std::byte, it's char which is implementation defined as to whether it is signed or unsigned; and std::byte isn't really unsigned or signed. $\endgroup$ Commented Dec 12, 2023 at 7:33
  • 2
    $\begingroup$ (A comment as I'm not yet able to post answers to this Q) You should maybe have a look at more recent languages which may have learned a bit from the mistakes of the past. For example, Rust has a fairly symmetrical set of explicitly signed and unsigned integer types for the various bit lengths. There are u8 and i8 (unsigned and signed 8-bit integers) as well as u16/i16, u32/i32, u64/i64 and u128/i128. $\endgroup$ Commented Dec 12, 2023 at 8:59
  • 1
    $\begingroup$ @Hans-MartinMosner The language has signed and unsigned for arbitrary fixed sizes. The question is specifically about the concept of bytes. While I thought that the solution would be an alias like byte := i8 or byte := u8 in your terminology, I guess Matthieu convinced me that this might not be a good idea. $\endgroup$
    – feldentm
    Commented Dec 12, 2023 at 17:05

9 Answers 9

48
$\begingroup$

Frame Challenge: Bytes should not be integers.

Bytes are bytes, a fixed-size sequence of bits, typically 8 on modern computers.

Should an array of bits be signed or unsigned? Neither, the question is nonsensical: an array cannot be signed or unsigned in the first place.

There's a long-standing tradition in programming languages to conflate integers and array of bits, which can be seen in that bit-wise operations are typically defined on integers. I argue that this "blending" of the two roles is a mistake: a violation of the Single Responsibility Principle.

Yes, it's all registers at the end... should a pointer be signed or unsigned? Surely nowadays the question seems nonsensical, even though in B pointers and integers were the same type, and in CPU they are manipulated through the same registers.

Just like pointers are different from integers, I hereby argue that array of bits should be different from integers, and bit-wise functions are only sensible on array of bits.

The conversion between integer and array of bits (of the same number of bits) should exist -- just like that between floating point or decimal an array of bits -- and it should ideally boil down to a no-op, but from a type system, the two should be separate.

$\endgroup$
14
  • 12
    $\begingroup$ This is quite a hot take. There are plenty of reasons to do arithmetic with 8-bit numbers ─ e.g. certain PRNGs and cryptographic algorithms, or emulating 8-bit processors ─ so either you accept that 8-bit numbers should exist (in which case this answer only quibbles about what they should be called, when the question is whether they should be signed or unsigned), or you're arguing that that arithmetic should not be supported directly. $\endgroup$
    – kaya3
    Commented Dec 10, 2023 at 17:29
  • 2
    $\begingroup$ @kaya3: I am must admit that byte immediately evokes memory view in my mind, and the mention of std::byte definitely reinforced it. Re-reading the question, I suppose it could be understood as asking about 8-bits arithmetic... but if so I would argue it is quite unclear as in my first reading it completely eluded me. If it wishes to mention 8-bits arithmetic, it should really mention so. $\endgroup$ Commented Dec 10, 2023 at 19:06
  • 17
    $\begingroup$ @kaya3: "There are plenty of reasons to do arithmetic with 8-bit numbers". Indeed. That's why the language should also have both a signed and an unsigned 8-bit integer type, distinct from byte. $\endgroup$
    – Tomeamis
    Commented Dec 11, 2023 at 15:02
  • 7
    $\begingroup$ @kaya3 notice how you said "8-bit numbers" and not "bytes"? Because you already accepted that they are semantically different. Yes, it makes sense to operate on 8 bit numbers. Who then happen to also be bytes. Just like I can drive a car but not drive an object. 8 bit numbers are a subtype of byte, but thats unidirectional $\endgroup$
    – Hobbamok
    Commented Dec 12, 2023 at 10:23
  • 2
    $\begingroup$ @Hobbamok I said "8-bit numbers" instead of "bytes" because the context of the discussion makes clear that if I had said "bytes", the person I was talking to could have quibbled over whether I was using the word "bytes" correctly, and I thought that would be a useless discussion to have. Since you want to have it anyway, I think there is nothing wrong with using the word "bytes" as the question does to refer to a numeric type, and I think it is clear that the question is about numeric types and not opaque bit patterns. (It's tagged integer and it asks about signedness.) $\endgroup$
    – kaya3
    Commented Dec 12, 2023 at 18:00
20
$\begingroup$

Bytes should be unsigned

There are a few reasons most languages have unsigned bytes by default:

  • A more useful range. Bytes can only represent 256 different numbers so using them effectively is extra important. It is more common to want to represent a bigger number like 200 than -1, so the 0-255 is more practical than -128-127.

This isn't really an issue for bigger numbers that usually default to signed. Even an signed short can go up to more than 30,000 which is not something you would typically count to.

  • Bytes are most often used as a sequence, rather than as an actual number. You rarely use a type like a byte to store numbers intended as numbers, more often they represent bits of binary data, characters etc.

The "numeric properties" only get in the way of accessing the raw data. Adding a special meaning to the upper half of the byte space would just add extra complexity to any serialization, parsing, string manipulation, etc. code that may want to operate on byte sequences.

$\endgroup$
7
  • $\begingroup$ I'd like to suggest one additional reason to keep them unsigned (this isn't enough to warrant a separate answer): it makes the hex representation consistent. For example, Java's bytes are signed, which means you can't do byte b = 0xff (that's out of range). But if you have byte b = -1 and then view it as hex, that becomes 0xff. This is true of any signed type, but it's very common to view bytes as hex, so it's especially noticeable there. (This is basically a special scenario within your point that "bytes are most often used as a sequence, rather than as an actual number.") $\endgroup$
    – yshavit
    Commented Dec 11, 2023 at 18:15
  • $\begingroup$ I think you meant to say "Even a signed short can go up to more than 30,000", since you're making the point that even 32767 is often enough, not needing UINT16_MAX 65535. $\endgroup$ Commented Dec 12, 2023 at 13:26
  • 1
    $\begingroup$ @yshavit: I would think hex notation could be improved by saying that 0y123 would represent a number whose bits are all set except for the last 12 bits which would represent the bit pattern 0x123. The number below zero would in hex thus be represented interchangeably as 0yFF, 0yFFFF, 0yFFFFFFFF, etc., without needing to treat the number of leading F's as significant. $\endgroup$
    – supercat
    Commented Dec 12, 2023 at 23:50
  • 1
    $\begingroup$ @supercat I'm not quite sure what you mean, sorry. Are you suggesting an alternate form of representing hex digits, that basically represents just the negative-int space of twos complement? If so, I think the thing is that very often when you're representing hex, you're actually thinking of it as a bitset — not as an int, signed or otherwise (html color codes are one obvious exception). But having the underlying data type be an unsigned int maps a bit better to that bitset perspective than a signed int would. $\endgroup$
    – yshavit
    Commented Dec 13, 2023 at 1:01
  • $\begingroup$ @yshavit: The idea would be that 0y followed by digits would yield the signed value that starts with an infinite number of F's followed by the specified value. Especially useful when masking the bottom bits of a number, and one wants to leave the upper bits alone. $\endgroup$
    – supercat
    Commented Dec 13, 2023 at 4:31
10
$\begingroup$

Signed bytes make bit-packing difficult, because when a signed byte is converted to a wider integer type, sign extension means that the wider integer might use more than just the low 8 bits. Consider packing e.g. an IPv4 address into a 32-bit integer: the signed version needs & 0xFF to unset the sign-extended bits.

fn pack_ipv4(a: u8, b: u8, c: u8, d: u8) -> u32 {
    (a as u32) << 24
    | (b as u32) << 16
    | (c as u32) << 8
    | (d as u32)
}

fn pack_ipv4_signed(a: i8, b: i8, c: i8, d: i8) -> u32 {
    (a as u32 & 0xFF) << 24
    | (b as u32 & 0xFF) << 16
    | (c as u32 & 0xFF) << 8
    | (d as u32 & 0xFF)
}

The main reason to have bytes at all is their compact representation. So it should be as easy as possible to convert between the packed and unpacked representation. Unsigned bytes are better for this, since conversions don't involve sign extension.

$\endgroup$
2
  • $\begingroup$ Making the signed version output a i32 might be a more fair comparison, since you wouldn't need to set the sign for the leftmost byte $\endgroup$
    – mousetail
    Commented Dec 10, 2023 at 13:23
  • 1
    $\begingroup$ @mousetail Well, you wouldn't need the & 0xFF for the left-most byte either way, because the shift will drop all of the sign-extended bits. But personally, either way, I would write it this way because it's more obviously correct. The alternative would be to omit the one & 0xFF and write a comment explaining why it's not needed. $\endgroup$
    – kaya3
    Commented Dec 10, 2023 at 13:25
8
$\begingroup$

This still boils down to "what do you intend to do with them?"

As others have noted, sign extension is a nuisance when deserializing a wider integer, which suggests that unsigned is more useful.

Conversely, if x is signed, x<0 will tell you whether its top bit is set without having to know how wide it is.

I'm not convinced by arguments over the semantics imposed on the bytes; everything is a bunch of bits if you look closely enough, and arithmetic operations can be useful low-level operations on things that are not semantically "numbers".

Even the idea of "number" is a conflation of several things, which ideally a type system would keep distinct:

  • identifiers
  • ordinals
  • quantities
  • sequences
$\endgroup$
4
$\begingroup$

If they're signed, then you might have to worry about sign representation issues.

If they're unsigned by default, then their set of values will always have a consistent mapping to their bit pattern, regardless the sign representation.

But this almost doesn't even matter, because Two's Complement is the prevailing sign representation, and others are seldom, if ever, seen, especially in 2023.

$\endgroup$
3
$\begingroup$

I feel like someone has to make the argument from the other end, so I guess that person is me.

CPUs typically support both

In the vast majority of cases, CPUs don't care. It's just as easy for them to do signed 8-bit arithmetic as unsigned 8-bit arithmetic.

However, there are some asymmetries. The SSE2 pcmpgtb instruction, for example, performs a 16-way parallel greater-than test on signed 8-bit values. Unlike ARM Neon, there is no equivalent instruction for unsigned values.

Programmers can, of course, do various shenanigans if less-than is the only operation you need; just subtract 128 from each value prior to performing the test, for example.

Nonetheless, if optimisations such as automatic vectorisation are desired goals of a language, not allowing access to the "fast" version would seem to be a bad idea.

$\endgroup$
1
  • $\begingroup$ If you're considering vector operations, then rather than add/subtract, what you want is XOR, since you can flip the sign bit of each operand within a word in a single operation. e.g. if you have 4 16-bit values in each 64-bit word, you can use for (size_t i=0;i<n;++i) word[i] ^= 0x8000800080008000; to ensure that the subsequent signed vectored comparison gives the correct answer $\endgroup$ Commented Jan 25 at 14:11
2
$\begingroup$

A file is an ordered set of bytes.  A string is an ordered set of characters.  A network packet is an ordered set of octets.

We wouldn't think of the file as being composed of signed bytes, nor the string composed of signed characters, nor the network octets being signed.  These argue for unsigned bytes.

I'd like to hear analogous arguments in favor of signed bytes, but I can't think of any.

$\endgroup$
5
  • 1
    $\begingroup$ When we think of a file as a sequence of bytes, we don't even assume that those bytes are supposed to be interpreted as numbers ─ often they are supposed to represent characters, for example, or some other kind of binary data. If 01100001 represents the character 'a' then it doesn't matter to us what it would mean if it represented a number instead. So I don't think this really explains why an 8-bit number type should be unsigned. $\endgroup$
    – kaya3
    Commented Dec 9, 2023 at 19:06
  • 1
    $\begingroup$ @kaya3 Isn't that exactly what this answer is arguing for? A byte is not a number, they do not have signs. For u8 and i8 one should provide both options. $\endgroup$
    – Bergi
    Commented Dec 10, 2023 at 12:52
  • 3
    $\begingroup$ If a bit pattern isn't meant to represent a number then yes, it cannot be signed; but also it cannot be unsigned, because that is also something which only makes sense if the type is numeric. You could equally argue that strings, lists and hashmaps cannot be signed (because they aren't numbers) so therefore they are unsigned; but nobody goes around talking about unsigned strings or unsigned lists, and it would be nonsense to ask if hashmaps should be signed or unsigned. So I think the question must be understood as asking about numeric byte types. $\endgroup$
    – kaya3
    Commented Dec 10, 2023 at 13:04
  • $\begingroup$ Perhaps you're arguing for an opaque byte type, neither signed nor unsigned. One can do with bytes whatever is supported, i.e. convert to string, write to file, send as packet. But not inspect their individual values without make a choice of what numeric representation you want to see. Perhaps that choice should be left to the code to decide on demand and at use rather than pre-determined by numeric properties of a type. $\endgroup$
    – Erik Eidt
    Commented Dec 10, 2023 at 15:37
  • $\begingroup$ Sadly I've seen large projects (cough Exim cough) where strings are now "unsigned" because the author couldn't identify the few cases where implicitly signed char was causing a problem, so instead they did a huge bulk update to make everything explicitly unsigned char, and then added a bunch of conversions for when library calls expected bare char. Those conversions were blind casts, meaning they weakened the typechecking for future code. $\endgroup$ Commented Dec 26, 2023 at 4:31
2
$\begingroup$

I would suggest having four integerish types of each size except the largest, specifically "signed number", "signed ring", "unsigned number", and "unsigned ring". The largest size should have all but "unsigned number". Specifying things this way will ensure that for any computation performed between two integer "number" types, there will always be a type that can hold the values of both operands. Since "byte" wouldn't be the largest type, it should be available in all four forms.

The difference between number types and ring types would be that number types would be expected to behave like natural integers for all non-erroneous computations, while n-bit ring types would have specified "wrap-around" behavior similar to a ring mod 2ⁿ. An operation between two number types promote both operands to to a signed type large enough to handle both operands, an operation between a number and a ring would convert the number to the ring type, and operations on ring types would always yield a result of the same ring type (or an error if an operation on incompatible ring types would prevent that).

Although treating computations involving unsigned "number" types as though both were promoted to a larger signed type may not be the cheapest way of handling computations involving such types, it would be less likely than other approaches to yield results that were astonishing and inconsistent with programmer intention, especially if programmers use ring types when wrapping is intended.

$\endgroup$
10
  • $\begingroup$ Can you explain how to type literals in your proposal? I will not follow it in Tyr as I do not share your view on numbers, but I would be interested in understanding it. It could be interesting if I'd be tasked to work on a more math-oriented language. $\endgroup$
    – feldentm
    Commented Dec 11, 2023 at 15:57
  • $\begingroup$ @feldentm: Storing an integer literal or other integer "number" and a ring type would implicitly convert the number to that type. Likewise using a binary operator with a number and a ring type. The only times I could see any need to care about the type of an integer literal would be in cases where it was an operand to a binary operator which could use a type smaller than the largest type, but whose result was used as the left-hand operand of a right-shift, division, or remainder operator, or where the type might affect operator overloads selection, and I think casting in such cases... $\endgroup$
    – supercat
    Commented Dec 11, 2023 at 16:06
  • $\begingroup$ ...would seem like a reasonable approach. $\endgroup$
    – supercat
    Commented Dec 11, 2023 at 16:06
  • $\begingroup$ @feldentm: Suppose foo and bar are of an unsigned type representing the ring of integers that are congruent mod 65536. Saying foo = bar+123; when bar is congruent [mod 65536] to 65530 would make bar congruent [mod 65536] to (65530+123), which would in turn be congruent [mod 65536] to 117. Because 123 and 65,659 are congruent mod 65536, saying foo = bar+65659; would have the same effect. Whether the literal was an 8-bit, 16-bit, 32-bit, or 64-bit value would be irrelevant to the result, which would always be truncated to 16 bits regardless. $\endgroup$
    – supercat
    Commented Dec 11, 2023 at 18:31
  • $\begingroup$ So you would add an omnipotent literal type? I'm struggling a bit with what the CT evaluation of something like 2 << 64 + 1 would be. But maybe I make some hidden assumptions here. $\endgroup$
    – feldentm
    Commented Dec 12, 2023 at 17:02
2
$\begingroup$

The question "should bytes be signed?" isn't really a meaningful language design question. Generally speaking, we don't start by deciding what names we're going to have and then figure out what those names should refer to. Instead, we start by deciding what things we're going to have, and then we figure out what those things should be named.

The meaningful questions are:

  1. Should your language offer an unsigned 8-bit integer type?
  2. Should your language offer a signed 8-bit integer type?
  3. What should the above types be called? In particular, should one of them be called "byte"?

The answers to questions 1 and 2 depend on what your goals are. At one extreme, Python 3 offers no fixed-width integer types; the only integer type you get is int, which theoretically can hold integers of any size whatsoever. That's because Python has a focus on ease of use, and for most purposes, an arbitrary-sized integer type is very easy to use. Near another extreme, the LLVM language offers integer types of every width from 1-bit up to 8388608-bit. That's because LLVM has a focus on being able to produce efficient machine code, and all bit widths are potentially useful for efficiency.

I will mention that by convention, the range of numbers from 0 to 255 seems to be used much more often than the range from -128 to 127, so if you're going to offer only one type or the other, you should probably offer the unsigned 8-bit integer type.

Once you've decided which types to offer, the next question is what to call them. Rust takes a very simple and sensible approach to naming integer types: an unsigned 8-bit is called u8, a signed 8-bit is called i8, an unsigned 16-bit is called u16, a signed 16-bit is called i16, and so on. It's hard to find any fault with this system!

In common usage, the word "byte" generally means "nonnegative integer less than 256," so if you have an unsigned 8-bit integer type, naming it "byte" would certainly make sense. If you have a signed 8-bit integer type, naming it "byte" would be confusing; something like "int8" would be a much better name.

TL;DR: Don't name anything a "byte" until after you've already decided what integer types you're going to have. It's more useful to have an unsigned 8-bit integer type than a signed 8-bit integer type. After you've decided on your integer types, either name the unsigned 8-bit integer type "byte," or don't name any of them "byte."

$\endgroup$
4
  • $\begingroup$ Tyr has both types for quite some time. Could you cite some statistic on why one would use signed over unsigned? $\endgroup$
    – feldentm
    Commented Feb 10 at 9:53
  • $\begingroup$ @feldentm I don't have any such statistics. $\endgroup$ Commented Feb 10 at 14:41
  • $\begingroup$ It’s weird that you mention Python as a "counterexample" since it does have bytes which are immutable arrays of fixed size values. There just isn’t any specific type for an individual byte (which are commonly represented by a single-element bytes or an int in the range 0-255). $\endgroup$ Commented Feb 12 at 10:24
  • 1
    $\begingroup$ @MisterMiyagi Well, I wasn't really intending to give Python as a counterexample of anything. I was giving it as an example of a language that offers neither a type that means "an unsigned 8-bit integer" nor a type that means "a signed 8-bit integer." $\endgroup$ Commented Feb 12 at 12:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .