Also really cool to see more and more rust crates releasing their 1.0 versions. I feel like one of the big jumps for Rust (for me at least) will be when critical non-stdlib libraries are all 1.0 and there's less "which one do I choose". I understand competition is good in situations (more so opinionated frameworks) but I do love that I don't think about what regex library to choose since there's one obvious winner with tons of community support and resources.

But also, consolidation at a given layer of an ecosystem allows that layer to be treated as a foundation so the next layer on top of it can be where the competition happens instead

> I’ve always been a bit puzzled

I'll respond with my reasons, but I want to emphasize that I am be descriptive, not prescriptive.

The main reason why I don't just start with 1.0 out of the gate is because I generally want 1.0 to indicate some level of maturity and stability. That is, once I publish 1.0, ideally, I won't publish a 2.0. Or if I do, that timeline will be measured in years. It takes a while to get that kind of confidence with a library's API. If I had started with bstr 1.0, then this blog post would be talking about bstr 3.0. Not 1.0. Empirically, bstr 1.0 would not have been the commitment to stability that I want 1.0 to mean.

So, first question at this point is usually: well why not just increase x in x.0.0 as needed? It's okay to have 1.0 and 2.0 and 3.0. We have semver after all!

What I say to that is, yes, absolutely, you can do that. But it's absolutely a preference with respect to how often you want to release breaking change releases. My preference is to do it very rarely. Or as rarely as I can manage. The main practical reason for it is that breaking change releases create churn, and they lead to transition periods where, in the best of cases, compilation times take a hit.

For example, if I released regex 2.0, no code would break. At some point, people would start migrating to it. And for some period of time, it's likely that many projects would be building both regex 1.0 and regex 2.0 in their dependency trees. regex is not exactly lightweight, and so people are going to hunt down these issues in their trees and get everyone to migrate to regex 2.0. It's work. It's tedious. It's annoying. If I start putting out new breaking change releases of the regex crate frequently, then I'm going to annoy people in a way that is proportional to the frequency of releases. By committing to a policy that 1.0 means "I'm unlikely to publish a breaking change release for at least a few years," then that 1.0 is going to be a signal to folks that they are signing up for a dependency that is probably not going to cause them churn.

It's also especially important for bstr, because folks want to use it as a public dependency. So if I'm releasing semver incompatible releases frequently, then that's going to cause a lot of painful churn for users of bstr. It no longer just becomes a matter of compilation times. But you'll need to get your entire dependency tree migrated over, or else you risk things not working if multiple crates try to interoperate via bstr's API.

I suppose the next question at this point is, "but it's just a version number, why attach special significance to it that isn't in semver?" semver is useful for communicating breaking changes. And I think it's also useful to use the version number to communicate stability as well. But to be totally clear here: I am (EDIT) NOT trying to be an advocate here. I'm not saying this is what you or what everyone should do. There are trade offs here. I tend to build library crates that others build on, so my bias is to move slowly. But if I built crates (and I do) that are closer to the application (or even an application itself), then I'm generally much happier to just push out breaking change releases at a higher frequency.

I think the last question is, "but anything goes in 0.x.y, so says semver, so now people never know if they're getting a breaking release or not." Indeed, that is what semver says, and if that were how Cargo implemented semver, I'd probably start with 1.0 releases. (Or at the very least, publish a 1.0 release much much sooner.) But Cargo does not implement semver that way. With Cargo, 0.x.y is semver incompatible with 0.(x+1).z. That is, incrementing the leftmost non-zero digit in a version creates a new semver incompatible release from Cargo's perspective. So I get all the benefits of semver when I use 0.x.y, without needing to publish 1.0.0. The main downside is that the 'minor' and 'patch' components of the version number get collapsed into one number. But I can live with that until I publish 1.0.

Worth highlighting this part. If a crate exposes types that are likely to be exposed from the public APIs of other crates, then that changes the calculus of how disruptive a breaking change will be.

Second, and probably more interesting: Have you thought of some ways to formalize these notions of stability in different situations?

Third, dreaming: I can imagine a space for some kind of cross-language, computer-readable description of what functions/methods/identifiers have been (i) removed; (ii) deprecated; (iii) changed; (iv) added. In some cases, such a description could be extracted by static analysis.

2) No.

3) See: https://github.com/rust-lang/rust-semverver --- But also, this is only ever going to be a "best effort" sort of thing. Semver isn't just about method additions or deletions, but also behavior.

In effect, this lets you get the benefits of semver (communication of breaking changes) without publishing 1.0. Thus, there is less pressure in the Rust crate ecosystem to publish 1.0.

It's especially helpful that the library doesn't require you to opt into its own dedicated types, and instead defines extension methods on existing types.

Thanks, Andrew!

Fun fact: bstr 0.1 went the route of defining its own dedicated types! See: https://docs.rs/bstr/0.1.4/bstr/

But it did indeed quickly prove to be pretty annoying. Because you still really want to use &[u8] in places because it's so ubiquitous. But to get access to the byte string methods, you had to explicitly convert it to another type.

The reason why I went that route initially was so you'd always get the good Debug impl. But it ended up not being worth it sadly. This issue discusses it a bit more: https://github.com/BurntSushi/bstr/issues/5

> Note also that the files are sorted before concatenating, so that the result is guaranteed to be deterministic.

No locale was defined and the example sort command used cannot be considered deterministic. The results could vary wildly on different systems just through the locale alone!

Two solutions: define the locale ("LC_ALL=C" before the command should be sufficient), or use the -V flag on sort.

> Some folks have expressed a desire for bstr or something like it to be put into the standard library. I’m not sure how I feel about wholesale adopting bstr as it is. bstr is somewhat opinionated in that it provides several Unicode operations (like grapheme, word and sentence segmentation), for example, that std has deliberately chosen to leave to the crate ecosystem.

Yes, ok, but could we -at least- have the same Debug impl bstr has ? I'd love to be able to print "human-readable" Vec<u8> :')

So the Debug impl for Vec<u8> is just the Debug impl for Vec<T>. Doing otherwise means specializing for Vec<u8>, and it's not totally clear to me that it makes sense to do that. Doing it effectively requires assuming that a Vec<u8> everywhere is UTF-8 or close to it.

I do mention that we could add a '[u8]::debug_utf8()' method that returns a type with a nice Debug impl for byte strings. Kind of like how we have 'Path::display()', but for the Display impl. But that is kind of annoying in a way that doesn't really apply to Display impls. It's very common to derive(Debug), and if the debug impl is only accessible via a method, then derive(Debug) doesn't work. So then you have to write your Debug impl by hand, which is... annoying.

Anyway, point is, it's just not totally straight-forward to bring bstr into std.

As I said in the blog, I think the highest value thing that could be brought into std is substring search that works on &[u8].

The right way of doing this is to define custom attributes as part of derive(Debug) and derive(Display); the derive mechanism can already do this. There's no need for a wrapper type to be used.

Byte strings can be WTF-8. They can be anything. The problem is that there is no real way to (easily) get the underlying WTF-8 bytes of an OsStr on Windows. So there's no free conversion to and from byte strings.

I wrote more about this in the bstr docs (and don't miss the link to os_str_bytes): https://docs.rs/bstr/latest/bstr/#file-paths-and-os-strings

I'd be happy to answer more questions if you have them. :-) https://github.com/BurntSushi/bstr/discussions

¹) https://docs.microsoft.com/en-us/dotnet/api/system.span-1

²) https://docs.microsoft.com/en-us/dotnet/api/system.readonlys...

Oh, that's a solvable problem. Thanks!

Consider, for example, how you might use a routine like memchr[1] if all of your public APIs are generic iterator adapters.

And then once you get into things like regex engines, modifying them to work on Iterator<Item=u8> is a highly non-trivial affair. It is of course possible to write a regex engine that works on such things, but it's going to be limited in performance or capabilities. The way to make regexes and streaming work together is probably something more like Iterator<Item=&[u8]> (which is perhaps roughly analogous to what Hyperscan does). You really want blocks of bytes, not one-byte-at-a-time.

[1]: https://github.com/BurntSushi/memchr/blob/8037d11b4357b0f07b...

So I don't see how your links are relevant here.

However, I do agree that Erlang's Bit Syntax is, by far, the best handling I've ever seen for slicing and dicing bits and bytes at the very lowest level.

See if you can guess correctly what this produces:

    <<1:5, 2:4, 0xFFOO:15, "foo":binary>>

You use a hex literal syntax that doesn't exist--16#FF00 is Erlang and how on earth did you get "O" instead of "0".

Once I fix the brokenness:

10> <<1:5, 2:4, 16#FF00:15, "foo">>.

<<9,127,0,102,111,111>>

Looks ... unsurprising? Maybe it would be nice to get a truncation warning on that 16#FF00:15, but otherwise ... fine?

This looks like an advertisement that Elixir kind of sucks and you should use straight Erlang instead?

Note from downthread: this is broken even in Elixir, so I really don't know what your point was.

If you mean this equivalent bit of Elixir, then I got it right:

<<1::5, 2::4, 0xFFOO::15, "foo"::binary>>

I think it's all good and sensible, except that "0xFF00::15" should raise an error. The bit syntax silently masks out some stuff which can hide some errors.

Ok. Maybe I'm just dumb. But I was really surprised that the bit order goes from msbit to lsbit (while the byte order goes from lsbyte to msbyte). Since the 15 bit element crosses over the byte boundary (and overflows it) the arrangement -- 00s in byte 1 and 0x80 0x3F in bytes 2 and 3... is not obvious, to me).

In particular this makes it tricky to reinterpret packed structs coming from c, when using NIFs

Specifically, what I would have wanted, was lsb-msb in bytes, and an overflow error, or syntax in the descriptor modifier that is required if you might overflow.

Or, this would all be solved if we used an RTL layouts like the divine one intended.

You get the bits in the order you ask for in the expression, and then it encodes each valuee as you ask. Note that the default encoding is big endian, so the value 0xFF00 comes out as the bit sequence 1111111100000000 wherever it is.

Now converting from a C-struct you absolutely do have to think about byte order, but erlang has you there too. Let's say you've got this:

  #pragma pack(1)
  struct {
    uint16_t a;
    uint8_t b;
    uint8_t c;
    uint32_t d;
  };

  <<a::16-native, b::8, c::8, d::32-native>> = my_block_of_data

As I said above, erlang defaults to encoding/decoding each value as big-endian (e.g. <<0xFF00::16>> is equivalent to <<0xFF00::16-big>>. This is the default because big endian is used for the vast majority of comms protocols (in that context it's also called 'network byte order'). When interpreting values in memory though it can be either, depending on your processor (although mostly -little these days). Using -native as yoru endianness means "use big or little as appropriate for my processor". It's also a good signal that you're interpreting something from memory (as supposed to a protocol).