seanmonstar

Micro: Accessors can reveal internal representation

Tue, 21 Jul 2026 00:00:00 +0000

As a library maintainer, it’s common to receive requests from users to add a “simple accessor”. The data is right there. But that accessor can end up revealing internal representation details that you can no longer change.

This is one reason why it could be a better practice to return an impl Iterator instead of &[T]. The latter freezes you in place to always storing the data in consecutive memory. (Such as String::as_bytes().)

View types might help too, like Ref<'_>. Returning a reference requires the data already exist in that shape in self. You can’t change its shape in the accessor and return a reference to that.

Micro: Rust Errors generally shouldn't be PartialEq

Mon, 13 Jul 2026 00:00:00 +0000

Errors might have internal details that don’t make sense to compare. Like positional data, or a source chain. Even if you don’t have those details yet, committing to a public API of PartialEq can restrict you from internal refactoring.

The common reason to want this is for testing purposes. People want to assert an error case matches what they expect. But that doesn’t need equality. matches! , kind(), or is_*() patterns fit better.

A public, exhaustive, flat enum could choose differently. But those look more like a kind(). If you have private error details, I don’t think they make sense for comparing.

Some interesting discussion a few years ago.

Micro: Owning my microblog with POSSE

Tue, 23 Jun 2026 00:00:00 +0000

POSSE is the practice of publishing on your own site first, and pushing that out to other services. To get the content where people are, while linking back to the original owned post.

I now have a /micro blog on my own website, and automation (a GH action) to break up new posts and send them to Mastodon and Bluesky as threads. It even has it’s own rss feed.

This new setup lets me nudge people more towards my own property. As various other properties come and go, my online identity and content lasts longer.

And it solves one of my longer-standing concerns about microblogging: it feels like they can be more important content now, because they’re mine.

hyper User Survey 2025 Results

Tue, 26 May 2026 09:02:00 +0000

hyper is the widely-used HTTP library for Rust. We ran the first user survey for hyper in November 2025. Here’s the results and some analysis.¹

Why did we do a survey?

We ran a survey to make sure we focus on the right things. It’s part of being user-driven, working backwards.

The amount of work required to get some data has an outsized return on your investment. When you know nothing, just a little bit of work means you now know something. And knowing what users need is a requirement to making an actually useful tool.

I keep high-touch, high-context relationships with some users that greatly informs me on what is needed. And reported issues also provide a way to see what is wrong. But a survey provides a new lens for identifying what is needed.

Sure, surveys have their own biases to deal with, such as self-selection. And voting in a survey costs very little, so each individual answer has less weight. But it helps quantify things we supposedly know from just a few users. Does this problem only show up in weird circumstances, or do most people run into it? Are there things that many people want, but generally don’t get a chance to tell me?

The results

We had a total of 206 unique respondents. I’ve broken up the results by theme and question below.

Current usage

Many of the questions were about current usage patterns. To get more context on answers, to see what works, and also to see what people don’t mention using.

Versions: Nearly every single respondent is using hyper v1.x, but about 13% also continue to use the end-of-life v0.14.x.

Roles: Over 80% said they use hyper in a server, and the same amount in a client. 30% said they use both modes together as a proxy.

Runtimes: Unsurprisingly, 99% use hyper with the Tokio runtime. But 10% said they also at times use a different runtime. Most commonly that’s with smol, but io-uring runtimes made up a couple percent points.²

TLS: hyper provides the HTTP, and asks users to bring the S. 93% said they use rustls. 30% make use of native-tls, 15% openssl specifically, and it trails off from there.

On top of hyper: regarding the rest of the local ecosystem, 86% use reqwest, and 79% use Axum. About half selected Tower. For future surveys, it’d be very helpful to know if people use Tower without Axum or reqwest, or perhaps mentioned it because they just know its a dependency of Axum. A few wrote-in Tonic, which in hindsight, duh of course.

Number of engineers: half of respondents said they (and possibly a colleague) are the only ones at their company using hyper. Another 22% said 4-10, 12% said 11-50, and 17% said 50 or more. It’s likely some places focus on higher levels, but many other companies have a lot of engineers working with hyper!

Years of usage: Nearly half of respondents have been using hyper for 1-3 years. 12% for less than a year, 30% for 4-6 years. And 11% said they’ve been using hyper for forever. Me too, me too…

Industries: 111 responses, which I grouped and sorted from most common to least. Software, cloud infrastructure, security. Finance, healthcare. Robotics, automotive, space, EV charging. Embedded. Media, streaming, events. Databases. Education. Government. Telecom, radio, chat. Geospatial. AI. Browsers. Games. Rust compiler.

It’s just so humbling to think this code is helping in so many different places.

Future work

Another set of questions were related to work we could do in the future. This helps inform prioritization, and things we may not have thought of.

Feature requests: One of the questions came preloaded with common feature requests, and asked the respondents to rank them. Here’s the accumulated ranking:

Metrics/Tracing/Events API
HTTP/2 Performance
Better Middleware
Add HTTP/3 Support
Documentation and guides
io-uring support

Write-in features: the follow-up question was a blank text box to write in any feature requests not in the above list. I grouped the common answers and sorted: legacy-client successor, original header casing/ordering, buffer control, improved errors.

One write-in suggestion wasn’t really about a new feature, but keeps popping back up in my mind: how to get releases that are more stable.³

The hard parts: We also asked what was hard about hyper. Grouped and sorted answers: integrating TLS, upgrading to v1, middleware complexity, advanced body streaming, combining all pieces for production.

Topic ideas: we asked for suggestions for topics for future blog posts and talks. This certainly doesn’t need to be only from me, so if you see something interesting, fill the need! But after grouping the topics, they are: hyper internals, best practices, testimonials and success stories, security, tower usage, retries.

Contributing

We also asked about some general contribution questions, to look at the contributor health of the project.

The standout here was: what would help you to contribute more?

The top response, a little over half of all respondents, said an improved contributing guide. Just a few percent under that want an updated roadmap. 20% said more responsive triage and reviews. And 15% said they could use increased mentoring.

The insights

Looking it all over, this was awesome. I definitely to take some specific things with me. Well, sure, there’s goodies all throughout the above results. But, at a high level, here’s some thoughts, especially after combining it with other conversations I’ve had.[^not-everthing]

Of the technical things, I suspect I’ll be spelunking in h2 first, eking out performance refactors. It can use some love. Metrics is requested all the time, and it looks like I can collaborate on a design and review, but I believe there’s a couple other contributors interested in the implementation. I do keep thinking about ways to configure a buffer pool or something. I think the extended testing idea has some potential.³ And I’d love to see HTTP/3 finally land in hyper proper this year, at least as an unstable feature. I want the other things too, but… time, y’know? Though, as mentioned in the next paragraph, it’d be a win to onboard others to work on and own those things.

Improving contribution health, sustainable personnel of hyper, is high on my list this year. I’ve started working on a collaborator guide. I’m considering some new collaborators that have already been doing good work. I want to try GitHub’s rotating auto-assign reviewers, to improve review times and spread review load. And I welcome help linking up our existing ROADMAP into some more concrete and mentored issues.

Outro

There it is. The hyper user survey 2025 results. Thank you to everyone who responded!

Does any of this interest you? Want to run with one of the ideas? Join us!

This took a bit longer to write up than I meant. I have to admit, I felt some strong burnout at the beginning of the year. Sometimes being a maintainer is lonely. New year contract negotiations don’t help. But I’m feeling better now. ↩
Did you know you can bring your own runtime? Also, this is one reason I don’t believe the FUD about a single runtime. Also also, hyper::rt works fine, I don’t see any need for the types to be in libstd. That wouldn’t unblock anything. ↩
I can write this up more later, but initially, I’ve learned the slow way about feature windows. On top of those, I’d love to talk to customers who would be interested in having their test suites try hyper master before a release. Talk to me. ↩ ↩²

2025 in review

Tue, 27 Jan 2026 11:32:00 +0000

Come along with me as I review the past year. Heh, I often start these kinds of posts right at the start of the year, but it takes a few weeks longer than I ever expect to think them through.¹

Two years of being independent

After a second year of operating as an independent open source maintainer, it’s starting to feel more normal.

In terms of personal execution, it felt pretty fantastic, actually. Thanks to high-touch conversations from my retainers, I knew what was needed; there was an underlying trend. And I was able to spec out a grant that made a project out of that trend. All while managing to do the necessary maintenance work that the ecosystem requires. Granted, it did occasionally feel like a conflict of priorities, but that’s life.

Honestly, though, I wasn’t so sure when trying to plan this all out initially.

Maintaining hyper

Perhaps the biggest deal for hyper this year was launching our first user survey. I’ve thought of doing it a few times over the years, but finally remembered in Q4 to launch it. Thanks to all who answered! I’ve looked through the results, and I think this will be extremely useful. Some stats real quick: 96% of respondents have upgraded to hyper v1.x, most commonly combine it with Tokio (99%) and rustls (92%).² A proper analysis coming soon!

katelyn martin joined us as a collaborator, and has continued to be a multiplier with kind reviews and maintenance glue. And general maintenance doesn’t stop, including growing security reports (more below).³

Besides all that, I took on a larger project for the year. You see, after updating the roadmap at the end of the previous year, I started to focus on one of the four defined areas: improved hyper-util. This lined up with what many have been asking for.

I did that by modularizing parts out of reqwest.

Modularizing reqwest

Most of my year was spent on modularizing reqwest. Or, from another angle, giving back the building blocks that reqwest has accumulated over the years. A lot of functionality that people rely on in reqwest started life as internal glue, and this was the year I finally pulled many of those pieces out into places where the rest of the ecosystem could use them too. Between reqwest and hyper‑util, that work ended up producing quite a few releases: 14 for reqwest itself, and 8 for hyper‑util.

Ages ago, I added a bunch of features that you expect any client to have directly into reqwest. Later, as tower-http grew, it copied some of those same features. Meanwhile, reqwest was used in weirder and weirder places, so we hardened those features, and tossed in some tests to check for the weird. But tower-http never saw any of that.

This year, we completely tossed the redirect and decompression code from reqwest, depended on the tower-http pieces, and then allowed the test suite to find the difference. The tower-http layers got those fixes backported, and now everyone benefits.

We also created new things, but still modular.

Easier retries were added to reqwest, making use of the lower-level pieces in tower. I’m still interested in ways to improve the feature, so more people can use retries more safely.

reqwest has grown extensive support for connection proxies. But an increasingly common pattern was people using reqwest only for the proxy support; they didn’t need any other feature. So I extracted proxy matchers and proxy connectors (tunnel, socks) into hyper-util.

The largest piece was designing and implementing composable pools for hyper-util. In many ways, this was my highlight of the year. It’s a problem I’ve been thinking about since … 2018? I’d done a lot of research throughout the years, and never found anything quite like it. Now, it’s not quite “done”, but it’s a base that allows a lot of new layers and compositions to be explored.

To end the year, we released v0.13 with rustls as default. It’s a big improvement for most people. But. I am not currently happy with how difficult it is to build the defaults on some other targets (Windows, Cranelift, cross-compiling). I want that fixed. Maybe that’s improvements to upstream aws-lc-rs; it looks like it’s already been improved to not need cmake. Or maybe we use a different default crypto provider on some targets.

The value of deadlines

The work on composable pools was hard. The reason it had taken me years to finally try was that I wasn’t sure about some of the design. After staring hard at it during the summer, I did solve some of the questions. But there was one problem towards the end that consumed another month or so of staring. And this time, I couldn’t stop staring.

With a hard deadline set, however, there was no possibility of waiting longer. Instead, I had to settle with shipping what I had, and accepting that it can always be better.

And that’s also the beauty of deadlines: they keep you user-driven. As long as I’m staring hard at a problem, holding back shipping, users have nothing. But software doesn’t need to be shipped all at once. It’s a lesson I’ve learned before, and yet it pops up to, uh, delight me over and over.

I feel like I go through waves: I hate setting a deadline, and many times feel disappointed at not shipping all the glory that was in my head. But I always appreciate that at least they got something.

Security takes time

We take security seriously, and the amount of reports we receive is slowly increasing. This past year, we had a 8 in total, including our first AI slop report (yay!).

The biggest one resulted in a large coordinated disclosure event. But hyper was just fine.

That didn’t stop it from being stressful trying to handle reports while simultaneously sticking to feature deadlines.

It is a reminder, though, that this is often urgent and important work that must be handled, but that traditional pay-for-features doesn’t support. Sponsorships and retainers make this sort of maintenance much more sustainable.

Talks

On the 10th anniversary of Rust 1.0, I gave a talk for the Rust for Lunch meetup. It was sort of ‘lessons using Rust for 10 years’, but also ‘why you should consider Rust’.

And I did a podcast episode on Netstack.FM, discussing the history of Rust’s networking ecosystem.

Pondering

Last year, I liked just sharing some questions I’m thinking about. It wasn’t a promise to work on them actively, but I look at them from time to time to see if there’s something that I can tackle soon.

Here’s just a few things I’m thinking about at the start of 2026:

How do I balance keeping up with LLM advances while keeping my mind and skills sharp?
How far can one reasonably go with typestate builders, considering ergonomics and correctness?

I wonder.

See some previous yearly reviews: 2024, 2023, 2022, 2021. ↩
Though a significant amount also use other runtimes and TLS libraries. As mentioned, more coming soon. ↩
There were three feature releases of hyper and two of http, bringing support for things like on_informational and constants for EARLY_HINTS. There was also five bug fix releases of h2, hardening the HTTP/2 story for Rust. ↩

reqwest v0.13 - rustls by default

Tue, 30 Dec 2025 10:48:00 +0000

To end out the year, here comes a new major release of reqwest, the opinionated higher-level HTTP client for Rust.

We don’t really need major breaking versions to keep providing value. Improvements keep coming all the time. But we did need one to make one particular big adjustment, and we’ve taken the opportunity to clean up other things too. At the same time, we strove make it disrupt as little as possible, especially if you stick to the defaults.

reqwest v0.13.0 is out now! Read on for why.

rustls is now the default TLS backend

The biggest deal is that reqwest now sets its default TLS feature to use rustls, instead of native-tls.

Granted, native-tls has its place. It provides a unified library that uses the “native” TLS implementation on each target. SecureTransport on macOS, schannel on Windows, and OpenSSL on Linux (as the most likely already installed). It was the right choice several years, while rustls was young. But rustls is now safer and faster than most choices.¹ Seems like an obvious improvement. But would people want the better choice?

A recent hyper user survey found that 93% of respondants already use rustls. 30% said they also use native-tls at times, so we will continue to provide that option. And while a survey is already biased, it’s also most certainly true that the vast majority of users just allow the default options, and it works for them. That will continue to be true for most everyone.

So, if you use the default options, things just got better for you.

Certificate verification features are consolidated

Previously, reqwest had many crate features to enable various ways to load sources of root certificates. Too many. It’s a bit of a mess.

We consolidate the options into just a few simple choices:

Defaults to using the native platform verifier.
If the target supports it, tls_certs_merge() can add additional certificates. If it doesn’t, a builder error is returned.
If you really need a specific certificate, use tls_certs_only(), which will always work, and doesn’t include native verification.

These basic options should allow for any use case, while reducing complexity within reqwest. For instance, if an application really needs to use the webpki-roots, they can be configured with tls_certs_only(your_roots).

Soft-deprecation to improve option naming

While in there, most ClientBuilder methods were given better names. The previous methods have been soft-deprecated. That means that you can keep using the old names, without warnings, but they are documented as deprecated, and they will eventually be removed in a later breaking change.

But in most cases, the improved name is mostly to help with understanding. It doesn’t help to trigger a bunch of additional warnings when you upgrade. Perhaps we’ll add a deprecations crate feature for those who want to clean up.

The upside is that all the methods now start with tls_. They are grouped together in the docs, but it will also help exploration with autocomplete.

Other crate feature adjustments

As this is a breaking change version, we did include one making the RequestBuilder::query() and form() methods optional features, disabled by default. With these as optional, it is now possible to build reqwest without serde.

We also made native-tls imply ALPN automatically. This will mean most people who don’t think about it will now get HTTP/2 upgrades. The reason it was opt-in previously is because older native libraries might not have the symbols required. They are quite old at this point. If you still need the previous behavior, there is now native-tls-no-alpn.

Thanks!

Thanks to all who contribute, use, sponsor, fix, complain, and help reqwest be what it is! Here’s v0.13.0.

This does drop i686-pc-windows-gnu support for the default features. The reality is that a lot of things don’t work for that target. Even Rust itself has it as a Tier 2 target. We do still support it with native-tls, though. ↩

hyper-util Composable Pools

Wed, 03 Dec 2025 08:48:00 +0000

I’m so excited to announce hyper’s new composable pool layers!¹

As part of making reqwest more modular, we’ve designed a new connection pool, and made the pieces available in hyper_util::client::pool. But this is more than just a “hey, we have a Pool, it moved over there.” We’ve literally pulled apart the pool, in a way I haven’t found elsewhere.

Building a purpose‑specific pool is now straightforward. Add the features you want, even custom ones, and skip the bloat, no forks required.

Read on to see what exactly we solved, how, and what comes next. If you just want to use them, here’s the docs. Everyone else, let’s dive in.

We started with the users

We started with the users, looking back over past issues filed, common questions in chat, and private conversations explaining what they needed to do. Boiled down, that got us to these requirements:

A full-featured pool, like the one in legacy, must be possible.
Microservices shouldn’t have to handle multiple protocols or hostnames.
Some clients need custom keys for the pool.
Others need to limit new connections made at a time.
Or cap the total number of connections.
Customize connection expiration based on idle time, max lifetime, or even poisoning.
And importantly, allow custom logic not already thought of.

From past experience combining middleware, I had a strong feeling the pool requirements could be broken up into tower layers. But what would that even look like? Would it be horrible to use?

To answer that, we took the requirements and considered the developer experience of using layers. It had to feel nice. Not just to write, but also to come back to and read.

I then sketched out several of these layers to make sure they could actually work. Once most of it was working, the proposal was ready.

The initial 4 working pools

No plan survives contact with the enemy. We originally proposed five pool types, but launch with just the following four: singleton, cache, negotiate, map.

The singleton pool wraps a connector² that should only produce a single active connection. It bundles all concurrent calls so only one connection is made. All calls to the singleton will return a clone of the inner service once established. This fits the HTTP/2 case well.

The cache pool maintains a list of cached services produced by a connector. Calling the cache returns either an existing service, or makes a new one. When dropped, the cached service is returned to the cache if possible. Importantly for performance, the cache supports connection racing, just like the legacy pool.

The negotiate pool allows for a service that can decide between two service types based on an intermediate return value. Unlike typical routing, it makes decisions based on the response (the connection) rather than the request. The main use case is supporting ALPN upgrades to HTTP/2, with a fallback to HTTP/1. And its design allows combining two different pooling strategies.

The map pool isn’t a typical service like the other pools, but rather is a stand-alone type that maps requests to keys and connectors. As a kind of router, it cannot determine which inner service to check for backpressure until the request is made. The map implementation allows customization of extracting a key, and how to construct a connector for that key.

Ineffably unstable

I knew this work would land in hyper-util first, because it’s not stable yet. Being so freshly designed, changes are expected after some more real-world usage. Still, I wanted to shield earlier adopters from breaking changes. At the same time, valuing performance and flexibility, I wanted to push as much as reasonably possible into the type system.

When initially tinkering during the summer, I had one of those thoughts. The kind that clangs like a giant lock snapping open: what about type-state builders and unnameable types? I took a side quest, and tackled the warp v0.4 upgrade, to test out this API design. That post explains it a bit more.

The various threads were all coming together.

With each pool concept a tower service, once composed, a user shouldn’t care what it is beyond being some impl Service. I tested this out in reqwest, and yea, I don’t need to name the types. While I did need a type, I was able to store a dyn Service, and inference handled the rest.

Real world usage: in reqwest

Once those main pieces seemed ready, I needed a real example to test drive them. Tool-makers that don’t use their tools make bad tools, after all.

I started by replacing the legacy pool inside reqwest. Part of the larger diff in reqwest is handling all of reqwest’s different pool configuration options.

But, putting the default case together is pretty self-explanatory:

// Note: some noise has been trimmed
let http1 = (
    pool::cache(exec),
    util::http1_request_target(),
    util::http1_set_host(),
    util::meta(MyMetaIdleAt::new),
    conn::http1(),
);

let http2 = (
    pool::singleton(),
    conn::http2(),
);

let pool_layers = tower::layer::layer_fn(move |svc| {
    pool::negotiate::builder()
        .fallback(http1.clone())
        .upgrade(http2.clone())
        .inspect(|conn| conn.is_negotiated_h2())
        .connect(svc)
        .build()
});

let pool_map = pool::map::builder::<http::Uri>()
    .keys(|dst| scheme_and_auth(dst))
    .values(move |_dst| {
        pool_layers.layer(connector.clone())
    })
    .build();

And it works! Making the full-featured pool was one of the requirements: check. But, the next part was even more important.

As I mentioned before, I punted one of the proposed types: expire. Expiration is a necessary concept to a pool. But try as I might to fit the various generic shapes, it just wasn’t happening. Thankfully, this work had a hard deadline. And deadlines keep you user-driven: let them have something now, it can always be better later.

To prove the general design allowed expiration, I implemented a specific version of it directly in reqwest.

tokio::spawn(async move {
    loop {
        tokio::time::sleep(idle_dur).await;
        let now = Instant::now();
        let Some(pool) = pool_map.upgrade() else { return };
        
        pool.lock().unwrap().retain(|_key, svc| {
            svc.fallback_mut().retain(|svc| {
                if svc.inner().inner().inner().is_closed() {
                    return false;
                }

                if let Some(idle_at) = svc.meta().idle_at {
                    return now > idle_at + idle_dur;
                }
                true
            });
            svc.upgrade_mut().retain(|svc| {
                !svc.is_closed()
            });
            !svc.fallback_mut().is_empty() || !svc.upgrade_mut().is_empty()
        });
    }
});

The ease of adding it helped solidify to me that this was definitely the right design. I was able to slot in a meta layer tracking idle time, and then use that to retain services. I placed that layer in right next to some of the other HTTP/1-specific layers. Easy!

Being modular opens up customization

With the ability to build a stack for your pool, consider an example of how we can start to solve other requirements listed earlier.

let svc = ServiceBuilder::new()
    // cached connections are unaware of the limit
    .layer(pool::cache())
    // in-flight handshakes are limited
    .concurrency_limit(5)    
    .layer(conn::http1())
    .service(connect::tcp());

It also allows adding in layers we don’t currently have, such as per-host connection semaphores, or a few layers up over all hosts. Adding new functionality isn’t blocked on us, and no one has to “pay” for features they don’t need.

I can’t wait to see what else is done with the design!

Pools ready

The hyper_util::client::pool module is now available in v0.1.19. Go check the docs, and try to build cool things. Please file issues if parts are missing, we’ll keep iterating.

I’ve been working on this feature set for long time. It’s something I started thinking about years ago, and after months of work this year, it feels awesome to finally be able to release it.

Thanks to my sponsors, retainers, and grants for making this all possible!

I mean, who isn’t excited to announce anything? /s ↩
All “connectors” are actually MakeServices, which are jsut a Service that produces a Service. It doesn’t have to create a connection, but it reads better when talking about pools. ↩

hyper User Survey 2025

Tue, 18 Nov 2025 10:20:00 +0000

I’m excited to announce the inaugural hyper user survey! (Update: it ended.)

Part of being user-driven is knowing what our users want, what they’re trying to do, and then we can figure out how best to help.

This survey should only take about 5 minutes, no questions are required, and responses are anonymous. The survey will be open for a month, until December 17th, and then we’ll analyze the results and share with you all what we find.

Thanks for taking the time to help make hyper better!

~~Take the hyper user survey for 2025.~~

Podcast: Netstack.fm, story of Rust's networking with hyper

Tue, 02 Sep 2025 09:24:00 +0000

Last week I was a guest on the Netstack podcast. We talked abit about how I got into Rust, how async Rust developed, and the story behind hyper and its surrounding ecoystem.

We started (and ended) with my goal of better software:

On your about page, you say that “Rust is the least bad option.” Can you elaborate a bit on that?

Yeah, I love Rust. I think Rust solves a ton of problems, but I also don’t want to be stuck when something better comes along, which it inevitably will. Then yeah, I’ll move to that. To me, all these tools are the means to an end, which is to make better software.

The rest of the conversation was really fun to talk about:

7:54: beginning of hyper, async Rust
13:20: hyper as something bigger
15:36: splitting off hyper-util
17:35: the http and headers crates
22:51: motivation behind warp
29:00: reqwest as the opinionated layer
30:17: open source independence
31:31: HTTP/3 in hyper and reqwest
39:40: hyper in 5 years, contributors, ownership

hyper HTTP/2 (Didn't) MadeYouReset

Wed, 13 Aug 2025 08:25:00 +0000

A new HTTP/2 attack vector was disclosed today called MadeYouReset. hyper’s h2 is negligably affected, weathering the attack well. But, we have provided patches just in case. We published patches weeks ago, so if you’ve been keeping up-to-date, you’re fine! If not, you’re most likely fine, but you can upgrade now. No CVE or security advisory is included with this.

This sort of work is supported exclusively through retainers, get in contact to support.

Follow along for the background information, if you’re curious!

What is the attack

The attack is quite similar to the previous Rapid Reset attack from a couple years ago. That is, it starts new requests up to the limit imposed by the server. And then instead of sending explicit RST_FRAMES, it sends malformed frames that force the server to reset the stream.

So far, the described behavior is normal HTTP/2 flow. The problem occurs in some implementations that don’t respond well to quick cancelation.

As the attack describes, in some implementations it allows for more stream concurrency than might have been configured. See more in VU#767506.

What hyper does

A couple years ago, we added a defense mechanism to h2 that keeps count of how many times a frame from the remote causes a local reset, and once a configurable limit is reached, the connection is abruptly closed. This mechanism caught nearly all variants of the attack.

We had forgotten one code path dealing with WINDOW_UPDATE frames. But what happens when new streams are created and then a bad window frame is received?

In h2, the reset is sent to user code. Check the request body will notice the error. Additionally, hyper’s server code using h2 is essentially waiting simultaneously on the user’s service to reply, and for any reset to be received. Once the reset is received, the service future is canceled.¹

So, not much. When a simple hello world server example was tested against the attack, the only thing that happened was increased CPU usage. Which makes sense, it’s receiving and canceling more streams.

Stream concurrency is enforced, since all futures are canceled immediately, and a signal is sent to anything listening for longer.

All of this before any additional patch.

What we did

We received a report² about the attack from security researchers³ on May 23, 2025. The attack includes many variants, so we set out trying to determine if hyper was vulnerable. We knew that hyper already added a defense mechanism for this exact sort of abuse. Only one of the variants did not get cut off, but saw increased CPU usage, so we dug in. It turns out the original feature forgot to check the limit when WINDOW_UPDATE frames were bad.

We determined that while the variants was not caught, it also did not break any guarantees. The increase in CPU usage was obvious, since it was necessarily handling more requests in general. Also important to our determination: the attack is observable by user code, so any user code can react however they wish.

The researchers did a great job of explaining the concept behind the attack, providing a proof-of-concept script that we could test with, and conversed respectfully and thoughtfully as we went through the investigation.

Because the code of the aforementioned mechanism was already public, and the damage quite benign, we decided to release a patch weeks before the general embargo. It was just a refactor, converting it from logic-based to utilizing the type system more, so no other cases are “forgotten”. That patch was released on June 30, 2025.

For similar reasons, we also determined there was no need for a CVE or security advisory.

The researchers came back a little afterwards showing that in some cases, servers using h2 under this attack could be forced to crash. They even brough a new reproducible proof-of-concept. After analyzing it, we determined that the problem was not that of hyper/h2, but that of user code which might have unbounded queues and does not react to cancelation.

We decided to stick with our original determination.

What you can do

After reading all that, what more can you do? Again, you’re likely fine! But here’s some good ideas, regardless:

If you’ve been staying up-to-date, you’re already done!
If you have a server running hyper, it would be wise to make sure your server handlers really can cancel easily.
You can run cargo update -p h2 to upgrade to the latest versions (at least 0.4.11 and 0.3.27).
If you’re running hyper v0.14 (so h2 v0.3.x), it’s time to upgrade to v1. After nearly 2 years, patches likely won’t be backported anymore.

There’s one more thing you can consider: set up a retainer to make sure security reports get taken care of. This was a considerable amount of important work. This was several months of higher-stress but important work which doesn’t look like shipping a new feature.⁴

Besides funding that important work, I’m also happy to provide pre-disclosure and hands-on guidance to those with a retainer.

This difference from Rapid Reset is also important. In Rapid Reset, the resets were not observable by user code, so they could not react to the wasting of resources. With MadeYouReset, however, users can if they need to. ↩
I just realized today that I cannot publish the GHSA as an informational, since it does not include a severity or CVE. GitHub, please? ↩
The original reporters were Gal Bar Nahum, Anat Bremler-Barr, and Yaniv Harel of Tel Aviv University. I primarily interacted with Gal, which was a 5/5 experience, would repeat. ↩
On top of what has been outlined here, there’s a lot of coordination work involved, dealing with VINCE and other vendors. ↩

warp v0.4

Wed, 06 Aug 2025 11:05:00 +0000

Yesterday, warp v0.4 was released. warp is a Rust web server framework, with a focus on functional programming and type system routing.

Upgrading is likely pretty simple, the main API stayed very similar. The biggest deal is that it should be easier for you to stay up-to-date on dependencies. You could stop there. But if you want to know what else happened, off we go!

Filters remain the focus

The biggest raison d’être of warp is its Filter system. It was why I made a different web framework originally, and it remains the reason to consider warp even now.

Should you use warp? That depends. If you just want a standard, super fast, featureful Rust server framework, one that looks like ways you’ve coded servers before, you probably want Axum. But, if you like functional programming, and (ab)using the type system, I think warp is pretty cool.

Consider an example:

let update = warp::path!("todos" / u32)
    .and(warp::header("accept"))
    .and(warp::body::json())
    .map(|id, todo: Todo| {
       // ... 
    });

Every filter extracts something from the request (even if that’s just ()). They can all be .and()ed together in any order, it will result in natural, typed arguments you can map.

You can build up layers of filters by just putting another .and() on anything. You can make a group of filters with .or().

Once you have combined all into a master filter, you can convert them into a tower Service, which you can then add any other middleware from other libraries, and serve with an HTTP implementation, such as hyper.

Upgraded to hyper v1

The underlying HTTP dependencies, those which are public, have been upgraded to hyper v1.

This should improve interoperability if used with other libraries such as reqwest or tower-http. Their latest versions depend on v1 of hyper and http. So adding in more middleware, or using a higher-level HTTP client should mean less dependencies compiled, and no conversion required between types.

It also means warp users will be able to stay on maintained versions of hyper and its dependencies.¹

Crate features on a diet

In previous versions, a few heavy features were enabled by default. Now, the default features are much slimmer.

Keeping with filters being the focus, the default features only include the Filter system, and a way to convert them into an impl Service. You can take that service and use it with your own build of hyper, with HTTP/1 or 2 or whatever enabled, and warp won’t care.

If you want the simple server runner, just enable the server feature, and then warp::serve(filters) is back, along with all the hyper implementation to run it.

Likewise, the warp::test helpers have been put behind a test feature. You can enable it in your dev-dependencies for testing, and while keeping that code out when building for release.

The tls feature was dropped completely. It existed previously as a simple way to add TLS to your server, but it’s not strictly required in a server framework, and it adds a bit of maintenance churn to stay up-to-date and safe. Anyone wanting TLS with their warp server can make an accept loop using the TLS implementation of their choice, and use warp’s filters as a Service.

Finally, the multipart and websocket features were removed from default, but are still available to enable. This was a planned change once they became features in the first place, since they enable a lot of code that you might not need.

Iterating on the Server builder

The Server API was, uh, iterated on.

Before, it had a whole bunch of bind_*, run_*, and serve_* methods that slowly grew as people wanted more options to configure the server. Now, most all of them are gone.

Part of the reason is because it’s not the focus of warp. The Filter system is. I figured the Server is meant to be a simple way to get your filters serving requests. If you wanted more customization, make your own.

But there’s another reason, too.

I’ve been noticing more and more places where I wish the API of some crate was at the same time simpler, and also more powerful, putting options into the type system. The warp Server was an excellent place to iterate on this.

It is now an unnameable builder making use of expandable type state.

It encodes as generics into the Server<F, A, R, Etc...> various parts of how the server works. The filter, the acceptor, the runner loop. Depending on what methods you call, new ones get unlocked. For instance, bind() will make the acceptor a TcpListener, but once one is set, it doesn’t make sense to be able to call it again.

The type is rendered in the docs, but otherwise not publicly exported, so you can’t name it. There’s a benefit to this, as the API designer and maintainer. I can add more options, needing more generics, without it being a breaking change. As long as add a default that keeps it all compiling, there are now instances of Server<F, A, R> that need to be updated to Server<F, A, R, E>.

And the reality is that the name of the type is not generally interesting to you as a user. You just need a fluid way to add options, and once it’s all combined away, you just await a future.

I’m hopeful this pattern carries over well to other builders as well.

I’ve backported important bug fixes to hyper v0.14 (and dependencies like h2), but I think that’s about done. It’s been almost 2 years, time to focus on v1. (If you need help, reach out to me!) ↩

reqwest retries

Tue, 15 Jul 2025 10:39:00 +0000

One part of working on a more modular reqwest was adding support for retries.

Intermittent failures are a fact of life, and a simple retry could have made things so much better. It’s a somewhat frequent feature request for reqwest. It’s also a very common pattern to implement manually, and far too easy to do incorrectly. Since reqwest’s internals are shifting more and more to a stack of services, it felt like an ideal time to explore this.

While skeleton pieces do exist in tower::retry, they require Expert Mode. Doing things right is tricky. We can do better.

Policy composition

A retry policy is several things at once. But each of those things often is only one of a few choices. So what if we made it easier to compose your choices? That is, we can provide the common choices, and a builder to combine them.

Budgets by default

Retry counts alone are insufficient, but absurdly common. Please don’t do this. We can educate and document and warn and wave our hands, but we can only plead so much.

Budgets will be default on. Sure, there’s an escape hatch if you absolutely must. But this way we can help you fall into the pit of success.

And to help users better understand what the budget results in, configuration looks like .max_extra_load(0.3), instead of implementation parameters of the windowed bucket.

Scoped retries

Budgets are basically a must. But budgets also don’t make sense across different targets. Just because requests to frobnica.te start failing doesn’t mean retries should also be disabled to storali.ze. This dichotomy perturbed me quite a bit, actually, and so I put off trying to solve the whole thing for a few months. But after letting my mind rest, looking at the exact same problem revealed an obvious solution: scopes.¹

I believe the most likely use of scopes is based on hostname. So that’s the easiest constructor to start a builder: retry::for_host(host). But it’s actually a sealed trait, to allow continued exploration. Maybe a closure is needed by some people? We’ll find out!

Cloning requests

In many languages, this isn’t even a consideration. You can just share a pointer, and wave hands mostly reuse it.

In Rust, with ownership, we need to keep a clone of the request for a potential retry.

At the hyper level, we don’t know enough about the request body to automatically clone it. At reqwest’s level, we do know enough: we can absolutely clone when it’s a simple non-streaming body. (Even possibly more, discussed below.)

Classifiers

You might have a very custom combination of routes and responses. Or an API where every idempotent request has standard status codes. Or you might need a deeper understanding of specific connection or protocol errors.

So to start, there’s a simpler classify_fn option, where you provide a closure to determine whether to retry or not.

.classify_fn(|req_rep| {
    match (req_rep.method(), req_rep.status()) {
        (&GET, Some(SERVICE_UNAVAILABLE)) => req_rep.retryable(),
        // or check req_rep.error()?
        _ => req_rep.success(),
    }
})

Another possible avenue to explore is the composition of classifiers. For example, what if you combined something like protocol_nacks.or(idempotent.and(service_unavailable))? Is this a good idea? Would people want this? If you want to use something like that, file an issue or PR.

Probable enhancements

Besides the above, there’s a few other things we could add to reqwest::retry:

Backoffs: They’re pretty easy to add, though the benefit they bring is only really noticeable in specific situations. This is why they aren’t included yet or would not even be enabled by default.
Replayable Body: Retrying needs to clone the request, and a streaming body is not freely able to be cloned and restarted. But it’s desirable by users, so we could make it an easy configurable option, such as .max_replay_body_bytes(). We could likely learn a bit from how Linkerd implemented it.
Per-Request Retries: More and more, users ask for configuration for a specific request that differs from the Client. We’ve been working on general per-request config system.

Exploring and upstreaming

We’re exploring these ideas in reqwest, with the goal of upstreaming the more generic ideas to tower or tower-http.

Maybe that’s a builder:

tower::retry::builder()
    .clone_reqs(|req| Some(req.clone()))
    .matcher(|req, res| res.status() == RETRY_AFTER)
    .max_extra_load(0.3)
    .build();

Or maybe something closer to middleware, a trait to compose pieces, with and() or .or():

tower_http::retry::clone_replayable(8192)
    .and(idempotent)
    .and(server_error)
    .or(protocol_nack)
    .and(budget(0.3))

Maybe a combination! Who knows?

Check it out

There’s a pull request implementing some of these ideas, and your feedback and usage will help us to keep refining it.

The two-pronged goal is to allow you to easily get retries in reqwest, and to also be able to get that easier configuration in any tower stack.

Give a whirl!

Scoping budgets wouldn’t be something to care about if the client were already built for a specific endpoint. That’s a common way to build a stack of tower layers. But reqwest is open-ended, you can send requests to anywhere with an address. So, scopes felt required. ↩

Body::poll_progress

Tue, 22 Apr 2025 10:01:00 +0000

This describes a proposal for a cancelation problem with hyper’s request and response bodies. hyper is an HTTP library for the Rust language.

Background: what is the `Body` trait?

The Body trait used by hyper is meant to represent a potentially streaming (asynchronous) body of a request or response. It sorta looks like Stream or AsyncIterator.

The biggest reason for a different trait was because we needed a stable trait for hyper v1, and Stream/AsyncIterator were not going to be (and still aren’t) stable. With it being a separate trait, though, we also added some HTTP-body-specific methods to it.

Its similarity to Stream/AsyncIterator means we run into a problem with forwarding into a sink. But that we own the trait also lends to a decent solution, which this post outlines.

First though, piping woes.

Problem: backpressured cancelation

Piping stream-like things into sink-like destinations feels natural, and looks quite simple. Elegant even. They can be piped together, and backpressure occurs naturally. However, they lack a mechanism to completely propagate cancelation. More specifically, cancelation while backpressure is currently applied.

Consider an example:

while let Some(frame) = body.frame().await? {
    // what if body (^) cancels while we wait here?
    dst.send(frame).await?;
}

This simple loop is piping a Body into some sort of sink. The way most streams and channels work, this mostly propagates backpressure. As long as the dst is not able to send a frame, we don’t poll for another one, and whatever is the source of the body will back up.

However, it has a flaw. Those familiar with writing proxies may notice it immediately. The destination might not have space, and so we’ll wait for it to get more. But the body could give up during that time. Since the task is only waiting on when the dst is ready, it could wait a significant amount of time and never notice that the body (source) has canceled.

Why not just timeouts?

One initial question was “why not just timeouts”. Like, why not just add body_write_timeout() or something to hyper’s connection builders. Fair. That would work in some cases, for sure.

The thing is, sometimes timeouts are inappropriate.

For example, a transparent proxy may not want to force timeouts where they didn’t exist before. They’re willing to wait nearly forever. They just want to cancel the forwarding if the sender gives up.

Additionally, a chain of potential timeouts means that cancelation propagation can be delayed longer and longer, as each hop in the chain has to wait its own timeout.

More generally, yet not

This issue can also exist for Stream/AsyncIterator. A poll_progress idea was outlined by withoutboats, though it was solving a different problem.

I do think there’s some crossover that may affect each other. But it’s also worth considering separately.

I’m not talking about a concept that some for await syntax could magically care for. I don’t want that. Like at all. In the above example, it’d be impossible for a compiler to determine which action I want to take. Maybe the source ended cleanly. I don’t want that to cancel sending the last item.

Also, and this is a biggie: Body will need to solve this problem quicker.

Solution: `Body::poll_progress()`

We propose to add a method to the Body trait:

pub trait Body {
    // existing methods ...
    
    fn poll_progress(
        self: Pin<&mut Self>,
        cx: &mut Context<'_>,
    ) -> Result<(), Self::Error> {
        Ok(())
    }
}

The purpose of the method is allow for it to make any “progress” that might result in cancelation, while not producing another frame. The details:

It’s a poll_* function, so it allows a body to poll something else async, like a timeout, or perhaps some shutdown channel.
Returning Ready(Ok(())) immediately is fine, it simply means there’s nothing that the body would do before another frame could be made available.
However, returning an Err would indicate that the body is canceled, and a piping task can interpret that to shutdown the forwarding operation.

We could provide some utilities to aid use of this function, such as http_body_util::BodyExt::progress().

Consider an example, something that could be even be used in hyper internals, but could also occur elsewhere:

loop {
    // produce a frame
    let frame = select! {
        Some(frame) = body.frame() => frame,
        dst.hup() => {
            // unexpected EOF
            return;
        }
    };

    // send it
    select! {
        dst.write(frame) => {
            // continue
        },
        Err(e) = body.progress() => {
            dst.abort();
            return;
        }
    }
}

Alternatives: not `poll_closed`

An alternative we considered was making this method be a poll_closed. It had the welcome effect of meaning we could await body.closed(), and that is fairly self-documenting. However, it ran into several problems.

Being an addition to an existing trait, it needs a default implementation. But a default for poll_closed would likely end up being confusing. If it returns closed by default, then existing impl Bodys will suddenly start aborting early. If it returns not-closed, then any naive task that might await body.closed() would hang forever.

The return value also felt confusing however we put it. Does closed return a Result? Both Ok and Err would still mean the body had closed.

Request for Comments

I wrote this up because it felt like a big enough change to a fundamental mechanism in the ecosystem that it’d could benefit from more eyes and comments. It kinda-sorta looks like AsyncIterator::poll_progress, but it’s also not. Still, are there things we should prepare for? Would wg-async have thoughts? Others who are using hyper deeply?

Could it be simpler? Is it flexible enough?¹

Comments on the pull request are most welcome!²

Calls for completely changing hyper’s design to not need such a function aren’t helpful. hyper is stable at v1. ↩
Thanks to Steven Fackler and Oliver Gould for helping iterate on this design. ↩

A More Modular reqwest

Tue, 04 Mar 2025 13:05:00 +0000

reqwest came out 8 years ago. It was meant to be a higher-level HTTP client, with batteries and opinions included. And it continues to fulfill that role very well.

To do so, reqwest combined or implemented a lot of features that weren’t easily accessible elsewhere in Rust. Things like redirect handling, connection proxies, and compression.

People love it. It’s the most popular HTTP client used in Rust code. As of today, it gets downloaded 10 million times per month from crates.io (which doesn’t include any internal mirrors). It helps engineers all around the world.

People also wish they could better customize it. They want to add in custom logic somewhere. This makes sense. And the ecosystem has gotten even better at reuse. It’s time for reqwest to help be more reusable.

First, what is the goal?

Easy and modular

Top priority is that reqwest remain easy to use. It should be the best default for most people. If you don’t know what customizations you want, you shouldn’t have to care.

It should also be possible to customize. To tinker. To grab just a single piece. To re-order the pieces. To add in completely new pieces.

Some concrete work

So, how do we get there? Well, here’s a start.

The connection pool can be changed from a single Pool type into a series of Services that allow better composition, such as a racing cache pool, singleton pool for HTTP/2, an Either pool to combine them. This allows people to use less complicated pools than one that includes connection racing. It makes way for using tower’s load balancing services directly within reqwest, even.

The HTTP Proxy internals can be refactored to be another Service which fits into the series of “connector” services reqwest uses. These include loading proxy data from environment variables, and providing HTTP/HTTPS/SOCKS proxies and tunneling. Other applications that only use reqwest for proxying can depend on only those utilities.

reqwest supports some basic redirect policies, and allows for custom ones. Eventually, tower-http added middleware to do that too. We can now use redirects from tower-http. It’d be better to update reqwest to depend on that middleware internally, allowing reqwest’s test suite to find any pieces lacking in the port.

We can add easy request retries. reqwest doesn’t really have retries, other than a very specific case: special HTTP/2 error cases. And conceptually, retries aren’t that different from redirects. Retrying requests is pretty common thing to ask for, and it’s pretty easy to do them wrong. There’s middleware in tower::retry, but it’s not easy. We could make it as easy as calling .retries(reqwest::retry::idempotent()) on a client builder. Or a few other common policies.

We can use decompression from tower-http. reqwest has automatic response body decompression. Eventually, it was added as middleware in tower-http. Instead of having the feature essentially duplicated, we can just depend on the modular version, and the test suite will keep the quality up. Doing it this way improves the middleware for everyone. For instance, when one user swapped from reqwest’s decompression to using tower-http, performance got worse. Eventually, a reused allocation was backported, bringing it back in line.

Better default TLS

Initially, reqwest made use of native-tls for HTTPS support. Eventually, we added optional support for rustls. But we’ve kept native-tls as the default. The reason was because it worked for more users, and more sites. However, I think it’s time to change the defaults.

At this point, reqwest’s defaults actually target the worst common denominator. It not longer feels right to make people opt-in to the better choice. As I said at the beginning, reqwest bundles opinions too.

Our opinion is that reqwest should have the best defaults for the most people. Those that need something different can opt-in to that.¹

So, reqwest will be changing it’s default-tls feature to rely on rustls. Likely with aws-lc-rs as the crypto backend. Optional support for native-tls will remain, for now.

The default-tls feature theoretically was designed to allow for such a change, since nothing should be exposed with it that isn’t supported by rustls. But they do have different build requirements, so it won’t be 100% smooth.

But it will be better for most people.

Getting to work

That’s the work. Keeping reqwest easy. Allowing more customization. Even letting people use fewer parts of it. While having strong opinions. Making it better for most people.

Want to be part of it? Help contribute code or documentation. Here’s the issue plan.

You can also sponsor the work. Roadmaps and priorities like these come about from conversations with companies that keep retainers.

All of it helps! To making HTTP requests better!

This could also be applied to other features too. I think there’s an argument to be made for switching the default DNS resolver to use hickory-dns instead getaddrinfo. ↩

2024 in review

Wed, 15 Jan 2025 12:05:00 +0000

I want to go through some highlights of the year. Thanks for coming along for the ride!¹

A year of being independent

This was my first entire year of my being an independent open source maintainer. I’m very happy with how it turned out!

I highly appreciate all who sponsor some amount each month, thank you all! It adds up. <3

I’ve found setting up advisor retainers to be very effective: I get to help engineers with private code, the higher context helps prioritize features or fixes that are needed, and the maintenance work that people don’t always want to think about is funded.

I’m hopeful about this next year, and I’ll need your help to continue.

hyper

One of the things I’m most excited about is that in the beginning of the year, we added 3 new collaborators to hyper: @Noah-Kennedy, @dswij, and @tottoto.

More collaborators improve the health of the project, reduces review load, increase diversity of thought, etc. Some ask how they can be become one, so I spent some time writing what a collaborator is in hyper’s contrib docs.

There were 12 releases of hyper, not to mention explorations in hyper-util, and maintenance in dependencies like http, h2, h3, and httparse.

A few of the new features we got this year include support for sending and receiving HTTP/1.1 Trailers, improved connection selection with try_send_request, and a server connection helper with GracefulShutdown.

Wrapping up the year, I updated the roadmap, highlighting our focus after stabilizing v1. One area that’s already started to see improvements are the hyper docs and guides.

reqwest

A lot of cool work was done in reqwest.

v0.12 was the big one, upgrading to hyper v1.0, back in March. But it didn’t end there, reqwest kept getting better and better throughout the year.

An improved timeout option was added, read_timeout, which better matches how many want to detect a broken connection, instead of an entire request deadline.

HTTP/3 support returned, unstable but better, with IPv6 and content-length fixes.

Integration with rustls gained support for the new aws-lc backend, and reqwest gained several cargo feature options to combine it with the various sources of certificate roots.

To go along with that, reqwest also gained support for certificate revocation lists. I’m glad to hear that building rustls with aws-lc is getting easier, I hope to make that the default in reqwest this year.

Near the end of the year, reqwest gained support to inject connector layers, which allow users to customize connection establishment much more than hard coded options.

That last one is actually a taste of what is coming to reqwest. I’ve often mentioned a desire for making reqwest work even more as a stack of middleware, and I have much more of a plan now. Details in another blog post soon!

Security

Maintaining these libraries means keeping them secure. Over the past year, I dealt with various security reports, which are always quite expensive. They require dropping what I am doing, digging in with research, and testing to fully understand the report quickly. Some were determined not an issue, others were bugs that we fixed but not worth yelling about.

One of the more consuming reports was HTTP/2 Continuation Flood. It wasn’t limited to hyper, instead affecting many implementations. hyper was somewhat affected, and we coordinated and got updates out on the coordinated disclosure date. Thanks to the sponsors I was able to do this work.

Podcast

I got to join Allen Wyma on the Rustacean Station again, and we talked about hyper reaching v1, access to advising from a maintainer, middleware, retry storms, and more.

bye curl

Back in 2020, we added support for hyper to be an HTTP backed in curl. It languished at “nearly complete”, and in November I put out a call for a champion to keep it alive.

It seems we as an industry didn’t really want it, so it has now been removed as an option in curl.

But that’s ok. Both projects were improved by the process. I appreciated being able to work with Daniel; he is an excellent maintainer that truly cares about quality.

My focus is more on the audience that wants to write software in Rust. More and more services and tools are popping up everyday that must be fast and secure as they talk to other computers. Teams pick Rust and hyper and its ecosystem to deliver their experiences. I’ll continue to empower them.

Questions to ponder in 2025

I wanted to share a few questions that I’m pondering as I head into 2025. It feels better than pretending to know more definite plans.

How can I encourage more collaborators? It’s normal for some to change focus, so we always need more.
What would a generic configuration builder look like? I imagine a zero-cost, injectable HList of properties to configure possibly unknown stacks of things. But how?
Can std::error ever be good? E: Error, Box<dyn Error>, and anyhow have too many sharp edges exposed.

See some previous yearly reviews: 2023, 2022, 2021. ↩

hyper Roadmap 2025

Tue, 10 Dec 2024 11:48:00 +0000

After a year since hyper 1.0, we’re updating the ROADMAP. hyper is an HTTP library written in Rust, used by many in production.

The purpose of the ROADMAP is to highlight what is highest priority in order to continue orienting hyper towards its VISION.

It’s based on interviewing our users to see make sure what the most important things are. Writing it down is a way of letting users know what’s coming next.

Open source is guiding

Open source is not a company, and so people can work on and contribute whatever they want.

But we can guide contributions, by outlining what is most important for the project. Thus, a secondary purpose of the roadmap is to be a guide for those interested in contributing, but not sure what to do specifically.

Focus Areas

The roadmap breaks down four focus areas:

Documentation: Invest in docs, guides, and examples to ensure users can learn how to use hyper.
hyper-util: Provide new helpers for use with hyper, and stabilize and graduate some into hyper itself.
HTTP/3: Harden the h3 crate (reqwest uses this already!), and use it to power conn::http3 in hyper.
Observability: Improve visibility into hyper’s operations through stable support for events, tracing, and metrics.

A living document

The ROADMAP is a living document. As we make progress, or as priorities change, the document can too. Proposals for additions and edits are always welcome.

Contribute

If you’ve wanted to get started contributing to a Rust open source project, help us build hyper! Pick one of the focus areas that interest you, and reach out in an issue or Discord. We’d be thrilled to mentor you.

hyper in curl Needs a Champion

Tue, 19 Nov 2024 09:48:00 +0000

tl;dr - hyper in curl is nearly complete, but it needs a champion. Without a partner actively engaged that wants to enable and ship, it’s now on the path for being deprecated and removed.

It needs a champion, a backing vendor or distro. Will that be you?

UPDATE 2024-12-21: curl has dropped hyper.

Why would you put a hyper in a curl?

Why would you? Memory safety.

Company after company, product after product, report after report. Memory un-safety. Causes. Serious. Issues.

curl is everywhere. Billions of installations. It’s likely all humans accessing the internet use curl.

A self-analysis of curl finds that half of curl’s vulnerabilities are C mistakes. Memory safety.

As I said when when we told the world about it:

Considering how much curl is used, this was an opportunity to make the internet safer.

hyper is the most mature HTTP library written in Rust. By making hyper a possible HTTP backend for curl, the code used for the most ubiquituous protocol could be made safer. Certainly true for HTTP/1, even more so with the much bigger (code-wise) HTTP/2 and HTTP/3.

So, why do this? Oh, right. Ahem. Memory safety.

Most of the work is done

Let me back up a little.

In 2020, we started exploring the idea. I designed and built a C API for hyper. Daniel refactored curl to allow for HTTP backends, and integrated hyper.

We got it nearly complete. Adventurous tinkerers were able to build and use it on their personal machines. Over 95% of curl’s large test suite was passing.

I gave a talk for curl up 2022 about the progress.¹

We’re ready to finish, technically.

Over the finish line

Funding for an engineer to complete the work is available.

But the upkeep of the feature isn’t free, in both the curl and hyper repositories. Because of that, and without a commited organization wanting to ship it, it’s planned to be removed at the start of 2025.

So, what exactly could change that? What is needed?

Champion required

A champion, if you want it.²

A backing vendor or distro that wants to enable and actively use the backend. A launch partner. Many people know what it’s like to work on a large new feature, ask people to try it out, and everyone is too busy, assuming someone else will. A launch partner actively tests it and provides feedback.

There’s more incentive to partner than ever, as we see companies successfully make code safer. And this project is so close, adopting now can have a large impact compared to the remaining effort.

Reach out to me if you want this to happen. Sooner rather than later. Let’s make the internet safer!

UPDATE 2024-12-21: curl has dropped hyper.

Recently, a few more things have been improved on hyper’s side. For instance, @nnethercote and @jsha significantly improved the C docs, and @hjr3 added HTTP/1.1 trailers support that curl needed. ↩
People always end up doing exactly what they want. There’s a loud rewrite-it-in-Rust sub-community. Here’s an opportunity. Actions show what people actually want. ↩

hyper HTTP/2 Continuation Flood

Wed, 03 Apr 2024 14:11:00 +0000

Patches are available for h2, v0.4.4 and v0.3.26, to harden against a newly disclosed HTTP/2 attack vector, mostly for servers. If you need help, reach out for support.

If you’re curious about more, read on.

What is the attack?

In HTTP/2, there are a bunch of frame types. Some of those frames are related to sending headers (or fields). Since there is a maximum frame size, in order to send more headers than fit, a peer can send CONTINUATION frames that indicate more headers for the message.

That mechanism is the source of a new attack, trying to send an infinite number of CONTINUATION frames to starve or kill a server. This was a coordinated release, with various implementations involved. VU#421664 has details about the others.

hyper’s `h2` is somewhat affected

The HTTP/2 specification doesn’t state a default value for SETTING_MAX_HEADER_LIST_SIZE. You should set that to something reasonable for your use case. But even if you don’t, h2 picks an emergency default, which is fairly high so to as to not reject requests for users who need big requests.

Once a stream of headers and continuation frames reach the limit, either configured or the large default, memory growth will cap there. h2 will keep reading and decoding, but discarding any new headers. Really, it’s trying to keep the HPACK table in sync, and will send back a stream error once finished of “too big”.

The problem is that h2 won’t stop reading as long each CONTINUATION frame says there are more coming. So the memory stays the same, but it will use CPU to read and discard. And since the headers are never “complete”, the application doesn’t get to know about the processing of frames.

However, if deployed on Tokio (or some other runtime with a similar feature), the task budget will prevent it from locking out other tasks. Other requests can be served, and responded to. But it will make it slower.

A degradation of service.¹

What we did

Besides testing and determining the above, we also worked on a fix.²

We realized that while a lower SETTING_MAX_HEADER_LIST does help, it’s not the only thing that needs to be checked. The reason for that is because a CONTINUATION frame could be sent with the smallest header value, and thus take longer to reach that maximum. That will still consume resources processing frames.

We figured that a maximum needed to be set on the allowed CONTINUATION frames. But a naïve hard-coded number has its own problems.

The maximum frame size isn’t usually changed, which means you’re stuck with 16kb frames. And if you do want to allow message headers bigger than that, then you need to allow CONTINUATION frames. Since both frame size and header list size are configurable, a hard-coded limit wouldn’t work.

We settled on calculating how many CONTINUATION frames would be needed for a legitimate message to send up to the max header list size. We also add a little bit of padding to that number, to account for frames that may not be perfectly packed.

The patch has been tried out on some production servers, and found no false positives.

What you can do

If you don’t use HTTP/2 on a server, then don’t worry.
- If you do, but are not exposed to untrusted L7 traffic, also don’t worry.
If you don’t set the SETTING_MAX_HEADER_LIST_SIZE, you should consider doing that.
You should run cargo update -p h2 in your application to pull in the latest versions with fixes (v0.4.4 and v0.3.26).

If you need help, reach out for support. Or send thanks, so that months long security work like this can be done in the future.

For this reason, we didn’t create a CVE. I’m sure some well tell me I’m wrong. I’m already skeptical of CVEs for denial-of-service; it’s a big deal for some people, and not at all for others. I’m also very wary of alert fatigue. A degradation of service, where things are just slower, does not feel like a “wake-up-your-team” moment. (We did submit a RustSec advisory to nudge people to upgrade.) ↩
Noah Kennedy and I have worked on this on and off for a few months, while waiting for the VINCE deadline. Thanks Noah! ↩

Podcast: Rustacean Station, hyper 1.0, and independent maintainership

Fri, 29 Mar 2024 10:09:00 +0000

I was recently a guest on the Rustacean Station podcast. It was nice to catch up after a couple years since my last appearance on the show. We spoke about hyper, how and why it became v1.0, becoming an independent maintainer, future work, and more.

I thought I’d grab a few fun and interesting quotes, with their timestamps:

Sponsoring is not the Mafia (11:48)

The lowest engagement is sponsorship where it’s like, what am I getting from your sponsorship? If you turn it around and think about, well, a lot of times open source is like this common good that everyone kind of benefits from and I don’t want to pay for it because someone else is going to pay for it, right? It’s kind of like this Russian roulette of like, well, if no one pays for it, then eventually that thing is going to go away.

So sometimes when I talk to the companies, I just ask them, hey, how much are you using this stuff? You’re using it a lot. Cool. How annoying or how much work would it be to have to maintain it yourself if I were to just disappear? Oh, then we’d have to put a whole engineer on that or something like that. Okay. So if you think of it as a business risk mitigation to sponsor, then I can keep doing it. And then it becomes like, okay, well, we would pay a full-time engineer to do this all year. We can pay a way smaller amount to just have the sanity check that this is not going away.

And so what do they get? They get a business risk mitigation.

You’ll have to excuse me, but I’ve been watching a lot of Sopranos. And so when you said, how bad would it be if this thing happened to go away? It sounded a little bit like a tactic they might use to secure the contract.

(laughs) Yeah, it’s not like an insurance racket.

Access to the maintainer / advising (12:50)

There’s more things that I can offer and I do offer. Another one is people want to be able to ask, hey, we’re using your stuff and we want to know are we using it correctly or, hey, we’ve been trying to use it and we have this problem and we don’t understand why. Could you take a look?

If it’s a private project, people aren’t going to post their source code into a public issue. I’m not going to have the time to go and take a look unless it’s like, hey, let’s get a retainer and I can now sign an NDA. I’m not going to steal their code, all that legal stuff.

It’s like getting an advisor, a reviewer, and then I can then take that knowledge and go back and be like, okay, so I now have knowledge that like company so-and-so is using this in a way that I didn’t expect. How can I make it better for them? So like, both sides benefit, but it’s only possible if I set up contracts and everyone’s legally happy.

Breaking people, not kneecaps (30:55)

If something would be a breaking change, then we label it with a breaking change. And it’s not something we can do right now. We can close it or postpone it or something. And then, you know, maybe three years from now, go and take a look at all the issues that are labeled that and say like, yeah, you know what, this would be worth breaking for. But at the same time, I’d love to not have to break people, even though, you know, the promise is only for three years. I’d love to not have to.

Did you just say break people? We did go back to the mafia reference, right? But no relation, right?

Exactly. Yeah. No breaking kneecaps.

Client middleware (32:12)

The other major thing is improving middleware. There’s all this stuff in tower, and it’s great. Like there’s Axum. You can use it to make really powerful servers. But the point of this middleware was actually that you could use it both ways. You could use it for servers, but you could also use it for clients. And that doesn’t work as well.

The most popular thing to use for clients is reqwest. And it doesn’t fit in to tower middleware. It has a bunch of extra stuff like redirects and some other things that it does. Various kinds of timeouts, which would be great middleware, but it’s currently all just wrapped up in reqwest.

And then also, if you even just look at the middleware in the first place, it’s like opaque. You have to understand how to do service discovery to then use load balancing. You have to understand how to build a retry policy before you can fit in retries and understand what is a retry budget.

It’s complicated. And it’d be nice if people could just get, hey, this is better for 95% of use cases. And if you really want to go and tweak your retry policy, sure you can. There’s the escape hatch over there.

But that’s my other thing to work on. Make reqwest and like Tower’s client middleware merge together a whole lot nicer so you can have much better stacked clients.

Retry storms and budgets (33:50)

Since there wasn’t a really easy plug-in, then people implement retries themselves. And that includes making themselves vulnerable to retry storms and just not doing it in a safe way. And the thing is that we have this middleware and things that would protect you from that, but they’re just complicated enough that people are like, ah, I can just retry in a loop.

But then you smash the server once things start falling apart. And it’d be so much better if you just add in a retry layer. It’s going to do things wisely. And maybe you say, you know what, on this URL, never retry it, but otherwise do the default thing.

It’d be so much nicer if you didn’t have to understand how bad retries can go.

If you just do a simple counter, then you might have counters at various layers. And maybe your counters aren’t shared. And now even though you’re trying to be nice to the server, you’re still retrying on thousands of times, whereas a shared budget or something would notice after a few retries, hey, the server’s overloaded. Let’s stop pounding it. Because as long as we keep trying, it’s never going to get back up. That’s one of the problems with retry storms.

The linkerd blog had a good post on how you can fix that using Tower middleware. But I’d prefer it if people didn’t have to read that to use it.

And plenty of other topics

Go have a listen, I hope you find it informative!

reqwest v0.12

Wed, 20 Mar 2024 08:13:00 +0000

Today marks the v0.12 release of reqwest, a higher-level, batteries-included HTTP client for the Rust language.

What’s new

The headline feature of reqwest v0.12 is the upgrade to hyper v1. reqwest does a lot of custom work to add features to hyper internally, but doesn’t need to expose too much of it publicly. This was what took the majority of the work to upgrade. Still, as outlined in the the hyper v1 announcement, the parts that are public have stabilized. The most obvious improvement to most users of reqwest will be the stable integration with http v1 types. This should remove the last blocker for a lot of people to finish upgrading to hyper v1.

With the breaking change, we made a few more improvements. Several optional features have been added which previously were required, such as http2 and charset. This allows disabling them and making the compilation time and size lower if not needed. We also converted all the implicit optional dependency features to dep: syntax.

We had to disable the HTTP/3 feature, but it was experimental anyways. It’s a goal to bring it back as soon as possible. We might also be able to add some other QUIC backends, such as s2n-quic.

See the release for more details.

Coming next

As mentioned a few times before on this blog, my focus next is to make a bunch of the powerful features in reqwest available as middleware. reqwest will remain easy to use. But it will be possible for people configure their own client stacks, without needed to fork or copy code. It will also mean even if you use the reqwest default easy stack, you can more easily integrate it with other tower middleware, such as retries or load balancing.

Thank you!

reqwest usage keeps on growing, and it’s both humbling and exciting to watch! For example, rustup uses reqwest as its default download backend, helping all Rust developers to keep their compiler up to date. Oxide uses reqwest in their Progenitor library that powers their OpenAPI clients.

Thank you for all your contributions, both in reporting issues, and in helping fix them. <3

If your company uses reqwest, consider becoming a sponsor. Additionally, for private advice, reviews, security help, and access to the maintainer, get in touch for support.

2023 in review

Tue, 16 Jan 2024 08:29:00 +0000

A year of change, and of stability. Let me briefly reminisce and highlight what happened in 2023.

Independent

I became an independent maintainer in June 2023. This was a somewhat scary decision, but it did exactly what I hoped it would: my work-life balance feels fantastic. I’m also glad to be able to do similar to what I did at AWS—meeting with and advising teams with serious deployments—but with a wider variety of use cases. (Send me an email if that interests you.)

hyper

First, some stats about hyper over the past year. There were 90 unique authors this year, which is 40% growth from 2022! We had two more people sign up to be triagers.

v1 🚀

We released hyper v1 in November. What a ride. It brought changes, moving the less stable side out into hyper-util. But it also signaled a core that won’t be changing any time soon. Stability.

The ecosystem caught up quickly. There were releases for tower-http, headers, and Axum ready to go just a couple weeks after. It enabled some other cool things, like the general availability of the AWS SDK for Rust.

We closed out the year with v1.1, bringing back core pieces needed to make graceful shutdown in servers easier again.

Security

I reviewed at least 10 security reports, perhaps a couple more I didn’t keep track of. This includes the wider HTTP/2 rapid reset attack that hyper wasn’t affected by. The amount of time I spend on security reports keeps on increasing. That makes sense, we announced stability of v1, which surely made more people take a look. It also is a sign of more production deployments, with companies wanting to audit their dependencies.

HTTP/3

Another priority this year was to make progress on hyper’s HTTP/3 support, currently under development in the h3 crate. We released some initial 0.0.1 releases (and a few more subsequent ones), specifically to make it easier for people to use. reqwest gained unstable HTTP/3 support, using h3, and some brave users have enabled it, found it working well, and are now asking if we can make it stable. A couple of other fine folks worked to make the h3-webtransport crate, building on top of h3.

Future Focus

Doing a bit forward looking, what’s the plan for 2024? Well, of course it could change at any moment, but these seem to be the things people most ask me for, and most need.

HTTP/3 in hyper

I hope to make significant progress towards getting HTTP/3 support directly in hyper. I’ll work on a proper proposal, but here’s some unordered steps in that direction. Stabilizing the feature in reqwest. Set up an auto-updating h3 server for interop testing. Dig away at the compliance report, both by labeling more of the parts already working, and adding any missing parts. Propose how to expose it in hyper, which will be tricky so as to not tie hyper to any specific TLS library. And then get it actually added, likely as a hyper_unstable_h3 feature to start.

Level up Client middleware

There’s a lot of great middleware now. But it can still feel like it requires an expert to use it properly. I’ve been hinting at for a while that I’d like to do for clients what Axum has done for servers. I hope to do that with reqwest.

reqwest does a lot of useful things, but if someone wants to customizing it beyond the options that reqwest exposes, they have to reimplement a lot (or live with a fork). I want to make most of reqwest’s features tower middleware. reqwest will still have a standard “recommended” client. But it should be easier to build up your own custom stack.

And while we’re at it, I hope to make some of the most important and yet most difficult middleware much easier to sprinkle in: retries, limits, and load balancing.

On-going maintenance

I also must carve out explicit time for maintenance work. There’s bugs that need fixing. Reviewing and triage takes a lot of my energy. I want to improve the docs and guides. The amount of security reports received is growing, and those take time to investigate and respond or patch and disclose, depending on their validity and severity. This also includes time with my sponsors, which helps identify maintenance work priorities.

Want to join us?

hyper v1

Wed, 15 Nov 2023 16:44:00 +0000

I’m excited to announce v1.0 of hyper, a protective and efficient HTTP library written in the Rust programming language. hyper provides asynchronous HTTP/1 and HTTP/2 server and client APIs, allowing you to bring your own IO and runtime.

It’s been exciting and humbling to watch users build awesome things. Cloudflare uses hyper within Oxy, its next generation proxy framework to handle traffic at considerable scale. After Discord’s 5x improvement to @mention response times a few years ago, they have moved most of their critical systems to depend on Rust and hyper. curl has a currently experimental HTTP backend built on hyper with the goal of making the Internet safer.

Marc Brooker, a Distinguished Engineer at AWS, commented:

When building our new container-loading data plane for AWS Lambda, we expected to need a custom binary protocol. In production, we’ve found the overhead of hyper to be SO low that we are excited for it to continue powering our services.

Johan Andersson, CTO at Embark, said:

We have been using and relying on hyper for the last 5 years for our gRPC and REST services, tools, libraries, and embedded in our next game built in Rust. It has been rock solid across all of our usages, and it really is a foundational library for the Rust ecosystem. Congrats on 1.0!

The best way to get started is to check out the guide.

Stability here we come

Over the past 9 years, hyper has grown from a web developer’s side project into a solid library powering huge network applications. It’s time to grow up. After bringing async/await support in v0.14, we focused on providing a set of basic APIs that would keep hyper safe, fast, and flexible. This meant removing some of the more opinionated “higher level” pieces. Those belong elsewhere, like hyper-util, reqwest, Axum.

This release signals some stability. Major versions, like 1.0, are stable for at least 3 years.¹ We also keep a MSRV that is at least 6 months old.² We’ll add new features, and we still have a couple places to experiment: in the hyper-util crate, and hyper_unstable compiler flags.

Starting in v0.14.25, we added a backports feature which brings the new core APIs to you immediately. Combine that with the deprecated feature, and you’ll be guided to making your existing code ready for the upgrade to 1.0. Be sure to check out the upgrade guide!

The most immediate next steps are to update the other core parts of the ecosystem that depend on hyper: reqwest, Axum, Tonic. But after that, there’s plenty more to do. You’re welcome to come join us!

HTTP/3

I would like this to be my next focus. We’ve been building up the h3 crate, and reqwest has unstable support now. I’d like to stabilize the feature in reqwest, and explore how we can make it available in hyper directly. Then we can have easy HTTP/3 servers, too!

The trickiest question is making it available without being tied to a TLS/QUIC library. Then, users could choose to use quinn, or s2n-quic, or msquic, or any other.

Stabilize in curl

The biggest parts of making hyper work in curl are done. Someone with experience in Rust and C could make a huge dent in Internet safety helping to get it over the finish line.

Middleware

There’s some excellent middleware available already in tower and tower-http. But several of the important ones are just a little (or a lottle) too difficult to add to a stack. I’d also love for there to be some recommended stacks for servers and clients, that bundle together the right middleware that most people would need. To that end, I’ve mentioned before breaking open reqwest such that all of its features are middleware you can customize.

Tracing and Metrics

It’s possible to currently get a decent set of logs using tower_http::trace. It’d be better if you could get more fine-grained traces and metrics. Probably with some stabilized integration with tracing directly in hyper. Maybe some sort of hyper-metrics, similar to tokio-metrics.

`io_uring`

Part of the reason we made hyper have its own IO traits was to be able to adapt them for completion-based IO. I believe having decent support and benchmarks could be had pretty soon, by a motivated individual.

Thanks

A huge thank you to all our amazing contributors. You’ve made this project the success it is, and helped move hyper along the journey to 1.0. I’d like to follow up with a separate post specifically thanking you all.

Thanks to the companies who have sponsored the creation of hyper: AWS, Buoyant, Mozilla, Rust Foundation, Fly.io, Embark and others.

Your company could also become a sponsor or get support!

Besides some correctness mistake that must be fixed ASAP. ↩
We realize that some users just cannot upgrade that fast, and we care about them. ↩

hyper HTTP/2 Rapid Reset Attack: Unaffected

Tue, 10 Oct 2023 15:07:00 +0000

Today, the world has been made aware of a potential vulnerability affecting most HTTP/2 implementations, sending a rapid amount of streams and resets.

If you use hyper, even just it’s h2 dependency, you are safe. hyper is not affected. Especially if you have h2 v0.3.18 or newer. We manually verified that an example hyper server responds correctly. Big thanks to @Noah-Kennedy for all the help.

If you want to read more, checkout CVE-2023-44487, or these other breakdowns.

That’s it!

You’re still here. You want to know the “why”?

Well, for two main reasons.

We added in specific detection of this problem back in April. A related flaw was reported against hyper, with the added requirement of a consistently flooded network. We fixed that. It had a CVE and RUSTSEC advisory for it, so you should have upgraded, right?

But even without that fix, the damage that could be done was local. The bigger concern of this newly announced vulnerability seems to be when the receipt of the HEADERS frame triggers more work in the handlers that needs to then be canceled. The way hyper handles frames, it will cancel out the stream before ever making it available for handlers, so the cost is local. Without the fix, and only if the user can flood the network, then hyper could consume a lot of memory keeping track of all the suddenly reset streams. If they can’t flood the network, then no problem at all.

So if you’ve upgraded since April, you’re safe. By the way…

Handling security by dealing with reports, and working with coordinated disclosures like today are a significant part of maintaining hyper. If you appreciate that hyper is kept secure, consider sponsoring. Being able to have more support during security disclosures is something that you can setup with me privately.

Was async fn a mistake?

Thu, 28 Sep 2023 16:58:00 +0000

This stabilization PR for async fn in traits made me think: was async fn in Rust a mistake?

I mean, I dunno. Maybe it wasn’t. But play along for a moment.

By the way, I don’t mean that async/await in Rust itself is a mistake. That’s a Big Deal. It allows companies to deploy some serious stuff to production. And async and await syntax is a huge save. I don’t want to lose that. Writing manual futures and poll functions is megasad.

I’m specifically talking about the async fn sugar. What if we didn’t have it, and instead just returned impl Futures, and used async blocks inside the functions?¹

The current async fn is really nice, if you fit the expected usage. If none of the differences with impl Future ever cause you problems, then great! But I do run into them. Other people seem to also.

What’s so bad?

Some of these differences cause problems that don’t have decent solutions. (Do you know the differences?)² If you have to deal with one of them, suddenly you need to use different syntax.

And now, people need to understand both. And keep the subtle differences in their head when they read. Does that make things better? Or worse?

It’s the only place that has a magic return type. It makes lifetimes weird. With suggestions to reign them in. It leads to all sort of proposals about how to customize the return type. #[require_send], async(Send), Service::call(): Send, and I’m sure there’s others.³ I also am thinking about generators and streams, since they could also end up with magic return values.

So was it mistake? I think it may have been. Don’t worry, I don’t want to take it away from you, if you disagree!⁴

What if the alternative was nicer?

But I did wonder about this. What if we had the following features ready:

Repurpose bare trait syntax to mean impl Trait. It’s been enough editions, right?
Ability to forgo naming an associated type name.
Stealing the feature from Scala where functions can equal a single expression.

Then asynchronous functions could look like this:

fn call(&self, req: Request) -> Future<Response> = async {
    // ...
}

That’d be a nice improvement.

Yea, I know, it’s a little more writing. But I am in the optimize-for-reading camp. We read much more than we write. So if I have to write a few more characters at a function definition, but it makes the reading experience more understandable, that’s a massive win. ↩
I’ve been involved in async Rust since the beginning. I know how it used to be, I was part of the group making it better, and I pay close attention to all the new proposals. I still mean what I said: none of the solutions look nice. ↩
Return Type Notation (RTN) syntax is probably the least gross. But it raises a bunch of questions. Does it work for all functions? If not, why not? If so, do I check I::Iter or I::into_iter(). And also to consider: Rust’s strangeness budget! ↩
I could see an argument that it’s sort of like for, while, and loop. A more convenient syntax when it works, and you can use the others when you need more control. That argument breaks down when async fn is part of a trait definition. But anyways, I really just want the less-sugared way to be little nicer. ↩

I'm an independent open source maintainer

Thu, 27 Jul 2023 15:00:25 +0000

tl;dr - I’m independent, sponsor me!

I’m doing something new. I’m an independent open source maintainer! In the beginning of June, I left my position at AWS.¹

I’m still focused on Rust, async, and HTTP stuff. Projects like hyper, reqwest, h3, tower, and any other new ideas that come along. I just won’t be doing so as an employee.

So, then how do I get paid? Let me just clear up a couple ways I’m not. I’m not making separate licenses. I’m not charging for features.² I’m not selling prioritization on roadmaps. Rather, I plan to make maintenance work my primary focus.

Maintenance can feel like riding a squared unicycle while juggling water balloons. Some of those balloons are:³

Designing proposals, interviewing users, re-writing those proposals.
Coding, coding, coding.
Triaging a never-ending supply of issues.
Spelunking in ancient code paths to understand and fix weird bugs.
Following a proper security policy with responsible disclosure, collaborating privately, and preparing detailed reports.
Reviewing pull requests for quality and sticking to the vision, and hopefully teaching potential collaborators.
Writing articles and giving talks, as a form of marketing and teaching.
Pretending to be a project manager.

It’s a lot of work, so who would pay for all that?

Does your company depend on my work? Become a sponsor! Consider it a form of business risk mitigation. You can use GitHub Sponsors or Patreon. I can also work with an invoice system, for any requiring that.

I am also interested in some deeper relationships with companies that want more. What exactly those relationships will look like will evolve. It would likely be things that look like office hours, support or private advice. If you want to explore that with me, reach out at sean@seanmonstar.com.

I learned a lot from my 3 years at AWS. Many lessons, some anti-lessons. Overall, I’m very grateful for my time there. But I had been planning this change for a while. And it was quite refreshing taking off a few weeks before jumping back into it all. ↩
A win about being independent is that no single company is deciding what features should be added. ↩
This would be a good subject for another article. There’s a lot more to it, and it’d probably be surprising to people how many hats are needed to maintain popular open source libraries, besides “just being a programmer”. At least, if you want to do it well. ↩

Report on Surprise hyper CVE from 2023-04-11

Thu, 27 Apr 2023 19:49:36 +0000

Summary

A surprise CVE publicly filed for hyper on April 11, 2023 caused an emergency situation for several collaborators, and sent out dependabot warnings with no actionable advice. By day’s end, we identified a best-guess at what the cause of the low-severity vulnerability was. By the next morning, a fix was available.

That the issue should have been a CVE is uncertain.

The bigger concern is the way the CVE was filed bypassing the existing security policy. That is similar to finding a lighter in a school, and pulling the fire alarm. This COE discusses both why it may have happened, and how we can try to reduce future occurrences.

The impact

The RustSec¹ advisory explains the issue this way:

If an attacker is able to flood the network with pairs of HEADERS/RST_STREAM frames, such that the h2 application is not able to accept them faster than the bytes are received, the pending accept queue can grow in memory usage. Being able to do this consistently can result in excessive memory use, and eventually trigger Out Of Memory.

In reality, being able to consistently accomplish those conditions would be very difficult for an attack, and so the likelihood of this affecting anyone is minimal. Certainly low severity.

But the bigger impact was not this particular issue, but rather that a CVE caused a sudden panic for the maintainers and for users as dependabot alerted people with nothing that they could do.

The story

The original issue was filed on May 27, 2022. Trying to better understand, I asked some poorly worded follow-up questions. Another contributor filed a pull request trying to fix the underlying issue. Several collaborators reviewed that PR, but didn’t fully grasp what it was trying to fix. It then fell into the void.

On April 11, 2023, someone decided to file a public CVE for the described issue, without following the security policy. I commented on the issue that while the motivation for doing so was likely good-intentioned, it was the wrong way to go about it. GitHub imported the report, which started triggering dependabot warnings.² This surprised us, and at least four people dropped everything to handle the fire alarm.³

The first step was trying to determine a reproducible example. We didn’t notice at the time it was filed, but the original issue did not include full reproducible instructions. We tried to create some unit tests to mimic the behavior described, but couldn’t trigger the issue.

Eventually, we noticed that a modified test that stopped “accepting” requests from the connection, but still polled it, would cause the accept queue to grow. But hyper makes sure to have a task that is always accepting requests, unless you specifically ask it to stop. Thus, the modified test seemed like user error, but it was a just guess.

It just seemed too convuluted. Then we arrived at a much better guess.

We finally found a way to grow the accept queue even when continuously accepting, by creating a test to blast thousands of requests in a loop. Since h2s test suite uses in-memory IO streams, we are able to fill the read buffer to near infinity. That’s when we settled on our best guess: if someone can fill the socket’s read buffer faster than the server can pop requests, then the accept queue could grow unbounded. While there is a setting to limit concurrent requests, because these are immediately reset, the limit would never be checked.

After 14 hours, we had a fix written and reviewed. We determined that the issue was low severity, as the likelihood of being able to consistently attack was extremely low. And since we were adding a new limit, there was a possiblity of causing a new bug. So, better to not push something right before going to sleep.

The following morning we published the fix, as h2 v0.3.17. Surprising everyone who has rushed out new code, a new bug in it was indeed found. We then published v0.3.18.⁴

Five whys⁵

Why did someone file a CVE suddenly? We don’t know for sure, but we can guess.⁶ A related issue had been open for a year, not fixed, so perhaps the reporter thought this was the only way to move forward.
Why wasn’t the issue acted upon a year ago? When it was initially opened, the maintainers didn’t fully understand what the problem was. Follow-up questions were asked, but even our questions weren’t that clear. Eventually, we forgot about it.
Why was it forgotten? We didn’t have any recurring reason to check back and try to understand what the issue was. If it had been reported privately to the security address, it would have stayed high priority until it was solved or determined incorrect.
Why wasn’t the initial issue reported privately? Perhaps the original reporter didn’t know about the policy.

What we’re doing to prevent a next time

We can’t completely control someone randomly filing a new CVE and causing another fire drill. But there are other things we can improve at to reduce the likelihood of one.

Schedule routine triage. This could be a synchronous meeting, such as in a text channel, or an audio channel. Or maybe over Twitch. But it can also just be a thing that triagers agree to do asynchronously, with a brief routine report to make sure we actually do it.
- ⚠️ If you or your company uses hyper , this would be an especially useful way to help with maintenance. Have an engineer or two dedicate a few hours each month helping us triage.
Setup a bug report checklist. There is a triage guide for bug reports, which is a good thing. But that doesn’t mean everyone (me included!) always remembers all the steps. Checklists are famous in aviation and medicine for their effectiveness in saving lives. They can also help us make sure all issues are treated properly.
Update the issue templates to use forms instead. We do have an issue template in place, to try to get people to fill in more information initially. But it’s pretty easy to skip it. It’s possible using GitHub’s new forms instead of just a text template could guide people more often.

RustSec and the CVE database are different. RustSec was much more helpful, coordinating with us by waiting until the emergency panic was over, and then discussing the best way to describe the advisory. ↩
I updated the advisory on GitHub’s end to only indicate h2, not hyper. I also indicated my disappointment in GitHub’s amplifying of the alarm and making the day much more stressful. Their reply: “We do that sometimes XD”. Cool. ↩
Meanwhile, a reddit thread took off, watching the action, commenting, and mostly criticizing the actions of all involved. Thankfully, I didn’t read comments like “I don’t have any sympathy for the maintainers” until after the fix was completed. ↩
“At least this made you fix it, right?” No. This attitude is toxic. Doing it this way burns out everyone around who could fix it. There is a reporting process for a reason. It helps the most amount of people. Please use it. ↩
Not literally five questions, but an exercise to try to find the root cause, and to note any extra things that could be fixed along the way. ↩
Some people tried to infer bad motives, such as for clout or “another notch on a security researchers belt”. I see no reason to assume that with no evidence. ↩

This Month in hyper: March 2023

Fri, 07 Apr 2023 13:31:29 +0000

The days are growing longer up here, and work to make hyper better and better continues onwards!

There was some particularly exciting releases, and a bunch of conversations had in March 2023.

hyper 1.0

Preparing users for upgrading to 1.0

One of the important parts of the hyper 1.0 polish period is making sure upgrading from 0.14 to 1.0 is as smooth as we can make it. hyper v0.14.25 is here to help. @kxt and @oddgrd backported the client and server APIs from 1.0, and added opt-in deprecation messages to help you be ready to upgrade. The deprecation warnings are meant to help you, not to annoy you.

You can enable them to see where you can start preparing your code now:

[dependencies]
# besides whatever other features you've enabled...
hyper = { verion = "0.14.25", features = ["backports", "deprecated"] }

RC4 discussions

We’ve been discussing how to wrap up the last couple of changes for 1.0, to put out an RC4 to bake some. We have some decent answers, and can get to work.

hyper’s own Service trait will change from &mut self to &self. This better aligns with reality, as most services already need to share state behind some synchronization mechanism, and needing to &mut self in the method call doesn’t help.

hyper will use its own IO traits with forwards-compatibility in mind. We want to be able to support both poll-based and completion-based (think epoll vs io-uring) IO models.

HTTP/3

We’re working on HTTP/3 in a separate crate, h3, with the goal of fitting it into hyper.

reqwest includes experimental HTTP/3 support

With reqwest v0.11.15, you can try out HTTP/3, on the client side, in reqwest right now! It’s currently experimental, which means a couple things: it might not work perfectly. Let us know! It also might be disabled in new patches, as we fiddle with it. Lastly, you need to more explicitly opt-in to the instability.

That means that besides enabling the http3 feature from your Cargo.toml, you also need to pass RUSTFLAGS="--cfg reqwest_unstable" to the compilation job.

Huge thanks to @kckeiks for integrating h3 into reqwest!

h3-quinn upgraded to Quinn 0.9

We found a way to upgrade h3-quinn, using stream::unfold and BoxStream.

The h3 crate tries to be generic over any QUIC implementation. The h3-quinn crate implements the h3::quic traits for the Quinn crate. The traits currently use poll-based methods (async function in traits isn’t stable yet, and they also wouldn’t allow polling multiple things at the same time). Newer Quinn embraced using async fn on its types, instead of returning named futures, which is very fair thing to do. But it did make it hard to figure out how to implement poll-based methods over async fns.

I realized we could do a async-move-dance to solve this. We make an async move block, moving in the type and awaiting the future, which then returns a tuple of the original type and the return value. @Ralith made the suggestion to use stream::unfold, which streamlines that pattern. Then @inflation quickly wrote it up, and we were able to upgrade to Quinn v0.9.

Contribute

Want to help us out? Even trying the new releases out and give us feedback is extremely useful. Of course, contributing reviews is a great help too. Come by and say hi!

This Month in hyper: February 2023

Thu, 02 Mar 2023 15:38:06 +0000

After recapping the 2022 year, here’s what the amazing contributors have been doing to make hyper ever better during January and February of 2023.

Releases

hyper v0.14.24: fixes some expect-continue behavior, and reduces the internal max allocation in to_bytes.
h2 v0.3.16: adds a missing piece for Extended CONNECT, and several bug fixes (memory reduction, panics)
reqwest v0.11.14: adds Proxy::no_proxy(url) that works like the NO_PROXY environment variable, and several internal optimizations reducing copies and memory allocations.
tower-http v0.4.0: a new decompression layer for Requests, ServeDir and ServeFile now translates IO errors into Responses, and adds a more flexible ValidateRequest layer.

hyper 1.0

We released RC3, which fixed up some missing pieces in the API. Places needing an Executor now ask for one, and we added hyper::rt::bounds to publicly expose nameable but future-proof Executor trait “aliases”, so libraries building on top of hyper can use them as bounds. We also added a few state getters for SendRequest which were needed for the next exciting part.

The higher-level pooling Client from 0.14.x has been ported to hyper-util. This was the most common blocker preventing people from trying out the release candidates. You can now use the legacy::Client with hyper 1.0.0-rc.3, and have the normal connecting/pooling client experience you’re used to (see the example).

We’re still in the hyper polish period 💅. There’s still a little bit of time left to get us your feedback! It’s the most important part of this period. Or join us in one of the four polish areas (or help lead one)! You can also come chat with us about anything.

We took some extra time to focus on RC4, which has the last few breaking changes to go. Likely, hyper will vendor it’s own IO traits, and change Service::call to be &self instead of &mut self. See the related issues if you have feedback.

The extra time will also allow us to investigate having a security review done for 1.0, to prevent any gotchas.

HTTP/3

We’re working on HTTP/3 in a separate crate, h3, with the goal of fitting it into hyper.

Within the next couple days, we’d like to publish a v0.0.1 to crates.io. The API likely will change in the very near future, but knowing exactly how requires allowing experimenters like reqwest to try it out.

hyper-ish 2022 in review

Wed, 18 Jan 2023 16:28:08 +0000

Quite the eventful year! With 2022 over¹, I want to take some time highlight what’s happened in hyper and the immediately surrounding landscape.

A quick personal note, I wrote much more this year!² Yay!

hyper

Let’s start with some general things about hyper.

On the human side, there were 62 unique contributors to hyper in 2022. We added 1 more collaborator and 2 triagers, and even defined what those roles even mean.

In an effort to share the lovely work those humans do, I started writing monthly hyper updates.

And we celebrated 10 thousand stars this year! ⭐

hyper 1.0

As I mentioned in last year’s review, 2022 was the year we began to prepare to release a stable 1.0 of hyper.

We started with a v1.0 timeline (of which we’re in the final steps). After much discussion with users, we put into writing hyper’s VISION, which defined where we were going. Shortly aftewards came the v1.0 ROADMAP, outlining how to get there.

And then, a summer flurry of coding and hairy diffs.

On the other side, late last year, we published v1.0.0-rc.1. This represented the likely working library that we’d like set down into a stable parking spot for a while. While people check it out and give us feedback, we entered the hyper polish period, making sure everything feels good. That’s where we still are, for a short period more.

You can follow along on the project board, and specifically the 1.0 meta issue ties together non-issue related details.

hyper in curl

The work to make hyper an HTTP backend in curl from last year continued throughout this year. There’s just a few remaining tests in curl’s large HTTP suite that didn’t work when hyper is enabled. Several wonderful people showed up to dig in and find out exactly why.

To try to empower others to do, I wrote up a help-us-finish guide, explaining step-by-step how anyone could help us finish this all-important work. Later in the year, I streamed a hyper-in-curl debug session, where you could watch me follow the guide, and then bash the keyboard randomly while trying to understand what the issue is. You can still watch the recording, or see these details about what test was solved and the pull requests that came out of it.

I also joined Daniel Stenberg virtually by presenting about hyper in curl at curl-up 2022.

h3

We’ve been working on the h3 crate, providing HTTP/3 that is generic over any QUIC implementation, with the goal of integrating into hyper directly. The repository has had a working server and client which already interoperates with other implementations.

I’ve written about it in the monthly updates, but here’s some highlights I’m excited about:

We added 3 new collaborators who have been driving the work: @eagr, @g2p, and @Ruben2424.
@stammw implemented graceful shutdown for the server and the client.
@Ruben2424 added GREASE support via an default-on option which sends random reserved identifiers, such as frames, settings and streams, to help prevent ossification that would make future extensions harder.
@eagr made it so we now track compliance with the HTTP/3 specifications, by using special comments that are compared with the spec text, and it even outputs a report updated as part of our continuous integration.
@Ruben2424 also added h3spec to CI, and fixed the missing pieces it noticed.
@g2p documented the entire API.

While there’s still specific work that can be done on the h3 crate itself, it’s time to consider next steps to get it into user hands. To that end, there’s even a pull request for reqwest to use h3! There’s just some details to work out around publishing unstable versions so reqwest can depend on it. We hope this experimental support will help us iron out any usage annoyances, so we can start landing it in hyper proper.

tower

While tower isn’t tied to hyper, we’ve always meant for people to easily combine the two libraries to make powerful, opinionated HTTP stacks.

In the later half of the year, we started having discussions about making tower easier. Up until now, it has mostly felt like expert mode. But if done right, we shouldn’t be telling users “no, you don’t hold it that way” when they try to implement retries slightly wrong and storm their servers. So, to that end, Lucio put together a big issue outlining how we can make retries better.

Another discussion started about the path to tower 1.0. This brought some interesting questions around how Service handles backpressure, whether we could make that any better. Certainly, something else we would want to consider is if Service can make use of async fn in traits.

The tower-http repository continued to see additions. Mostly middleware that are specific to HTTP that many people would find beneficial, such as RequestBodyLimit, RequestBodyTimeout, ResponseBodyTimeout, and ValidateRequest.

I’ve also kicked around the idea loosely about cracking open the reqwest crate, and turning its various features into tower middleware. Then, reqwest is just a single opinionated way to build up a client stack. The community would be more empowered to customize the order of layers, adding or removing or swapping, and still have the power that they come to expect from using reqwest.

What are some possiblities in 2023?

Besides launch hyper 1.0, of course.

These are are all things that many people have asked for, and I’d like to see done. But realistically, most will require help from you!

Improved middleware
HTTP/3 in hyper
Tracing and Metrics
h2 performance improvements
An even-lower level http1 codec crate
Tower-ify reqwest

I’ll likely be focused at the top of that list, but would welcome anyone interested jumping into an issue (or discussing on Discord if you prefer). Really, the biggest success would be empowering others to be the leaders and owners on these things. Do you want to be one of them?

Sorry for the delay, illness struck right at the beginning of the year. ↩
I wrote barely anything in 2021, 2020… actually for quite a few years. I used to blog multiple times a month back in 2013, but kind of teetered off the following year. Anyways, I’m really liking it (again), so here’s to more! ↩

This Month in hyper: November 2022

Fri, 02 Dec 2022 15:50:45 +0000

The northern hemisphere starts to cool, the trees shed their leaves to conserve energy, cultured fans watch the quadrennial football tournament, and some magnificient contributors stay warm by continuing to make hyper ever better!

Releases

hyper v0.14.23

@jfourie1 found and fixed a nasty bug in hyper’s HTTP/2 client dispatcher, which could result in a stalled connection under high concurrency.

reqwest v0.11.13

The headline addition is a ClientBuilder::dns_resolver(), which now allows users to implement completely custom DNS resolvers for reqwest to use.

hyper 1.0

With the release of rc1 last month, I wrote about how we’re now in the hyper polish period 💅.

@programatik29 volunteered to co-lead the util area. After many contributions to get us to rc1, @oddgrd joined us to lead the docs area. @vi noticed that the HTTP/2 client SendRequest should implement Clone.

We still eagerly welcome you trying out the release candidate and providing us feedback. It’s the most important part of this period. Or join us in one of the four polish areas (or help lead one)! You can also come chat with us about anything.

hyper in curl

I fixed curl’s c-hyper.c to classify headers from CONNECT and 1xx responses as such, making two more unit tests pass. I streamed the process, in case you’d find it helpful to watch someone who mainly writes Rust flounder around debugging and fixing C. You can try it too!

HTTP/3

We’re working on HTTP/3 in a separate crate, h3, with the goal of fitting it into hyper.

@eagr made it so recv_trailers doesn’t require holding onto the SendStream side. @g2p cleaned up a huge swath of clippy errors.

@g2p and @Ruben2424 joined as h3 collaborators, thanks to their excellent and continued work!

Tower

Tower (and tower-http) are a protocol-agnostic RPC framework with middleware, and they combine nicely with hyper.

@alexrudy added a new BoxCloneServiceLayer, and @leoyvens fixed a couple bugs in CallAll. @davidpdrsn made BoxService implement Sync.

Oh, and while not Tower, there was a big new Axum v0.6 release!

seanmonstar

Micro: Accessors can reveal internal representation

Micro: Rust Errors generally shouldn't be PartialEq

Micro: Owning my microblog with POSSE

hyper User Survey 2025 Results

Why did we do a survey?

The results

Current usage

Future work

Contributing

The insights

Outro

2025 in review

Two years of being independent

Maintaining hyper

Modularizing reqwest

The value of deadlines

Security takes time

Talks

Pondering

reqwest v0.13 - rustls by default

rustls is now the default TLS backend

Certificate verification features are consolidated

Soft-deprecation to improve option naming

Other crate feature adjustments

Thanks!

hyper-util Composable Pools

We started with the users

The initial 4 working pools

Ineffably unstable

Real world usage: in reqwest

Being modular opens up customization

Pools ready

hyper User Survey 2025

Podcast: Netstack.fm, story of Rust's networking with hyper

hyper HTTP/2 (Didn't) MadeYouReset

What is the attack

What hyper does

What we did

What you can do

warp v0.4

Filters remain the focus

Upgraded to hyper v1

Crate features on a diet

Iterating on the Server builder

reqwest retries

Policy composition

Budgets by default

Scoped retries

Cloning requests

Classifiers

Probable enhancements

Exploring and upstreaming

Check it out

Body::poll_progress

Background: what is the Body trait?

Problem: backpressured cancelation

Why not just timeouts?

More generally, yet not

Solution: Body::poll_progress()

Alternatives: not poll_closed

Request for Comments

A More Modular reqwest

Easy and modular

Some concrete work

Better default TLS

Getting to work

2024 in review

A year of being independent

hyper

reqwest

Security

Podcast

bye curl

Questions to ponder in 2025

hyper Roadmap 2025

Open source is guiding

Focus Areas

A living document

Contribute

Background: what is the `Body` trait?

Solution: `Body::poll_progress()`

Alternatives: not `poll_closed`

hyper’s `h2` is somewhat affected

`io_uring`

Five whys⁵