seanmonstar

hyper HTTP/2 Continuation Flood

Patches are available for h2, v0.4.4 and v0.3.26, to harden against a newly disclosed HTTP/2 attack vector, mostly for servers. If you need help, reach out for support.

If you’re curious about more, read on.

What is the attack?

In HTTP/2, there are a bunch of frame types. Some of those frames are related to sending headers (or fields). Since there is a maximum frame size, in order to send more headers than fit, a peer can send CONTINUATION frames that indicate more headers for the message.

That mechanism is the source of a new attack, trying to send an infinite number of CONTINUATION frames to starve or kill a server. This was a coordinated release, with various implementations involved. VU#421664 has details about the others.

hyper’s h2 is somewhat affected

The HTTP/2 specification doesn’t state a default value for SETTING_MAX_HEADER_LIST_SIZE. You should set that to something reasonable for your use case. But even if you don’t, h2 picks an emergency default, which is fairly high so to as to not reject requests for users who need big requests.

Once a stream of headers and continuation frames reach the limit, either configured or the large default, memory growth will cap there. h2 will keep reading and decoding, but discarding any new headers. Really, it’s trying to keep the HPACK table in sync, and will send back a stream error once finished of “too big”.

The problem is that h2 won’t stop reading as long each CONTINUATION frame says there are more coming. So the memory stays the same, but it will use CPU to read and discard. And since the headers are never “complete”, the application doesn’t get to know about the processing of frames.

However, if deployed on Tokio (or some other runtime with a similar feature), the task budget will prevent it from locking out other tasks. Other requests can be served, and responded to. But it will make it slower.

A degradation of service.¹

What we did

Besides testing and determining the above, we also worked on a fix.²

We realized that while a lower SETTING_MAX_HEADER_LIST does help, it’s not the only thing that needs to be checked. The reason for that is because a CONTINUATION frame could be sent with the smallest header value, and thus take longer to reach that maximum. That will still consume resources processing frames.

We figured that a maximum needed to be set on the allowed CONTINUATION frames. But a naïve hard-coded number has its own problems.

The maximum frame size isn’t usually changed, which means you’re stuck with 16kb frames. And if you do want to allow message headers bigger than that, then you need to allow CONTINUATION frames. Since both frame size and header list size are configurable, a hard-coded limit wouldn’t work.

We settled on calculating how many CONTINUATION frames would be needed for a legitimate message to send up to the max header list size. We also add a little bit of padding to that number, to account for frames that may not be perfectly packed.

The patch has been tried out on some production servers, and found no false positives.

What you can do

• If you don’t use HTTP/2 on a server, then don’t worry.
  • If you do, but are not exposed to untrusted L7 traffic, also don’t worry.
• If you don’t set the SETTING_MAX_HEADER_LIST_SIZE, you should consider doing that.
• You should run cargo update -p h2 in your application to pull in the latest versions with fixes (v0.4.4 and v0.3.26).

If you need help, reach out for support. Or send thanks, so that months long security work like this can be done in the future.

Podcast: Rustacean Station, hyper 1.0, and independent maintainership

I was recently a guest on the Rustacean Station podcast. It was nice to catch up after a couple years since my last appearance on the show. We spoke about hyper, how and why it became v1.0, becoming an independent maintainer, future work, and more.

I thought I’d grab a few fun and interesting quotes, with their timestamps:

Sponsoring is not the Mafia (11:48)

The lowest engagement is sponsorship where it’s like, what am I getting from your sponsorship? If you turn it around and think about, well, a lot of times open source is like this common good that everyone kind of benefits from and I don’t want to pay for it because someone else is going to pay for it, right? It’s kind of like this Russian roulette of like, well, if no one pays for it, then eventually that thing is going to go away.

So sometimes when I talk to the companies, I just ask them, hey, how much are you using this stuff? You’re using it a lot. Cool. How annoying or how much work would it be to have to maintain it yourself if I were to just disappear? Oh, then we’d have to put a whole engineer on that or something like that. Okay. So if you think of it as a business risk mitigation to sponsor, then I can keep doing it. And then it becomes like, okay, well, we would pay a full-time engineer to do this all year. We can pay a way smaller amount to just have the sanity check that this is not going away.

And so what do they get? They get a business risk mitigation.

You’ll have to excuse me, but I’ve been watching a lot of Sopranos. And so when you said, how bad would it be if this thing happened to go away? It sounded a little bit like a tactic they might use to secure the contract.

(laughs) Yeah, it’s not like an insurance racket.

Access to the maintainer / advising (12:50)

There’s more things that I can offer and I do offer. Another one is people want to be able to ask, hey, we’re using your stuff and we want to know are we using it correctly or, hey, we’ve been trying to use it and we have this problem and we don’t understand why. Could you take a look?

If it’s a private project, people aren’t going to post their source code into a public issue. I’m not going to have the time to go and take a look unless it’s like, hey, let’s get a retainer and I can now sign an NDA. I’m not going to steal their code, all that legal stuff.

It’s like getting an advisor, a reviewer, and then I can then take that knowledge and go back and be like, okay, so I now have knowledge that like company so-and-so is using this in a way that I didn’t expect. How can I make it better for them? So like, both sides benefit, but it’s only possible if I set up contracts and everyone’s legally happy.

Breaking people, not kneecaps (30:55)

If something would be a breaking change, then we label it with a breaking change. And it’s not something we can do right now. We can close it or postpone it or something. And then, you know, maybe three years from now, go and take a look at all the issues that are labeled that and say like, yeah, you know what, this would be worth breaking for. But at the same time, I’d love to not have to break people, even though, you know, the promise is only for three years. I’d love to not have to.

Did you just say break people? We did go back to the mafia reference, right? But no relation, right?

Exactly. Yeah. No breaking kneecaps.

Client middleware (32:12)

The other major thing is improving middleware. There’s all this stuff in tower, and it’s great. Like there’s Axum. You can use it to make really powerful servers. But the point of this middleware was actually that you could use it both ways. You could use it for servers, but you could also use it for clients. And that doesn’t work as well.

The most popular thing to use for clients is reqwest. And it doesn’t fit in to tower middleware. It has a bunch of extra stuff like redirects and some other things that it does. Various kinds of timeouts, which would be great middleware, but it’s currently all just wrapped up in reqwest.

And then also, if you even just look at the middleware in the first place, it’s like opaque. You have to understand how to do service discovery to then use load balancing. You have to understand how to build a retry policy before you can fit in retries and understand what is a retry budget.

It’s complicated. And it’d be nice if people could just get, hey, this is better for 95% of use cases. And if you really want to go and tweak your retry policy, sure you can. There’s the escape hatch over there.

But that’s my other thing to work on. Make reqwest and like Tower’s client middleware merge together a whole lot nicer so you can have much better stacked clients.

Retry storms and budgets (33:50)

Since there wasn’t a really easy plug-in, then people implement retries themselves. And that includes making themselves vulnerable to retry storms and just not doing it in a safe way. And the thing is that we have this middleware and things that would protect you from that, but they’re just complicated enough that people are like, ah, I can just retry in a loop.

But then you smash the server once things start falling apart. And it’d be so much better if you just add in a retry layer. It’s going to do things wisely. And maybe you say, you know what, on this URL, never retry it, but otherwise do the default thing.

It’d be so much nicer if you didn’t have to understand how bad retries can go.

If you just do a simple counter, then you might have counters at various layers. And maybe your counters aren’t shared. And now even though you’re trying to be nice to the server, you’re still retrying on thousands of times, whereas a shared budget or something would notice after a few retries, hey, the server’s overloaded. Let’s stop pounding it. Because as long as we keep trying, it’s never going to get back up. That’s one of the problems with retry storms.

The linkerd blog had a good post on how you can fix that using Tower middleware. But I’d prefer it if people didn’t have to read that to use it.

And plenty of other topics

Go have a listen, I hope you find it informative!

reqwest v0.12

Today marks the v0.12 release of reqwest, a higher-level, batteries-included HTTP client for the Rust language.

What’s new

The headline feature of reqwest v0.12 is the upgrade to hyper v1. reqwest does a lot of custom work to add features to hyper internally, but doesn’t need to expose too much of it publicly. This was what took the majority of the work to upgrade. Still, as outlined in the the hyper v1 announcement, the parts that are public have stabilized. The most obvious improvement to most users of reqwest will be the stable integration with http v1 types. This should remove the last blocker for a lot of people to finish upgrading to hyper v1.

With the breaking change, we made a few more improvements. Several optional features have been added which previously were required, such as http2 and charset. This allows disabling them and making the compilation time and size lower if not needed. We also converted all the implicit optional dependency features to dep: syntax.

We had to disable the HTTP/3 feature, but it was experimental anyways. It’s a goal to bring it back as soon as possible. We might also be able to add some other QUIC backends, such as s2n-quic.

See the release for more details.

Coming next

As mentioned a few times before on this blog, my focus next is to make a bunch of the powerful features in reqwest available as middleware. reqwest will remain easy to use. But it will be possible for people configure their own client stacks, without needed to fork or copy code. It will also mean even if you use the reqwest default easy stack, you can more easily integrate it with other tower middleware, such as retries or load balancing.

Thank you!

reqwest usage keeps on growing, and it’s both humbling and exciting to watch! For example, rustup uses reqwest as its default download backend, helping all Rust developers to keep their compiler up to date. Oxide uses reqwest in their Progenitor library that powers their OpenAPI clients.

Thank you for all your contributions, both in reporting issues, and in helping fix them. <3

Sponsor and Support

If your company uses reqwest, consider becoming a sponsor. Additionally, for private advice, reviews, security help, and access to the maintainer, get in touch for support.

2023 in review

A year of change, and of stability. Let me briefly reminisce and highlight what happened in 2023.

Independent

I became an independent maintainer in June 2023. This was a somewhat scary decision, but it did exactly what I hoped it would: my work-life balance feels fantastic. I’m also glad to be able to do similar to what I did at AWS—meeting with and advising teams with serious deployments—but with a wider variety of use cases. (Send me an email if that interests you.)

hyper

First, some stats about hyper over the past year. There were 90 unique authors this year, which is 40% growth from 2022! We had two more people sign up to be triagers.

v1 🚀

We released hyper v1 in November. What a ride. It brought changes, moving the less stable side out into hyper-util. But it also signaled a core that won’t be changing any time soon. Stability.

The ecosystem caught up quickly. There were releases for tower-http, headers, and Axum ready to go just a couple weeks after. It enabled some other cool things, like the general availability of the AWS SDK for Rust.

We closed out the year with v1.1, bringing back core pieces needed to make graceful shutdown in servers easier again.

Security

I reviewed at least 10 security reports, perhaps a couple more I didn’t keep track of. This includes the wider HTTP/2 rapid reset attack that hyper wasn’t affected by. The amount of time I spend on security reports keeps on increasing. That makes sense, we announced stability of v1, which surely made more people take a look. It also is a sign of more production deployments, with companies wanting to audit their dependencies.

HTTP/3

Another priority this year was to make progress on hyper’s HTTP/3 support, currently under development in the h3 crate. We released some initial 0.0.1 releases (and a few more subsequent ones), specifically to make it easier for people to use. reqwest gained unstable HTTP/3 support, using h3, and some brave users have enabled it, found it working well, and are now asking if we can make it stable. A couple of other fine folks worked to make the h3-webtransport crate, building on top of h3.

Future Focus

Doing a bit forward looking, what’s the plan for 2024? Well, of course it could change at any moment, but these seem to be the things people most ask me for, and most need.

HTTP/3 in hyper

I hope to make significant progress towards getting HTTP/3 support directly in hyper. I’ll work on a proper proposal, but here’s some unordered steps in that direction. Stabilizing the feature in reqwest. Set up an auto-updating h3 server for interop testing. Dig away at the compliance report, both by labeling more of the parts already working, and adding any missing parts. Propose how to expose it in hyper, which will be tricky so as to not tie hyper to any specific TLS library. And then get it actually added, likely as a hyper_unstable_h3 feature to start.

Level up Client middleware

There’s a lot of great middleware now. But it can still feel like it requires an expert to use it properly. I’ve been hinting at for a while that I’d like to do for clients what Axum has done for servers. I hope to do that with reqwest.

reqwest does a lot of useful things, but if someone wants to customizing it beyond the options that reqwest exposes, they have to reimplement a lot (or live with a fork). I want to make most of reqwest’s features tower middleware. reqwest will still have a standard “recommended” client. But it should be easier to build up your own custom stack.

And while we’re at it, I hope to make some of the most important and yet most difficult middleware much easier to sprinkle in: retries, limits, and load balancing.

On-going maintenance

I also must carve out explicit time for maintenance work. There’s bugs that need fixing. Reviewing and triage takes a lot of my energy. I want to improve the docs and guides. The amount of security reports received is growing, and those take time to investigate and respond or patch and disclose, depending on their validity and severity. This also includes time with my sponsors, which helps identify maintenance work priorities.

Want to join us?

hyper v1

I’m excited to announce v1.0 of hyper, a protective and efficient HTTP library written in the Rust programming language. hyper provides asynchronous HTTP/1 and HTTP/2 server and client APIs, allowing you to bring your own IO and runtime.

It’s been exciting and humbling to watch users build awesome things. Cloudflare uses hyper within Oxy, its next generation proxy framework to handle traffic at considerable scale. After Discord’s 5x improvement to @mention response times a few years ago, they have moved most of their critical systems to depend on Rust and hyper. curl has a currently experimental HTTP backend built on hyper with the goal of making the Internet safer.

Marc Brooker, a Distinguished Engineer at AWS, commented:

When building our new container-loading data plane for AWS Lambda, we expected to need a custom binary protocol. In production, we’ve found the overhead of hyper to be SO low that we are excited for it to continue powering our services.

Johan Andersson, CTO at Embark, said:

We have been using and relying on hyper for the last 5 years for our gRPC and REST services, tools, libraries, and embedded in our next game built in Rust. It has been rock solid across all of our usages, and it really is a foundational library for the Rust ecosystem. Congrats on 1.0!

The best way to get started is to check out the guide.

Stability here we come

Over the past 9 years, hyper has grown from a web developer’s side project into a solid library powering huge network applications. It’s time to grow up. After bringing async/await support in v0.14, we focused on providing a set of basic APIs that would keep hyper safe, fast, and flexible. This meant removing some of the more opinionated “higher level” pieces. Those belong elsewhere, like hyper-util, reqwest, Axum.

This release signals some stability. Major versions, like 1.0, are stable for at least 3 years.¹ We also keep a MSRV that is at least 6 months old.² We’ll add new features, and we still have a couple places to experiment: in the hyper-util crate, and hyper_unstable compiler flags.

Starting in v0.14.25, we added a backports feature which brings the new core APIs to you immediately. Combine that with the deprecated feature, and you’ll be guided to making your existing code ready for the upgrade to 1.0. Be sure to check out the upgrade guide!

Next

The most immediate next steps are to update the other core parts of the ecosystem that depend on hyper: reqwest, Axum, Tonic. But after that, there’s plenty more to do. You’re welcome to come join us!

HTTP/3

I would like this to be my next focus. We’ve been building up the h3 crate, and reqwest has unstable support now. I’d like to stabilize the feature in reqwest, and explore how we can make it available in hyper directly. Then we can have easy HTTP/3 servers, too!

The trickiest question is making it available without being tied to a TLS/QUIC library. Then, users could choose to use quinn, or s2n-quic, or msquic, or any other.

Stabilize in curl

The biggest parts of making hyper work in curl are done. Someone with experience in Rust and C could make a huge dent in Internet safety helping to get it over the finish line.

Middleware

There’s some excellent middleware available already in tower and tower-http. But several of the important ones are just a little (or a lottle) too difficult to add to a stack. I’d also love for there to be some recommended stacks for servers and clients, that bundle together the right middleware that most people would need. To that end, I’ve mentioned before breaking open reqwest such that all of its features are middleware you can customize.

Tracing and Metrics

It’s possible to currently get a decent set of logs using tower_http::trace. It’d be better if you could get more fine-grained traces and metrics. Probably with some stabilized integration with tracing directly in hyper. Maybe some sort of hyper-metrics, similar to tokio-metrics.

io_uring

Part of the reason we made hyper have its own IO traits was to be able to adapt them for completion-based IO. I believe having decent support and benchmarks could be had pretty soon, by a motivated individual.

Thanks

A huge thank you to all our amazing contributors. You’ve made this project the success it is, and helped move hyper along the journey to 1.0. I’d like to follow up with a separate post specifically thanking you all.

Thanks to the companies who have sponsored the creation of hyper: AWS, Buoyant, Mozilla, Rust Foundation, Fly.io, Embark and others.

Your company could also become a sponsor or get support!