We now stop reading from the socket unless asked to,
when we reach the request body. The option
initial_stream_flow_size controls how much data
we read without being asked, as an optimization.
We may also have received additional data along
with the request headers.
This commit also reworks the timeout handling for HTTP/1.1
because the stray timeout message was easily reproducible
after implementing the flow control. The issue should be
gone for good this time.
While the protocol does not allow sending data before
receiving a successful Websocket upgrade response, we
do not want to discard that data if it does come in.
Now both HTTP/1.1 and HTTP/2 follow the documented format.
HTTP/1.1 was including an extra element containing the
StreamID before, which was unnecessary because it is also
given as argument to the callback.
HTTP/2 early_error will now include headers in its PartialReq.
This allows giving custom metadata to the metrics stream handler.
This can be useful to for example provide the name of the
module handling the request which is only known after routing.
But any user data is allowed.
When called multiple times the user data maps are merged.
Fix a case where Cowboy was waiting for more data that simply
did not come. Now Cowboy will generate an error immediately
when a header line has no colon separator.
These test cases come from known request smuggling attack
vectors. Cowboy was not vulnerable to any of them.
A number of HTTP/2 CVEs were documented recently:
https://www.kb.cert.org/vuls/id/605641/
This commit, along with a few changes and additions in Cowlib,
fix or improve protection against all of them.
For CVE-2019-9511, also known as Data Dribble, the new option
stream_window_data_threshold can be used to control how little
the DATA frames that Cowboy sends can get.
For CVE-2019-9516, also known as 0-Length Headers Leak, Cowboy
will now simply reject streams containing 0-length header names.
For CVE-2019-9517, also known as Internal Data Buffering, the
backpressure changes were already pretty good at preventing this
issue, but a new option max_connection_buffer_size was added for
even better control over how much memory we are willing to allocate.
For CVE-2019-9512, also known as Ping Flood; CVE-2019-9515, also
known as Settings Flood; CVE-2019-9518, also known as Empty Frame
Flooding; and similar undocumented scenarios, a frame rate limiting
mechanism was added. By default Cowboy will now allow 1000 frames
every 10 seconds. This can be configured via max_received_frame_rate.
For CVE-2019-9514, also known as Reset Flood, another rate limiting
mechanism was added and can be configured via max_reset_stream_rate.
By default Cowboy will do up to 10 stream resets every 10 seconds.
Finally, nothing was done for CVE-2019-9513, also known as Resource
Loop, because Cowboy does not currently implement the HTTP/2
priority mechanism (in parts because these issues were well known
from the start).
Tests were added for all cases except Internal Data Buffering,
which I'm not sure how to test, and Resource Loop, which is not
currently relevant.
This should limit the amount of memory that Cowboy is using
when a handler is sending data much faster than the network.
The new max_stream_buffer_size is a soft limit and only has
an effect when the cowboy_stream_h handler is used.
As a result we explictly reject path_info components that include
a forward slash, backward slash or NUL character. This only applies
to the [...] part of the path for dir/priv_dir configuration.
Also improve the tests so that they work on Windows.
For long-running connections it was possible for the connection
window to become larger than allowed by the protocol because the
window increases claimed by stream handlers were never reclaimed
even if no data was consumed.
The new code applies heuristics to fix this and reduce the number
of WINDOW_UPDATE frames that are sent. It includes six new options
to control that behavior: margin, max and threshold for both the
connection and stream windows. The margin is some extra space
added on top of the requested read size. The max is the maximum
window size at any given time. The threshold is a minimum window
size that must be reached before we even consider sending more
WINDOW_UPDATE frames. We also avoid sending WINDOW_UPDATE frames
when there is already enough space in the window, or when the
read size is 0.
Cowlib is set to master until a new tag is done.
A bug was fixed in cowboy_rest where when content_types_provided
returned a media type with a wildcard as first in the list, and
a request comes in without an accept header, then the media_type
value in the Req object would contain '*' instead of [] for the
parameters.
It allows disabling the chunked transfer-encoding. It
can also be disabled on a per-request basis, although
it will be ignored for responses that are not streamed.
It allows to temporarily disable Websocket compression
when it was negotiated. It's ignored otherwise. This
can be used as fine-grained control when some frames
do not compress well.
They allow the server to configure what it is willing to accept
for both the negotiated configuration (takeover and window bits)
and the other zlib options (level, mem_level and strategy).
This can be used to reduce the memory and/or CPU footprint of
the compressed data, which comes with a cost in compression ratio.
Returning the atom auto instead of a callback informs Cowboy
that it needs to handle range requests automatically. This
changes the behavior so that the ProvideCallback function
is called and then Cowboy splits the data on its own and
sends the response without any other user involvement other
than defining the ranges_provided/2 callback.
This is a quick and dirty way to add range request support
to resources, and will be good enough for many cases including
for cowboy_static as it also works when the normal response
body is a sendfile tuple.
