Doing so will let us notice when the connection is gone instead
of waiting for timeouts, at least in the cases where the remote
socket was closed properly. Timeouts are still needed in case
of TCP half-open problems.
This change means that the order of stream handler commands is
more important than before because socket errors may occur
during the processing of commands.
Note: This commit makes cowboy depend on cowlib master.
Graceful shutdown for HTTP/2:
1. A GOAWAY frame with the last stream id set to 2^31-1 is sent and a
timer is started (goaway_initial_timeout, default 1000ms), to wait
for any in-flight requests sent by the client, and the status is set
to 'closing_initiated'. If the client responds with GOAWAY and closes
the connection, we're done.
2. A second GOAWAY frame is sent with the actual last stream id and the
status is set to 'closing'. If no streams exist, the connection
terminates. Otherwise a second timer (goaway_complete_timeout,
default 3000ms) is started, to wait for the streams to complete. New
streams are not accepted when status is 'closing'.
3. If all streams haven't completed after the second timeout, the
connection is forcefully terminated.
Graceful shutdown for HTTP/1.x:
1. If a request is currently being handled, it is waited for and the
response is sent back to the client with the header "Connection:
close". Then, the connection is closed.
2. If the current request handler is not finished within the time
configured in transport option 'shutdown' (default 5000ms), the
connection process is killed by its supervisor (ranch).
Implemented for HTTP/1.x and HTTP/2 in the following scenarios:
* When receiving exit signal 'shutdown' from the supervisor (e.g. when
cowboy:stop_listener/3 is called).
* When a connection process is requested to terminate using
sys:terminate/2,3.
LH: Edited tests a bit and added todos for useful tests to add.
When calling cowboy_req:reply/4 with a body a crash will occur
resulting in a 500 response. When calling cowboy_req:stream_reply/2,3
and then attempting to send a body a crash will occur.
This is mostly to ensure that the GOAWAY frame is properly
received on Windows in some tests, but should be benefitial
also in production in particular when clients are slower.
This resulted in a badarith error due to the current flow being
set to infinity when the body has been fully read. A test case
has been added reproducing the issue.
The flow control is now only set to infinity when we are
skipping the request body of the stream that is being
terminated. This fixes a bug where it was set to infinity
while reading a subsequent request's body, leading to a
crash.
The timeout is no longer reset on stream termination.
Timeout handling is already done when receiving data
from the socket and doing a reset on stream termination
was leading to the wrong timeout being set or the right
timeout being reset needlessly.
This reduces the number of times we need to ask for more packets,
and as a result we get a fairly large boost in performance,
especially with HTTP/1.1.
Unfortunately this makes Cowboy require at least Erlang/OTP 21.3+
because the ssl application did not have active,N. For simplicity
the version required will be Erlang/OTP 22+.
In addition this change improves hibernate handling in
cowboy_websocket. Hibernate will now work for HTTP/2 transport
as well, and stray or unrelated messages will no longer cancel
hibernate (the process will handle the message and go back into
hibernation).
Thanks go to Stressgrid for benchmarking an early version of this
commit: https://stressgrid.com/blog/cowboy_performance_part_2/
When sending a complete response it is far more efficient
to send the headers and the body in one Transport:send/2
call instead of two or more, at least for small responses.
This is the HTTP/2 counterpart to what was done for HTTP/1.1
many years ago in bfab8d4b22.
In HTTP/2's case however the implementation is a little
more difficult due to flow control. On the other hand the
optimization will apply not only for headers/body but also
for the body of multiple separate responses, which may need
to be sent all at the same time when we receive a WINDOW_UPDATE
frame.
When a body is sent using sendfile however a separate call
is still made.
We now stop reading from the socket unless asked to,
when we reach the request body. The option
initial_stream_flow_size controls how much data
we read without being asked, as an optimization.
We may also have received additional data along
with the request headers.
This commit also reworks the timeout handling for HTTP/1.1
because the stray timeout message was easily reproducible
after implementing the flow control. The issue should be
gone for good this time.
While the protocol does not allow sending data before
receiving a successful Websocket upgrade response, we
do not want to discard that data if it does come in.
Now both HTTP/1.1 and HTTP/2 follow the documented format.
HTTP/1.1 was including an extra element containing the
StreamID before, which was unnecessary because it is also
given as argument to the callback.
HTTP/2 early_error will now include headers in its PartialReq.
This allows giving custom metadata to the metrics stream handler.
This can be useful to for example provide the name of the
module handling the request which is only known after routing.
But any user data is allowed.
When called multiple times the user data maps are merged.
Fix a case where Cowboy was waiting for more data that simply
did not come. Now Cowboy will generate an error immediately
when a header line has no colon separator.
These test cases come from known request smuggling attack
vectors. Cowboy was not vulnerable to any of them.
A number of HTTP/2 CVEs were documented recently:
https://www.kb.cert.org/vuls/id/605641/
This commit, along with a few changes and additions in Cowlib,
fix or improve protection against all of them.
For CVE-2019-9511, also known as Data Dribble, the new option
stream_window_data_threshold can be used to control how little
the DATA frames that Cowboy sends can get.
For CVE-2019-9516, also known as 0-Length Headers Leak, Cowboy
will now simply reject streams containing 0-length header names.
For CVE-2019-9517, also known as Internal Data Buffering, the
backpressure changes were already pretty good at preventing this
issue, but a new option max_connection_buffer_size was added for
even better control over how much memory we are willing to allocate.
For CVE-2019-9512, also known as Ping Flood; CVE-2019-9515, also
known as Settings Flood; CVE-2019-9518, also known as Empty Frame
Flooding; and similar undocumented scenarios, a frame rate limiting
mechanism was added. By default Cowboy will now allow 1000 frames
every 10 seconds. This can be configured via max_received_frame_rate.
For CVE-2019-9514, also known as Reset Flood, another rate limiting
mechanism was added and can be configured via max_reset_stream_rate.
By default Cowboy will do up to 10 stream resets every 10 seconds.
Finally, nothing was done for CVE-2019-9513, also known as Resource
Loop, because Cowboy does not currently implement the HTTP/2
priority mechanism (in parts because these issues were well known
from the start).
Tests were added for all cases except Internal Data Buffering,
which I'm not sure how to test, and Resource Loop, which is not
currently relevant.
This should limit the amount of memory that Cowboy is using
when a handler is sending data much faster than the network.
The new max_stream_buffer_size is a soft limit and only has
an effect when the cowboy_stream_h handler is used.
As a result we explictly reject path_info components that include
a forward slash, backward slash or NUL character. This only applies
to the [...] part of the path for dir/priv_dir configuration.
Also improve the tests so that they work on Windows.
