src: Do not busy-loop if a message exceeds the buffer size

If the length of a message exceeds the maximum length of the buffer, the buffer size will reach its maximum value and stay there forever, with no message ever being successfully processed. Since libwayland uses level-triggered epoll, this will cause the compositor to loop forever and consume CPU time. In libwayland 1.22 and below, there was an explicit check that caused messages exceeding 4096 bytes to result in an EOVERFLOW error, preventing the loop. However, this check was removed between d074d529 ("connection: Dynamically resize connection buffers").

To prevent this problem, always limit the size of messages to 4096 bytes. Since the default and minimum buffer size is 4096 bytes, this ensures that a single message will always fit in the buffer. It would be possible to allow larger messages if the buffer size was larger, but the maximum size of a message should not depend on the buffer size chosen by the compositor.

Rejecting messages that exceed 4092 bytes seems to have the advantage of reserving 4 bits, not 3, in the size field for future use. However, 0x0 is not a valid message size, so one can obtain a fourth bit by negating the meaning of bit 12 if bits 0 through 11 (inclusive) are 0. Allowing 4096-byte messages provides the far more important advantage that regressions compared to 1.22 are impossible and regressions compared to 1.23 are extremely unlikely. The only case where a regression is possible is:

• The receiving side is using libwayland 1.23.
• The sending side is either using libwayland 1.23 or is not using libwayland.
• The sender sends a message exceeding 4096 bytes.
• If the sender of the large message is the client, the server has increased the buffer size from the default value.

This combination is considered extremely unlikely, as libwayland 1.22 and below would disconnect upon receiving such a large message. 4096-byte messages, however, have always worked, so there was no reason to avoid sending them.

This is a normative protocol change. To ensure ecosystem-wide consistency, it is expected that other implementations of the protocol will also impose the same limit on message sizes.

A test is included that demonstrates the problem. If one runs the test case without the bug fix, the server spins and never processes the message. The test eventually times out. With the fix, a protocol error is delivered to the client and the test finishes immediately. The test requires a slight change to the I/O code so that if recvmsg() returns -1 with errno set to ECONNRESET ("connection reset by peer"), this is treated as EOF rather than as an immediately fatal error. Making this change allows the WL_DISPLAY_ERROR_INVALID_METHOD error to be received and set the display's last error to EINVAL.

As a bonus, send a more descriptive error for messages less than 8 bytes.

Signed-off-by: Demi Marie Obenour demi@invisiblethingslab.com
Fixes: d074d529 ("connection: Dynamically resize connection buffers").