Zulip Server 4.9 security release
We released Zulip Server 4.9 today! This is a security release, containing critical security fixes, as well as important cherry-picked bug fixes, since Zulip Server 4.8.
We strongly recommend that all installations upgrade to this new release. See the upgrade instructions in the Zulip documentation. If you need help, best-effort support is available on chat.zulip.org.
Reminder: upgrade your host operating system
As a reminder, we have deprecated support for Ubuntu 18.04 Bionic, because the upstream vendor no longer provides security support for important Zulip dependencies. Specifically, Ubuntu 18.04 remains supported in this release and will be supported in the rest of the Zulip Server 4.x series, but will not be supported in the upcoming Zulip Server 5.0 release.
We recommend planning to upgrade any Zulip servers you manage which are running on Ubuntu 18.04 over the next few months. See our OS upgrade documentation for how to correctly upgrade a Zulip server.
- CVE-2021-43799: Remote execution of code involving RabbitMQ. See below for more details.
- Closed access to RabbitMQ port 25672; initial installs tried to close this port, but failed to restart RabbitMQ for the configuration.
- Removed the
zulip.conf; all RabbitMQ instances will be reconfigured to have a nodename of
zulip@localhost. You can remove this setting from your
zulip.confconfiguration file, if it exists.
- Added missing support for the Camo image proxy in the Docker image. This resolves a longstanding issue with image previews, if enabled, appearing as broken images for Docker-based installs.
- Fixed a bug which allowed a user to edit a message to add a wildcard mention when they did not have permissions to send such messages originally.
- Fixed a bug in the tool that corrects database corruption caused by updating the operating system hosting PostgreSQL, which previously omitted some indexes from its verification. If you updated the operating system of your Zulip instance from Ubuntu 18.04 to 20.04, or from Debian Stretch to Debian Buster, you should run the tool, even if you did so previously; full details and instructions are available in the previous blog post.
- Began routing requests from the Camo image proxy through a non-Smokescreen proxy, if one is configured; because Camo includes logic to deny access to private subnets, routing its requests through Smokescreen is generally not necessary.
- Fixed a bug where changing the Camo secret required running
scripts/setup/compare-settings-to-templateto be able to run from any directory.
- Switched Let’s Encrypt renewal to use its own timer, rather than our custom
cron job. This fixes a bug where occasionally
nginxwould not reload after getting an updated certificate.
- Updated documentation and tooling to note that installs using
upgrade-zulip-from-gitrequire 3 GB of RAM, or 2 GB and at least 1 GB of swap.
CVE-2021-43799: Remote code execution vulnerability involving RabbitMQ
This release fixes a critical security vulnerability: CVE-2021-43799. Due to a combination of bugs (detailed below), some Zulip servers that have not been rebooted since installation have an insecure RabbitMQ configuration that can be exploited for remote execution of arbitrary code. This issue affects all previously released versions of Zulip.
The underlying vulnerability also affects any Debian or Ubuntu system that has
rabbitmq-server package. Unless the issue is separately blocked
by a firewall, just
apt install rabbitmq-server is sufficient to make your
system vulnerable to remote code execution due to this issue.
For context, RabbitMQ is a clustered database optimized around managing queues, written in Erlang. Zulip uses RabbitMQ to send data to other parts of the application.
The combination of bugs is as follows:
- RabbitMQ opens port 25672 as a
used for inter-node communication in RabbitMQ clusters. Port 25672 is secured
via an Erlang
however, Erlang’s default scheme for generating this cookie uses an insecure
randomizer, and can be easily brute-forced by a remote process that can access
that port. Neither the Debian/Ubuntu packages for
rabbitmq-servernor Zulip’s own configuration overrode that default randomizer. Successful access to the distribution port allows arbitrary execution of code, by design.
- Zulip’s configuration was intended to configure RabbitMQ to only listen on
localhostfor port 25672. However, a subtle bug in Zulip’s configuration resulted in some Zulip installations incorrectly having port 25672 listening on all interfaces until the service or server was restarted.
Together, these issues mean that a newly installed Zulip server is vulnerable to
a brute force attack leading to arbitrary code execution as the
system user. This system user only has direct access to RabbitMQ traffic (which
includes the content of all new Zulip messages sent on the server). But any
system user access on a server can potentially be combined with any unpatched
privilege escalation vulnerabilities in the host operating system for greater
Mitigation and patching
This vulnerability is mitigated by restarting RabbitMQ with
service rabbitmq-server restart, which will ensure Zulip’s existing
configuration to limit access to port 25672 is in effect.
The Zulip upgrade tool patches CVE-2021-43799 by regenerating the Erlang magic cookie using a secure algorithm, and fixing the bug that made it possible for port 25672 to remain open.
While we recommend installing this upgrade promptly, using a firewall to block port 25672 to your Zulip server is an effective way to mitigate risk (though doing so only prevents attacks coming from outside the firewall).
We also suggest that you use a firewall to block port 4369, the Erlang
“mapping demon” port, which is also accessible on all interfaces. This security
release does not close port 4369 down to
localhost, because multiple
overlapping bugs in Erlang and Ubuntu make it difficult to do so reliably. While
we are not aware of any vulnerabilities in the
epmd server listening on this
port, it is only of use in RabbitMQ clustering situations, and there is never
any reason to expose it to the public Internet.
Zulip development environments
Zulip development environments that are not using the default
configuration may also be vulnerable to this attack, depending on whether they
have a firewall limiting access to port 25672. Zulip development environments
should re-provision using
./tools/provision to generate a secure “magic
cookie,” mitigating the vulnerability.
CVE-2021-43799 was discovered by the Zulip security team. However, the underlying issues with Erlang’s “magic cookie” randomizer have been documented for years with publicly-available exploit code, so it is highly likely that threat actors are already actively exploiting the broader issue with RabbitMQ, as well as other services written in Erlang.
Previously, CVE-2018-1279 was filed for a similar vulnerability in RabbitMQ, but was overly-specific about the affected software, and understated the impact of the vulnerability. That CVE was noted by Debian, but its scope was misunderstood, and it was mistakenly marked “fixed” by an unrelated change in a different package.
We were very disappointed to discover that this important security issue with
Erlang and RabbitMQ has been public for years, and not addressed by upstream
projects before now. As part of preparing this release, we reported to both
Debian and Ubuntu that
apt install rabbitmq-server opens servers to remote
execution of code, with the goal of doing a coordinated public disclosure around
Edit: Debian published
rabbitmq-server 3.9.8-6 release to Debian unstable
in response to our report. This change has not yet been backported to stable
releases, and as of this writing, neither Debian nor Ubuntu has told us they
plan to issue a security advisory. For non-Zulip installs, upgrading to the
3.9.8-6 Debian release of
rabbitmq-server will correct existing vulnerable
Debian/Ubuntu installations with insecure magic cookies, as well as improve
cookie generation for new installations.
Our postmortem plans in response to this incident include accelerating the timeline on existing plans to replace RabbitMQ with a different tool for managing queues.
We love feedback from the Zulip user community. Here are a few ways you can connect: