We released Zulip Server 4.9 today! This is a security release, containing critical security fixes, as well as important cherry-picked bug fixes, since Zulip Server 4.8.

Upgrading

We strongly recommend that all installations upgrade to this new release. See the upgrade instructions in the Zulip documentation. If you need help, best-effort support is available on chat.zulip.org.

Reminder: upgrade your host operating system

As a reminder, we have deprecated support for Ubuntu 18.04 Bionic, because the upstream vendor no longer provides security support for important Zulip dependencies. Specifically, Ubuntu 18.04 remains supported in this release and will be supported in the rest of the Zulip Server 4.x series, but will not be supported in the upcoming Zulip Server 5.0 release.

We recommend planning to upgrade any Zulip servers you manage which are running on Ubuntu 18.04 over the next few months. See our OS upgrade documentation for how to correctly upgrade a Zulip server.

Notable changes

• CVE-2021-43799: Remote execution of code involving RabbitMQ. See below for more details.
• Closed access to RabbitMQ port 25672; initial installs tried to close this port, but failed to restart RabbitMQ for the configuration.
• Removed the rabbitmq.nodename configuration in zulip.conf; all RabbitMQ instances will be reconfigured to have a nodename of zulip@localhost. You can remove this setting from your zulip.conf configuration file, if it exists.
• Added missing support for the Camo image proxy in the Docker image. This resolves a longstanding issue with image previews, if enabled, appearing as broken images for Docker-based installs.
• Fixed a bug which allowed a user to edit a message to add a wildcard mention when they did not have permissions to send such messages originally.
• Fixed a bug in the tool that corrects database corruption caused by updating the operating system hosting PostgreSQL, which previously omitted some indexes from its verification. If you updated the operating system of your Zulip instance from Ubuntu 18.04 to 20.04, or from Debian Stretch to Debian Buster, you should run the tool, even if you did so previously; full details and instructions are available in the previous blog post.
• Began routing requests from the Camo image proxy through a non-Smokescreen proxy, if one is configured; because Camo includes logic to deny access to private subnets, routing its requests through Smokescreen is generally not necessary.
• Fixed a bug where changing the Camo secret required running zulip-puppet-apply.
• Fixed scripts/setup/compare-settings-to-template to be able to run from any directory.
• Switched Let’s Encrypt renewal to use its own timer, rather than our custom cron job. This fixes a bug where occasionally nginx would not reload after getting an updated certificate.
• Updated documentation and tooling to note that installs using upgrade-zulip-from-git require 3 GB of RAM, or 2 GB and at least 1 GB of swap.

CVE-2021-43799: Remote code execution vulnerability involving RabbitMQ

This release fixes a critical security vulnerability: CVE-2021-43799. Due to a combination of bugs (detailed below), some Zulip servers that have not been rebooted since installation have an insecure RabbitMQ configuration that can be exploited for remote execution of arbitrary code. This issue affects all previously released versions of Zulip.

The underlying vulnerability also affects any Debian or Ubuntu system that has installed the rabbitmq-server package. Unless the issue is separately blocked by a firewall, just apt install rabbitmq-server is sufficient to make your system vulnerable to remote code execution due to this issue.

For context, RabbitMQ is a clustered database optimized around managing queues, written in Erlang. Zulip uses RabbitMQ to send data to other parts of the application.

The combination of bugs is as follows:

• RabbitMQ opens port 25672 as a “distribution port,” used for inter-node communication in RabbitMQ clusters. Port 25672 is secured via an Erlang “magic cookie;” however, Erlang’s default scheme for generating this cookie uses an insecure randomizer, and can be easily brute-forced by a remote process that can access that port. Neither the Debian/Ubuntu packages for rabbitmq-server nor Zulip’s own configuration overrode that default randomizer. Successful access to the distribution port allows arbitrary execution of code, by design.
• Zulip’s configuration was intended to configure RabbitMQ to only listen on localhost for port 25672. However, a subtle bug in Zulip’s configuration resulted in some Zulip installations incorrectly having port 25672 listening on all interfaces until the service or server was restarted.

Together, these issues mean that a newly installed Zulip server is vulnerable to a brute force attack leading to arbitrary code execution as the rabbitmq system user. This system user only has direct access to RabbitMQ traffic (which includes the content of all new Zulip messages sent on the server). But any system user access on a server can potentially be combined with any unpatched privilege escalation vulnerabilities in the host operating system for greater access.

Mitigation and patching

This vulnerability is mitigated by restarting RabbitMQ with service rabbitmq-server restart, which will ensure Zulip’s existing configuration to limit access to port 25672 is in effect.

The Zulip upgrade tool patches CVE-2021-43799 by regenerating the Erlang magic cookie using a secure algorithm, and fixing the bug that made it possible for port 25672 to remain open.

While we recommend installing this upgrade promptly, using a firewall to block port 25672 to your Zulip server is an effective way to mitigate risk (though doing so only prevents attacks coming from outside the firewall).

We also suggest that you use a firewall to block port 4369, the Erlang epmd “mapping demon” port, which is also accessible on all interfaces. This security release does not close port 4369 down to localhost, because multiple overlapping bugs in Erlang and Ubuntu make it difficult to do so reliably. While we are not aware of any vulnerabilities in the epmd server listening on this port, it is only of use in RabbitMQ clustering situations, and there is never any reason to expose it to the public Internet.

Zulip development environments
Zulip development environments that are not using the default vagrant configuration may also be vulnerable to this attack, depending on whether they have a firewall limiting access to port 25672. Zulip development environments should re-provision using ./tools/provision to generate a secure “magic cookie,” mitigating the vulnerability.

Concluding thoughts

CVE-2021-43799 was discovered by the Zulip security team. However, the underlying issues with Erlang’s “magic cookie” randomizer have been documented for years with publicly-available exploit code, so it is highly likely that threat actors are already actively exploiting the broader issue with RabbitMQ, as well as other services written in Erlang.

Previously, CVE-2018-1279 was filed for a similar vulnerability in RabbitMQ, but was overly-specific about the affected software, and understated the impact of the vulnerability. That CVE was noted by Debian, but its scope was misunderstood, and it was mistakenly marked “fixed” by an unrelated change in a different package.

We were very disappointed to discover that this important security issue with Erlang and RabbitMQ has been public for years, and not addressed by upstream projects before now. As part of preparing this release, we reported to both Debian and Ubuntu that apt install rabbitmq-server opens servers to remote execution of code, with the goal of doing a coordinated public disclosure around this issue.

Edit: Debian published the rabbitmq-server 3.9.8-6 release to Debian unstable in response to our report. This change has not yet been backported to stable releases, and as of this writing, neither Debian nor Ubuntu has told us they plan to issue a security advisory. For non-Zulip installs, upgrading to the 3.9.8-6 Debian release of rabbitmq-server will correct existing vulnerable Debian/Ubuntu installations with insecure magic cookies, as well as improve cookie generation for new installations.

Our postmortem plans in response to this incident include accelerating the timeline on existing plans to replace RabbitMQ with a different tool for managing queues.

Community

We love feedback from the Zulip user community. Here are a few ways you can connect:

• Join chat.zulip.org and provide feedback directly to the development community!
• Follow us on Twitter, or join our announcement mailing list (or subscribe to the blog posts, which are mostly a subset of the already low-volume mailing list).