Uploaded file data loss incident

We have discovered and fixed a bug in Zulip that, in a rare situation, caused uploaded attachments to be mistakenly deleted after 30 days. Specifically, if someone chose to delete a message, Zulip deleted all the files that were attached to that message. This behavior was incorrect for files that were also attached to other messages (e.g., a reply quoting the original message).

We’ve sent email to all administrators of the affected Zulip Cloud organizations, with details about the files which were affected. If you are an administrator of a Zulip Cloud organization and have not received an email from us, your organization was not affected. Overall, 0.87% of the organizations in Zulip Cloud had one or more files affected, and 0.05% of organizations (about 1 in 2,000) had 10 or more affected files.

Keeping user data safe is paramount, and in this instance we failed, for which we’re very sorry. We’re taking steps to help prevent any similar incident in the future. For more details on the bug, on what we’ve done to recover data where possible, and on the steps we’re taking for the future, read on.

Details of the bug

For context, Zulip supports uploading files and attaching them to messages. Each uploaded file can then also be linked to from any number of other Zulip messages. By design, Zulip deletes an uploaded file when all of the messages linking to it are deleted.

The bug that caused this incident was present in the code deployed to Zulip Cloud on May 11th, 2023 as part of the new scheduled message sending feature. As a result, when an uploaded file’s URL was used in more than one message, that file was marked for deletion when any of the messages pointing to it were deleted, instead of when all such messages were.

A file marked for deletion is permanently removed 30 days later (or after a custom time period configured by the administrator of a self-hosted server), in accordance with Zulip’s message retention policy. This results in a broken link, or missing image, in any messages that still contain links to the file.

For Zulip Cloud users

While Zulip Cloud keeps database backups on hand for 30 days, for user privacy reasons, we do not keep extra copies of user-uploaded files. To respect the fact that a user has requested for the data to be deleted, uploaded files are immediately and irrevocably deleted once the 30 day message retention period has passed. Since the bug caused some files to be incorrectly marked for permanent deletion, we cannot recover those files.

In a small number of cases where we had a recent data export of an organization, we were able to recover files that were contained in those data exports.

We have fixed the bug, and have worked to make Zulip more robust so that this type of incident does not recur in the future. Specifically, we have written a much more comprehensive test suite to cover corner cases like this in the data cleanup process.

For self-hosted Zulip server admins

The bug was introduced in Zulip Server 7.0. If you are running Zulip Server 7.0 or later, we recommend upgrading to Zulip Server 7.3, which we have just released, which fixes the bug. Organizations running Zulip Server 6.x (and earlier releases) were not affected.

For Zulip servers running the Git main branch, the bug was introduced in commit 414658fc8e689ea74f13965bbcd72e7beed00df5. Servers running that commit or later should be upgraded to the latest version of main to get the fix.

A temporary workaround is to run the command rm /etc/cron.d/archive-messages, which will disable deletion of archived messages until you next upgrade Zulip and get the fix.