Copyright © 1999–2018 FastMail Pty Ltd
Over the last 24 hours, we’ve rolled out a new POP/IMAP server version for all users. This new server is the result of months of great work by Bron and includes many improvements and fixes. Not that many of the fixes are currently user visible changes, but they are significant internal improvements that help improve reliability, conformance and performance, and will allow us to build some future features we’re looking at.
Email replication improvements
Email replication has been made much more efficient and reliable. The format includes CRC auto-integrity checking features, so that any unexpected mismatches between both ends are automatically detected and fixed. It can also recover automatically from unclean shutdowns or machine crashes where “split brain” has occurred, automatically fixing up mailboxes and messages. The format has also been made future extensible, allowing more features to be added without compatibility problems.
Performance and integrity improvements
The internal mailbox format used to store emails has been significantly reworked. The new format has reliable locking semantics to remove all race conditions. It also stores and checks CRCs on all record data and cache data, and SHA1 checks on all message files. This ensures that any corruption in any data is detected early and can be dealt with. By moving around some of the data (such as the user seen state), and only lazily opening files as needed, the new format also improves performance in many common cases.
Strict MODSEQ, QRESYNC support and full IMAP test suite conformance
Recent extensions to IMAP allow clients to more quickly synchronise data between the server and the client (eg. CONDSTORE/MODSEQ and QRESYNC). While the server has supported CONDSTORE/MODSEQ for a while, unfortunately it was a bit buggy in some situations, causing message seen state to get out of sync. The server now correctly and accurately support CONDSTORE/MODSEQ, and also supports the current QRESYNC standard that will allow clients that support it to sync even faster. We also now correctly pass detailed IMAP stress tests.
Major code cleanups
All of these improvements have also been done with major internal code cleanups. This will allow us to continue building additional functionality and features more easily in the future, and to more easily fix and debug any other issues that are encountered.
Unfortunately no good deed goes unpunished, and even though we’ve been testing this code ourselves and on a sub-set of users for weeks with continuous improvements, unfortunately some bugs did get through when we finally rolled out to all users. Then in the attempt to fix these issues as quickly as possible, we also introduced some other issues. The net result was that for about 12 hours, there was a sequence of small but potentially annoying bugs that would have affected different sets of users.
- On first access, we upgrade a mailbox to the new format. During the upgrade, we found some existing caches had allowed invalid data to enter them, causing corruptions on upgrading which caused problems when accessing these mailboxes. These cases are now caught and new cache data is built from the underlying message files
- While reconstructing the mailboxes that had been incorrectly upgraded by the above code, a quota error caused some peoples quota to temporarily be double their actual used amount. This has been fixed now. If this bug sent a user over quota temporarily, it shouldn't be a problem. When a user is over quota, we return a temporary 4xx error, which means no messages should have been lost, the other side should just have re-delivered when they were back under quota.
- IMAP IDLE wasn't returning new messages, only updating existing messages, causing pushing of new messages to most email clients to not work
- Mail App has a bug with parsing IMAP IDLE unsolicited fetch responses that contain more than flags information. We've added a workaround for this Mail App bug
- The IMAP COPYUID response was producing a non-conformant result, which caused some programs to report an error (Outlook 2010)
- POP3 was using an optimised mode if a mailbox was empty. Unfortunately the code to mark a mailbox as "non empty" wasn't working properly when messages were delivered, but was working for IMAP logins. This meant that messages delivered wouldn't be downloaded by POP until you did an IMAP or web login
- The POP3 TOP command wasn't working, causing some programs (Outlook in POP mode) that download email headers to fail
- The POP3 UIDL command with a message ID was producing a non-conformant result, which was parsed incorrectly by some programs. This caused some POP programs to download the same message more than once, or to delete off the server before it should have
- Update: An update to UID sequence handling caused the mailbox status command to report unread messages as read and vice-versa, causing the unread count on folders to actually be the read count for a short while.
- Update: The XLIST extension wasn't working. This has been added back, so client that support it will automatically pick the right Sent Items, Drafts, Trash, Junk Mail folders when setting up a new account
- Update: NOOP on Mac Mail. Like the bug above with Mac Mail and IDLE, this was affecting the NOOP command as well
- Update: Storing the \Seen flag + another flag on a message that already had the \Seen flag would cause \Seen to actually disappear. This mostly manifested as when deleting a message, it would cause it to become marked as "unread" again
All these issues have now been fixed, and we’re closely monitoring all the server logs to see if there’s any other issues, but at this stage we believe that the new server and code is working correctly for all cases we’re aware of and for all clients, IMAP and POP.
All this new code is part of the open source project cyrus, and we’ll be pushing this code back to the main cyrus code base, which will eventually form the basis for a new cyrus version 2.4. For those interested in technical details, Bron will post to the cyrus mailing lists when he’s had a bit of time to compile all the documentation and technical details.