Over the last 24 hours, we’ve rolled out a new POP/IMAP server version for all users. This new server is the result of months of great work by Bron and includes many improvements and fixes. Not that many of the fixes are currently user visible changes, but they are significant internal improvements that help improve reliability, conformance and performance, and will allow us to build some future features we’re looking at.
Email replication improvements
Email replication has been made much more efficient and reliable. The format includes CRC auto-integrity checking features, so that any unexpected mismatches between both ends are automatically detected and fixed. It can also recover automatically from unclean shutdowns or machine crashes where “split brain” has occurred, automatically fixing up mailboxes and messages. The format has also been made future extensible, allowing more features to be added without compatibility problems.
Performance and integrity improvements
The internal mailbox format used to store emails has been significantly reworked. The new format has reliable locking semantics to remove all race conditions. It also stores and checks CRCs on all record data and cache data, and SHA1 checks on all message files. This ensures that any corruption in any data is detected early and can be dealt with. By moving around some of the data (such as the user seen state), and only lazily opening files as needed, the new format also improves performance in many common cases.
Strict MODSEQ, QRESYNC support and full IMAP test suite conformance
Recent extensions to IMAP allow clients to more quickly synchronise data between the server and the client (eg. CONDSTORE/MODSEQ and QRESYNC). While the server has supported CONDSTORE/MODSEQ for a while, unfortunately it was a bit buggy in some situations, causing message seen state to get out of sync. The server now correctly and accurately support CONDSTORE/MODSEQ, and also supports the current QRESYNC standard that will allow clients that support it to sync even faster. We also now correctly pass detailed IMAP stress tests.
Major code cleanups
All of these improvements have also been done with major internal code cleanups. This will allow us to continue building additional functionality and features more easily in the future, and to more easily fix and debug any other issues that are encountered.
Unfortunately no good deed goes unpunished, and even though we’ve been testing this code ourselves and on a sub-set of users for weeks with continuous improvements, unfortunately some bugs did get through when we finally rolled out to all users. Then in the attempt to fix these issues as quickly as possible, we also introduced some other issues. The net result was that for about 12 hours, there was a sequence of small but potentially annoying bugs that would have affected different sets of users.
All these issues have now been fixed, and we’re closely monitoring all the server logs to see if there’s any other issues, but at this stage we believe that the new server and code is working correctly for all cases we’re aware of and for all clients, IMAP and POP.
All this new code is part of the open source project cyrus, and we’ll be pushing this code back to the main cyrus code base, which will eventually form the basis for a new cyrus version 2.4. For those interested in technical details, Bron will post to the cyrus mailing lists when he’s had a bit of time to compile all the documentation and technical details.