Upgrading to Cyrus version 2.3.10

Cyrus version 2.3.10 is almost ready to be released.  As part of the preparations for release, we’ve been porting our Cyrus patches forwards and testing that it will all work correctly.

http://cyrus.brong.fastmail.fm/

The most exciting part of this new Cyrus release is full SHA1 GUID (unique identifiers) on each message.  This is a development which has grown out of our earlier work to use partial MD5 hashes as the UUID value in previous Cyrus releases instead of a locally calculated value, allowing better integrity checking across our replicas.  The SHA1 will provide much stronger cryptographic guarantees and simplify our backup infrastructure considerably as well (eventually – first there’s the changeover which will cause extra IO load on the backup server too – thankfully it has plenty of spare capacity, both disk and CPU)

We provide the totally non standard “FETCH GUID” command to allow this SHA1 value to be seen through IMAP as well (this is not in upstream Cyrus and is subject to change)

However – there is a downside.  The new index format (an additional 8 bytes per record) will cause every single mailbox to be “upgraded” when it is first accessed after the upgrade.  To ease the load on our servers, we will be phasing this upgrade in slowly, one store at a time.  As each store is upgraded, there will be a short downtime (approx 1 minute) followed by higher IO and hence slower access for all users until the indexes are upgraded.  The first time you access your mailbox after the upgrade will be particularly slow, as all your indexes get upgraded!

We will be running a background task trickling through and pre-upgrading mailboxes as well.

I’ll update you again, here and on the forum, when this process is finished.

Posted in Technical. Tags: , , , , . Comments Off

Emails and dates/times

There’s often some confusion with people regarding emails and the dates/times (datetime) shown on the email. The main problem is that there’s actually two main different datetimes associated with every email. There’s the sent datetime and the received datetime.

In many cases, these values are sufficiently close (a few seconds) that it doesn’t really matter which you see or use. However in some cases (eg delayed email, or email retrieved from another server to FastMail via a Pop Link) the datetimes can be significantly different, causing some confusion.

Mostly this revolves around that emails on the Mailbox screen show and are sorted by the received datetime, but if you view the headers of an email, the “Date:” field that most people see is actually the sent datetime. In fact historically, the received datetime is something that is created and stored internally in the email server itself when the email is delivered, it’s not visible as a field in the email headers at all.

Because the received datetime is only stored internally in the email server separate to the email itself, a number of things can cause this information to be lost, such as restoring emails from a backup, copying emails from one server to another, POP retrieving them from the server, etc.

To avoid the confusion this was causing, we decided to try and come up with a more consistent way of representing the internal date that would be stored with the email itself.

We decided to do this by noting that when an email is delivered to one of our email servers, the final hop in the email “Received:” headers shows this delivery step as something like this:

Received: from compute1.internal (compute1.internal [10.202.2.41])
     by store123m.internal (Cyrus v2.3.9) with LMTPA;
     Sat, 13 Oct 2007 04:27:43 -0400

The final part of that header shows the exact time the email was delivered to the mailbox (as per the definition of Trace fields in RFC2822). By using the time in the final “Received:” header, we basically have something that is constant in the email itself as something to reference when an email is moved, or restored from backup, or POPed from another server. This does have one potentially negative side-effect related to using IMAP to APPEND messages to a mailbox, but this is both a relatively rare case, and also we still believe the consistency of using the last “Received:” as the received datetime reference outweighs the potential negative.

At the same time we did this, we also wanted to allow users the option of when they used POP Links to retrieve emails from another server to use either the datetime the email was actually downloaded from the POP server as the received datetime, or use the “Date:” header on the email (eg sent datetime) as the received datetime. The way we decided to do this was by allowing a special “X-DeliveredInternalDate:” header. If that header is present, then it should be a datetime value in the standard RFC format, and that value will be used as the received datetime instead of looking at the datetime in the last “Received:” header. This provides flexibility in setting the received datetime, but at the same time it ensures that the received datetime won’t be lost by internal movements in and out of the server. The use of this is controlled on the Pop Links screen on each Pop Link by the “Received date” options which can be set to either “When POP occurs” or “Message sent date”.

This setup has been in place for about 6 months now and worked very well, giving a good balance of consistency and flexibility. The only issue that has really surfaced is that some programs use IMAP to APPEND emails with no “Received:” headers at all (eg Bynari Connector).

Posted in Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,556 other followers