Dec 3: Push it real good

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on December 2nd was an intro to our approach to security. The next post on December 4th is about our IMAP server hardware.

Technical level: lots

Bron (one of our team, you’ve heard of him) does this great demo when he shows FastMail to other people. He opens his inbox up on the screen, and gets someone to send him an email. A nice animation slides the new message into view on the screen. At the same time, his phone plays a sound. He uses his watch to delete the email, and it disappears from the screen too. It’s a huge “wow” moment, made possible by our push notification system. Today we’ll talk about exactly how we let you know when something interesting happens in your mailbox.

Cyrus has two mechanisms for telling the world that something has changed, idled and mboxevent. idled is the simpler of the two. When your mail client issues an IDLE command, it is saying “put me to sleep and tell me when something changes”. idled is the server component that manages this, holding the connection open and sending a response when something changes. An example protocol exchange looks like something like this (taken from RFC 2177, the relevant protocol spec):

S: * FLAGS (Deleted Seen)
S: A001 OK SELECT completed
C: A002 IDLE
S: + idling
...time passes; new mail arrives...
S: A002 OK IDLE terminated

It’s a fairly simple mechanism, only designed for use with IMAP. We’ll say no more about it.

Of far more interest is Cyrus’ “mboxevent” mechanism, which is based in part on RFC 5423. Cyrus can be configured to send events to another program any time something changes in a mailbox. The event contains details about the type of action that occurred, identifying information about the message and other useful information. Cyrus generates events for pretty much everything – every user action, data change, and other interesting things like calendar alarms. For example, here’s a delivery event for a system notification message I received a few minutes ago:

 "event" : "MessageNew",
 "messages" : 1068,
 "modseq" : 1777287, 
 "pid" : 2087223,
 "serverFQDN" : "sloti30t01",
 "service" : "lmtp",
 "uidnext" : 40818,
 "uri" : "imap://;UIDVALIDITY=1335827579/;UID=40817",
 "vnd.cmu.envelope" : "(\"Wed, 03 Dec 2014 08:49:40 +1100\" \"Re: Blog day 3: Push it real good\" ((\"Robert Norris\" NIL \"robn\" \"\")) ((\"Robert Norris\" NIL \"robn\" \"\")) ((\"Robert Norris\" NIL \"robn\" \"\")) ((NIL NIL \"staff\" \"\")) NIL NIL \"<>\" \"<>\")",
 "vnd.cmu.mailboxACL" : "\tlrswipkxtecdn\tadmin\tlrswipkxtecdan\tanyone\tp\t",
 "vnd.cmu.mbtype" : "",
 "vnd.cmu.unseenMessages" : 90,
 "vnd.fastmail.cid" : "b9384d25e93fc71c",
 "vnd.fastmail.convExists" : 413,
 "vnd.fastmail.convUnseen" : 82,
 "vnd.fastmail.counters" : "0 1777287 1777287 1761500 1758760 1416223082",
 "vnd.fastmail.sessionId" : "sloti30t01-2087223-1417556981-1"

The event contains all sorts of information: the action that happened, the message involved, how many messages I have, the count of unread messages and conversations, the folder involved, stuff about the message headers, and more. This information isn’t particularly useful in its raw form, but we can use it to do all kinds of interesting things.

We have a pair of programs that do various processing on the events that Cyrus produce. They are called “pusher” and “notifyd”, and run on every server where Cyrus runs. Their names don’t quite convey their purpose as they’ve grown up over time into their current forms.

pusher is the program that receives all events coming from Cyrus. It actions many events itself, but only the ones it can handle “fast”. All other events are handed off to notifyd, which handles the “slow” events. The line between the two is a little fuzzy, but the rule of thumb is that if an event needs to access the central database or send mail, it’s “slow” and should be handled by notifyd. Sending stuff to open connections or making simple HTTP calls are “fast”, and are handled in pusher. I’ve got scare quotes around “slow” and “fast” because the slow events aren’t actually slow. It’s more about how well each program responds when flooded with events (for programmers: pusher is non-blocking, notifyd can block).

We’ll talk about notifyd first, because it’s the simpler of the two. The two main styles of events it handles are calendar email notifications and Sieve (filter/rule) notifications.

Calendar email notifications are what you get when you say, for example, “10 minutes before this event, send me an email”. All the information that is needed to generate an email is placed into the event that comes from Cyrus, including the name of the event, the start and end time, the attendees, location, and so on. notifyd constructs an email and sends it to the recipient. It does database work to look up the recipient’s language settings to try and localise the email it sends. Here we see database and email work, and so it goes on the slow path.

The Sieve filter system we use has a notification mechanism (also see RFC 5435) where you can write rules that cause new emails or SMS messages to be sent. notifyd handles these too, taking the appropriate action on the email (summarise, squeeze) and then sending an email or posting to a SMS provider. For SMS, it needs to fetch the recipient’s number and SMS credit from the database so again, it’s on the slow path.

On to pusher, which is where the really interesting stuff happens. The two main outputs it has are the EventSource facility used by the web client, and the device push support used by the Android and iOS apps.

EventSource is a facility available in most modern web browsers that allow it to create a long-lived connection to a server and receive a constant stream of events. The events we send are very simple. For the above event, pusher would send the following to all web sessions I currently have open:

event: push
id: 1760138
data: {"mailModSeq":1777287,"calendarModSeq":1761500,"contactsModSeq":1758760}

These are the various “modseq” (modification sequence, see also RFC 7162) numbers for mail, calendar and contacts portions of my mail store. The basic idea behind a modseq is that every time something changes, the modseq number goes up. If the client sees the number change, it knows that it needs to request an update from the server. By sending the old modseq number in this request, it receives only the changes that have happened since then, making this a very efficient operation.

If you’re interested, we’ve written about our how we use EventSource in a lot more detail in this blog post from a couple of years ago. Some of the details have changed since then, but the ideas are still the same.

The other thing that pusher handles is pushing updates to the mobile apps. The basic idea is the same. When you log in to one of the apps, they obtain a device token from the device’s push service (Apple Push Notification Service (APNS) for iOS or Google Cloud Messaging (GCM) for Android), and then make a special call to our servers to register that token with pusher. When the inbox changes in some way, a push event is created and sent along with the device token to the push service (Apple’s or Google’s, depending on the token type).

On iOS, a new message event must contain the actual text that is displayed in the notification panel, so pusher extracts that information from the “vnd.cmu.envelope” parameter in the event it received from Cyrus. It also includes an unread count, which is used to update the red “badge” on the app icon, and a URL which is passed to the app when the notification is tapped. An example APNS push event might look like:

  "aps" : {
    "alert" : "Robert Norris\nHoliday pics",
    "badge" : 82,
    "sound" : "default"
  "url" : "?u=12345678#/mail/Inbox/eb26398990c4b29b-f45463209u40683

For other inbox changes, like deleting a message, we send a much simpler event to update the badge:

  "aps" : {
    "badge" : 81

The Android push operates a little differently. On Android it’s possible to have a service running in the background. So instead of sending message details in the push, we only send the user ID.

  "data" : {
    "uparam" : "12345678"

(The Android app actually only uses the user ID to avoid a particular bug, but it will be useful in the future when we support multiple accounts).

On receiving this, the background service makes a server call to get any new messages since the last time it checked. It’s not unlike what the web client does, but simpler. If it finds new messages, it constructs a system notification and displays it in the notification panel. If it sees a message has been deleted and it currently has it visible in the notification, it removes it. It also adds a couple of buttons (archive and delete) which result in server actions being taken.

So that’s all the individual moving parts. If you put them all together, then you get some really impressive results. When Bron uses the “delete” notification action on his watch (an extension of the phone notification system), it causes the app to send a delete instruction the the server. Cyrus deletes the message and sends a “MessageDelete” event to pusher. pusher sends a modseq update via EventSource to the web clients which respond by requesting an update from the server, noting the message is deleted and removing it from the message list. pusher also notices this is an inbox-related change, so sends a new push to any registered Android devices and, because it’s not a “MessageNew” event, sends a badge update to registered iOS devices.

One of the things I find most interesting about all of this is that in a lot of ways it wasn’t actually planned, but has evolved over time. notifyd is over ten years old and existed just to support Sieve notifications. Then pusher came along when we started the current web client and it needed live updates. Calendar notifications came later and most recently, device push. It’s really nice having an easy, obvious place to work with mailbox events. I fully expect that we’ll extend these tools further in the future to support new kinds of realtime updates and notifications.

Posted in Advent 2014, Technical. Comments Off on Dec 3: Push it real good

Dec 1: Email Search System

This blog post is part of the FastMail 2014 Advent Calendar.

The next post on December 2nd is Security – Confidentiality, Integrity and Availability.

Technical level: medium

Our email search system was originally written by Greg Banks, who has moved on to a role at Linked In, so I maintain it now. It’s a custom extension to the Cyrus IMAPd mail server.

Fast search was a core required feature for our new web interface. My work account has over half a million emails in it, and even though our interface allows fast scroll, it’s still impossible to find anything more than a few weeks old without either knowing exactly when it was sent, or having a powerful search facility.

Greg tried a few different engines, and settled on the Xapian project as the best fit for the one-database-per-user that we wanted.

We tried indexing new emails as they arrived, even directly to fast SSDs, and discovered that the load was just too high. Our servers were overloaded trying to index in time – because adding a single email causes a lot of updates.

Luckily, Xapian supports searching from multiple databases at once, so we came up with the idea of a tiered database structure.

New messages get indexed to tmpfs in a small database. A job runs every hour to see if tmpfs is getting too full (over 50% of the defined size), in which case it compacts immediately, otherwise we automatically compact during the quiet time of the day. Compacted databases are more efficient, but read only.

This allows us to index all email immediately, and return a message that arrived just a second ago in your search results complete with highlighted search terms, yet not overload the servers. It also means that search data can be stored on inexpensive disks, keeping the costs of our accounts down.

Technical level: extreme

Here’s some very technical information about how the tiers are implemented, and an example of running the compaction.

We have 4 tiers at FastMail, though we don’t actually use the ‘meta’ one (SSD) at the moment:

  • temp
  • meta
  • data
  • archive

The temp level is on tmpfs, purely in memory. Meta is on SSD, but we don’t use that except during shutdown. Data is the main version, and we re-compact all the data level indexes once per week. Finally archive is never automatically updated, but we build it when users are moved or renamed, or can create it manually.

Both external locking (Xapian isn’t always happy with multiple writers on one database) and the compaction logic are managed via a separate file called xapianactive. The xapianactive looks like this:

% cat /mnt/ssd30/sloti30t01/store23/conf/user/b/brong.xapianactive
temp:264 archive:2 data:37

The first item in the active file is always the writable index – all the others are read-only.

These map to paths on disk according to the config file:

% grep search /etc/cyrus/imapd-sloti30t01.conf
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(the ‘default tier’ is to tell the system where to create a new search item)

So based on these paths, we find.

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
3328 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.264
1520432 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.37
3365336 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2

I haven’t compacted to archive for a while. Let’s watch one of those. I’m selecting all the tiers, and compressing to a single tier. The process is as follows:

  1. take an exclusive lock on the xapianactive file
  2. insert a new default tier database on the front (in this example it will be temp:265) and unlock xapianactive again
  3. start compacting all the selected databases to a single database on the given tier
  4. take an exclusive lock on the xapianactive file again
  5. if the xapianactive file has changed, discard all our work (we lock against this, but it’s a sanity check) and exit
  6. replace all the source databases for the compact with a reference to the destination database and unlock xapianactive again
  7. delete all now-unused databases

Note that the xapianactive file is only locked for two VERY SHORT times. All the rest of the time, the compact runs in parallel, and both searching on the read-only source databases and indexing to the new temp database can continue.

This allows us to only ever have a single thread compacting to disk, so our search drives are mostly idle, and able to serve
customer search requests very quickly.

When holding an exclusive xapianactive lock, it’s always safe to delete any databases which aren’t mentioned in the file – at worst you will race against another task which is also deleting the same databases, so this system is self-cleaning after any failures.

Here goes:

% time sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong
compressing temp:264,archive:2,data:37 to archive:3 for user.brong (active temp:264,archive:2,data:37)
adding new initial search location temp:265
compacting databases
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3.NEW
renaming tempdir into place
finished compact of user.brong (active temp:265,archive:3)

real 4m52.285s
user 2m29.348s
sys 0m13.948s

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
368 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.265
du: cannot access `/mnt/i30search/sloti30t01/store23/search/b/user/brong/*': No such file or directory
4614368 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3

If you want to look at the code, it’s all open source. I push the fastmail branch to github regularly. The xapianactive code is in imap/search_xapian.c and the C++ wrapper in imap/xapian_wrap.cpp.

Posted in Advent 2014, Technical. Comments Off on Dec 1: Email Search System

Updating our SSL certificates to SHA-256

This is a technical post. The important points to take away are that if, like most of our customers, you’re using FastMail’s web client with a modern, regularly updated browser like Chrome, Firefox, Internet Explorer or Safari, then everything will be fine. If you’re using an old browser or operating system (including long-unsupported mobile devices like old Nokia or WebOS devices), it may start failing to connect to FastMail during December, and you’ll need to make changes to the settings you use to access FastMail. Read on for details.

For many years the standard algorithm used to sign SSL certificates has been SHA-1. Recently, weaknesses have been exposed in that algorithm which make it unsuitable for encryption work. It’s not broken yet, but it’s reasonable to expect that it will be broken within the next year or two.

A replacement algorithm is available, called SHA-256 (sometimes called SHA-2), and its been the recommended algorithm for new certificates for the last couple of years.

Back in April, we updated our certificates with new ones that used SHA-256. This caused problems for certain older clients that didn’t have support for SHA-256. After some investigation, we reverted to SHA-1 certificates.

Recently Google announced that they would start deprecating SHA-1 support this year. Chrome 40 (currently in testing, due for release in January) will start showing the padlock icon on as “secure, with minor errors”. Crucially, it will no longer display the green “EV” badge.

As a result, we are intending to update our certificates to SHA-256 during December. Its something we wanted to do back in April anyway, as we’d much prefer to proactively support modern security best practice rather than scramble frantically to fix things when breaches are discovered.

Unfortunately, this will cause problems for customers using older browsers. Most desktop browsers should not have any problem, though Windows XP users will need to update to Service Pack 3. Many more obscure devices (notably Nokia and WebOS devices) do not support SHA-256 at all, and will not be able to connect to us securely.

We will be attempting to support a SHA-1 certificate on and, but only if our certificate authority will agree to issue one to us. Once we have that information I’ll update this post.

If you have any questions about this change, please contact support.

Further reading:

Posted in Technical. Comments Off on Updating our SSL certificates to SHA-256 now redirects to

In preparation for our our move to, we’ll be doing some testing on So if you use the beta server, expect some changes and potential issues over the next few days.

Currently that means if you go to, you’ll immediately be redirected to This is expected. Note that you can’t currently create aliases or rename your account to This is expected. This will only be available from Thursday as described in the original blog post.

Posted in Technical. Comments Off on now redirects to

SSL 3.0 disabled due to security vulnerability

This morning Google published news of a new vulnerability in SSL 3.0. You can read more about it in the original announcement and in CloudFlare’s analysis of the problem.

This is a serious issue that can leak user data. Unfortunately there’s no workaround – the only option we have is to disable SSL 3.0 on our servers entirely. We don’t like having to do this because we want our users to be able to use any client they choose to access their mail, but when there’s a security hole and no way to plug it we have no choice but to break things for some people in order to protect everyone.

Happily, this should not affect the majority of our users. The only significant browser to be affected is Internet Explorer 6 on Windows XP, which will now not be able to connect to at all. Similar changes have been made to our IMAP, POP and other backend services, so you may also have connection issues with older mail clients.

If you are unable or unwilling to upgrade your client software at this time, you can use (web) and (IMAP/POP/SMTP), both of which support SSL 3.0. As always, we highly discourage the use of these service names because they leave your data open to attack, and we may remove them in the future.

Update 16 Oct 2014 01:00 UTC: We’ve heard of at least two mail clients (Airmail and Windows Phone) that can receive but not send mail. Changing the outgoing settings to use port 587 instead of 465 has resolved the problem for some users. If you’re seeing similar problems, give that a try.

Posted in Technical. Comments Off on SSL 3.0 disabled due to security vulnerability

Payment Issues

Recently there have been some problems with our payment processor.

One of these is that sometimes there is a significant delay in completing a transaction, so the funds are “reserved” but the transaction isn’t actually completed until a few weeks later.

The other problem was that a number of charges were made to some of our customers’ credit cards, in our name. The majority of these are USD 1.00 test charges that are used to verify that a card is valid. Normally we cancel these charges immediately and the charge doesn’t appear on the card statement. However the problem at our payment processor has resulted in some of these supposedly “cancelled” charges recently being applied to customer cards. A number of other charges which were supposed to be “cancelled” have also been processed. This was done in error and without any instructions from FastMail.

We are currently working with our payment processor to refund these erroneous payments. In some cases, the refunds are unable to be processed directly because the card has expired or been cancelled.

We may need to refund these charges via an alternative method, or, at the user’s option, instead credit the amount towards future FastMail renewals. We will be in touch with the affected customers once we have more information on the scope of the problem.

We would like to assure our users that we will make certain that all erroneous charges are corrected. This may take some time, so thanks for your patience.

Our apologies for any inconvenience or confusion. We will post more information as it becomes available.
– the FastMail team

Posted in Technical. Comments Off on Payment Issues

New phishing trick, data: URLs to avoid forgery reporting

This is a technical post about a new and interesting phishing technique seen today. Regular FastMail users can skip this post.

We saw an interesting new phishing attempt today that uses a relatively novel technique to try and hide the source of the attack and avoid it being reported as a web forgery.

Firstly the email itself looks reasonably well done (apart from the year in the subject being completely wrong), certainly it’s not the poor quality you often see. It looked like this (ANZ is an Australian bank):


Secondly, the email was sent using a compromised gmail account with a .edu address. In fact there were two separate emails, both from different compromised gmail .edu accounts. I imagine compromised gmail .edu accounts aren’t that easy to get, and this significantly reduced the chances of it being caught by any spam filter.

Thirdly, the phishing page itself is interesting in that it:

  1. Uses a standard link shortener for a redirect ( in this case)
  2. Which redirects to the phishing delivery page (a compromised page on
  3. That page however rather than hosting the HTML phishing login page directly, does this:

<script type="text/javascript">
        window.location="data:text/html;base64,... base64 encoded version of HTML phishing login page ...";

That data: URL is itself the phishing page content, which includes links to real ANZ website logos to make it look as authentic as possible, but has a form submit action to a compromised page on

This approach is interesting because it makes it impossible to report this page as a forgery using the standard Firefox "Report Web Forgery" action because Firefox thinks it’s a data: URL. Neat trick that makes it harder to remove or block in the long run.

I’ve reported this issue as a Firefox bug:

Posted in Technical. Comments Off on New phishing trick, data: URLs to avoid forgery reporting

Get every new post delivered to your Inbox.

Join 7,293 other followers

%d bloggers like this: