Dec 1: Email Search System

This blog post is part of the FastMail 2014 Advent Calendar.

The next post on December 2nd is Security – Confidentiality, Integrity and Availability.

Technical level: medium

Our email search system was originally written by Greg Banks, who has moved on to a role at Linked In, so I maintain it now. It’s a custom extension to the Cyrus IMAPd mail server.

Fast search was a core required feature for our new web interface. My work account has over half a million emails in it, and even though our interface allows fast scroll, it’s still impossible to find anything more than a few weeks old without either knowing exactly when it was sent, or having a powerful search facility.

Greg tried a few different engines, and settled on the Xapian project as the best fit for the one-database-per-user that we wanted.

We tried indexing new emails as they arrived, even directly to fast SSDs, and discovered that the load was just too high. Our servers were overloaded trying to index in time – because adding a single email causes a lot of updates.

Luckily, Xapian supports searching from multiple databases at once, so we came up with the idea of a tiered database structure.

New messages get indexed to tmpfs in a small database. A job runs every hour to see if tmpfs is getting too full (over 50% of the defined size), in which case it compacts immediately, otherwise we automatically compact during the quiet time of the day. Compacted databases are more efficient, but read only.

This allows us to index all email immediately, and return a message that arrived just a second ago in your search results complete with highlighted search terms, yet not overload the servers. It also means that search data can be stored on inexpensive disks, keeping the costs of our accounts down.

Technical level: extreme

Here’s some very technical information about how the tiers are implemented, and an example of running the compaction.

We have 4 tiers at FastMail, though we don’t actually use the ‘meta’ one (SSD) at the moment:

  • temp
  • meta
  • data
  • archive

The temp level is on tmpfs, purely in memory. Meta is on SSD, but we don’t use that except during shutdown. Data is the main version, and we re-compact all the data level indexes once per week. Finally archive is never automatically updated, but we build it when users are moved or renamed, or can create it manually.

Both external locking (Xapian isn’t always happy with multiple writers on one database) and the compaction logic are managed via a separate file called xapianactive. The xapianactive looks like this:

% cat /mnt/ssd30/sloti30t01/store23/conf/user/b/brong.xapianactive
temp:264 archive:2 data:37

The first item in the active file is always the writable index – all the others are read-only.

These map to paths on disk according to the config file:

% grep search /etc/cyrus/imapd-sloti30t01.conf
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(the ‘default tier’ is to tell the system where to create a new search item)

So based on these paths, we find.

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
3328 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.264
1520432 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.37
3365336 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2

I haven’t compacted to archive for a while. Let’s watch one of those. I’m selecting all the tiers, and compressing to a single tier. The process is as follows:

  1. take an exclusive lock on the xapianactive file
  2. insert a new default tier database on the front (in this example it will be temp:265) and unlock xapianactive again
  3. start compacting all the selected databases to a single database on the given tier
  4. take an exclusive lock on the xapianactive file again
  5. if the xapianactive file has changed, discard all our work (we lock against this, but it’s a sanity check) and exit
  6. replace all the source databases for the compact with a reference to the destination database and unlock xapianactive again
  7. delete all now-unused databases

Note that the xapianactive file is only locked for two VERY SHORT times. All the rest of the time, the compact runs in parallel, and both searching on the read-only source databases and indexing to the new temp database can continue.

This allows us to only ever have a single thread compacting to disk, so our search drives are mostly idle, and able to serve
customer search requests very quickly.

When holding an exclusive xapianactive lock, it’s always safe to delete any databases which aren’t mentioned in the file – at worst you will race against another task which is also deleting the same databases, so this system is self-cleaning after any failures.

Here goes:

% time sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong
compressing temp:264,archive:2,data:37 to archive:3 for user.brong (active temp:264,archive:2,data:37)
adding new initial search location temp:265
compacting databases
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3.NEW
renaming tempdir into place
finished compact of user.brong (active temp:265,archive:3)

real 4m52.285s
user 2m29.348s
sys 0m13.948s

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
368 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.265
du: cannot access `/mnt/i30search/sloti30t01/store23/search/b/user/brong/*': No such file or directory
4614368 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3

If you want to look at the code, it’s all open source. I push the fastmail branch to github regularly. The xapianactive code is in imap/search_xapian.c and the C++ wrapper in imap/xapian_wrap.cpp.

Posted in Advent 2014, Technical. Comments Off on Dec 1: Email Search System

Updating our SSL certificates to SHA-256

This is a technical post. The important points to take away are that if, like most of our customers, you’re using FastMail’s web client with a modern, regularly updated browser like Chrome, Firefox, Internet Explorer or Safari, then everything will be fine. If you’re using an old browser or operating system (including long-unsupported mobile devices like old Nokia or WebOS devices), it may start failing to connect to FastMail during December, and you’ll need to make changes to the settings you use to access FastMail. Read on for details.

For many years the standard algorithm used to sign SSL certificates has been SHA-1. Recently, weaknesses have been exposed in that algorithm which make it unsuitable for encryption work. It’s not broken yet, but it’s reasonable to expect that it will be broken within the next year or two.

A replacement algorithm is available, called SHA-256 (sometimes called SHA-2), and its been the recommended algorithm for new certificates for the last couple of years.

Back in April, we updated our certificates with new ones that used SHA-256. This caused problems for certain older clients that didn’t have support for SHA-256. After some investigation, we reverted to SHA-1 certificates.

Recently Google announced that they would start deprecating SHA-1 support this year. Chrome 40 (currently in testing, due for release in January) will start showing the padlock icon on as “secure, with minor errors”. Crucially, it will no longer display the green “EV” badge.

As a result, we are intending to update our certificates to SHA-256 during December. Its something we wanted to do back in April anyway, as we’d much prefer to proactively support modern security best practice rather than scramble frantically to fix things when breaches are discovered.

Unfortunately, this will cause problems for customers using older browsers. Most desktop browsers should not have any problem, though Windows XP users will need to update to Service Pack 3. Many more obscure devices (notably Nokia and WebOS devices) do not support SHA-256 at all, and will not be able to connect to us securely.

We will be attempting to support a SHA-1 certificate on and, but only if our certificate authority will agree to issue one to us. Once we have that information I’ll update this post.

If you have any questions about this change, please contact support.

Further reading:

Posted in Technical. Comments Off on Updating our SSL certificates to SHA-256 now redirects to

In preparation for our our move to, we’ll be doing some testing on So if you use the beta server, expect some changes and potential issues over the next few days.

Currently that means if you go to, you’ll immediately be redirected to This is expected. Note that you can’t currently create aliases or rename your account to This is expected. This will only be available from Thursday as described in the original blog post.

Posted in Technical. Comments Off on now redirects to

SSL 3.0 disabled due to security vulnerability

This morning Google published news of a new vulnerability in SSL 3.0. You can read more about it in the original announcement and in CloudFlare’s analysis of the problem.

This is a serious issue that can leak user data. Unfortunately there’s no workaround – the only option we have is to disable SSL 3.0 on our servers entirely. We don’t like having to do this because we want our users to be able to use any client they choose to access their mail, but when there’s a security hole and no way to plug it we have no choice but to break things for some people in order to protect everyone.

Happily, this should not affect the majority of our users. The only significant browser to be affected is Internet Explorer 6 on Windows XP, which will now not be able to connect to at all. Similar changes have been made to our IMAP, POP and other backend services, so you may also have connection issues with older mail clients.

If you are unable or unwilling to upgrade your client software at this time, you can use (web) and (IMAP/POP/SMTP), both of which support SSL 3.0. As always, we highly discourage the use of these service names because they leave your data open to attack, and we may remove them in the future.

Update 16 Oct 2014 01:00 UTC: We’ve heard of at least two mail clients (Airmail and Windows Phone) that can receive but not send mail. Changing the outgoing settings to use port 587 instead of 465 has resolved the problem for some users. If you’re seeing similar problems, give that a try.

Posted in Technical. Comments Off on SSL 3.0 disabled due to security vulnerability

Payment Issues

Recently there have been some problems with our payment processor.

One of these is that sometimes there is a significant delay in completing a transaction, so the funds are “reserved” but the transaction isn’t actually completed until a few weeks later.

The other problem was that a number of charges were made to some of our customers’ credit cards, in our name. The majority of these are USD 1.00 test charges that are used to verify that a card is valid. Normally we cancel these charges immediately and the charge doesn’t appear on the card statement. However the problem at our payment processor has resulted in some of these supposedly “cancelled” charges recently being applied to customer cards. A number of other charges which were supposed to be “cancelled” have also been processed. This was done in error and without any instructions from FastMail.

We are currently working with our payment processor to refund these erroneous payments. In some cases, the refunds are unable to be processed directly because the card has expired or been cancelled.

We may need to refund these charges via an alternative method, or, at the user’s option, instead credit the amount towards future FastMail renewals. We will be in touch with the affected customers once we have more information on the scope of the problem.

We would like to assure our users that we will make certain that all erroneous charges are corrected. This may take some time, so thanks for your patience.

Our apologies for any inconvenience or confusion. We will post more information as it becomes available.
– the FastMail team

Posted in Technical. Comments Off on Payment Issues

New phishing trick, data: URLs to avoid forgery reporting

This is a technical post about a new and interesting phishing technique seen today. Regular FastMail users can skip this post.

We saw an interesting new phishing attempt today that uses a relatively novel technique to try and hide the source of the attack and avoid it being reported as a web forgery.

Firstly the email itself looks reasonably well done (apart from the year in the subject being completely wrong), certainly it’s not the poor quality you often see. It looked like this (ANZ is an Australian bank):


Secondly, the email was sent using a compromised gmail account with a .edu address. In fact there were two separate emails, both from different compromised gmail .edu accounts. I imagine compromised gmail .edu accounts aren’t that easy to get, and this significantly reduced the chances of it being caught by any spam filter.

Thirdly, the phishing page itself is interesting in that it:

  1. Uses a standard link shortener for a redirect ( in this case)
  2. Which redirects to the phishing delivery page (a compromised page on
  3. That page however rather than hosting the HTML phishing login page directly, does this:

<script type="text/javascript">
        window.location="data:text/html;base64,... base64 encoded version of HTML phishing login page ...";

That data: URL is itself the phishing page content, which includes links to real ANZ website logos to make it look as authentic as possible, but has a form submit action to a compromised page on

This approach is interesting because it makes it impossible to report this page as a forgery using the standard Firefox "Report Web Forgery" action because Firefox thinks it’s a data: URL. Neat trick that makes it harder to remove or block in the long run.

I’ve reported this issue as a Firefox bug:

Posted in Technical. Comments Off on New phishing trick, data: URLs to avoid forgery reporting

Errors on classic mailbox screen and pop emails retrieved again for some users

A rollout of some new code today contained some errors that badly affected two separate areas of FastMail.

1. Errors selecting any emails to action on the classic interface would cause a fatal error

An internal misuse of an API meant that selecting any emails on the classic interface Mailbox screen and trying to action those emails would fail with a fatal error. Reading and applying actions to individual emails continued to work fine. The way this manifested itself unfortunately wasn’t picked up in our testing before being rolled out. It has been fixed now and we’ll update our tests to catch this. The error lasted for for about 3 hours.

2. Pop links for some users re-downloaded all emails again

A long term bug in the pop retrieve system resulted in a very rare and intermittent problem where some links for some users would forget all existing downloaded message. This means that in certain cases users that had set the “Leave on server” option might see existing messages that had been downloaded previously downloaded again, possibly several times. This obviously resulted in duplicate copies of the same message appearing in a folder.

Unfortunately a fix to this bug rolled out to one server for a short time actually made things worse, causing the same problem to occur for more users than the original bug.

A correct fix for the original problem and the subsequent bug has now been rolled out everywhere.

Users affected by this bug can find and remove any duplicate messages using the Advanced -> Folders -> Mass delete/Download/Remove duplicates … (button down the bottom) screen. Select the folder with the duplicates at the top, and use the “Remove duplicate messages” section to find and remove any duplicate messages.

Posted in Technical. Tags: . Comments Off on Errors on classic mailbox screen and pop emails retrieved again for some users

Get every new post delivered to your Inbox.

Join 6,880 other followers

%d bloggers like this: