Dec 2: Security – Confidentiality, Integrity and Availability

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on December 1st was about the Email Search System. The next post on December 3rd is all about how we do real-time push notifications.

Technical level: low

This is the first of a series of blog posts on security, both FastMail’s approach to various threats, and how the location of our servers interacts with security risks. We’re not digging into the technical details yet, just looking at an overview of what security means.

I always recommend that people read this humorous paper by James Mickens at Microsoft Research (pdf). There are a ton of security in-jokes there, but he makes some really good points.

Another great place to learn more about security best practices is Bruce Schneier’s blog. He’s been thinking about this stuff for a long time, and is one of the world’s acknowledged experts on computer security.

Security consists of three things: Confidentiality, Integrity and Availability. There’s a good writeup on wikipedia and also a fairly good post on blog overflow – except that it falls for the trap of defining integrity as only protecting information from being modified by unauthorized parties.

Honestly, the biggest “security risk” to data integrity in the history of email has been the unreliable hard drives in people’s home computers dying, and all the email downloaded by POP3 over the years being lost or corrupted badly in a single screeching head-crash. For us, the biggest integrity risk is hardware or disk failures corrupting data, and I’ll write more about some of the corruption cases we’ve dealt with as well.

We care about all three security components at FastMail, and work to strike a sensible balance between them. There’s a joke that to perfectly secure a server you need to encase it in concrete deep under ground, and then cut off the power and network cables. It’s funny because there’s a hint of truth.

To be useful, a server has to be online. And that server is running imperfect software on imperfect hardware, which may have even been covertly modified (not just by the NSA either – anyone with a big enough budget and no regard for the law can pull off something like that).

Thankfully, the same security processes and architectures that defend against system failures are also good for protecting again active attackers. We follow best practices like running separate physical networks for internal traffic, restrictive firewalls that only allow expected traffic into our servers, following security announcement mailing lists for all our software, and only choosing software with a good security record.

That’s the baseline of good security. In the following blogs, we will look at some of the specific things that FastMail does to protect our systems and our users’ data.

Posted in Advent 2014. Comments Off

Dec 1: Email Search System

This blog post is part of the FastMail 2014 Advent Calendar.

The next post on December 2nd is Security – Confidentiality, Integrity and Availability.

Technical level: medium

Our email search system was originally written by Greg Banks, who has moved on to a role at Linked In, so I maintain it now. It’s a custom extension to the Cyrus IMAPd mail server.

Fast search was a core required feature for our new web interface. My work account has over half a million emails in it, and even though our interface allows fast scroll, it’s still impossible to find anything more than a few weeks old without either knowing exactly when it was sent, or having a powerful search facility.

Greg tried a few different engines, and settled on the Xapian project as the best fit for the one-database-per-user that we wanted.

We tried indexing new emails as they arrived, even directly to fast SSDs, and discovered that the load was just too high. Our servers were overloaded trying to index in time – because adding a single email causes a lot of updates.

Luckily, Xapian supports searching from multiple databases at once, so we came up with the idea of a tiered database structure.

New messages get indexed to tmpfs in a small database. A job runs every hour to see if tmpfs is getting too full (over 50% of the defined size), in which case it compacts immediately, otherwise we automatically compact during the quiet time of the day. Compacted databases are more efficient, but read only.

This allows us to index all email immediately, and return a message that arrived just a second ago in your search results complete with highlighted search terms, yet not overload the servers. It also means that search data can be stored on inexpensive disks, keeping the costs of our accounts down.

Technical level: extreme

Here’s some very technical information about how the tiers are implemented, and an example of running the compaction.

We have 4 tiers at FastMail, though we don’t actually use the ‘meta’ one (SSD) at the moment:

  • temp
  • meta
  • data
  • archive

The temp level is on tmpfs, purely in memory. Meta is on SSD, but we don’t use that except during shutdown. Data is the main version, and we re-compact all the data level indexes once per week. Finally archive is never automatically updated, but we build it when users are moved or renamed, or can create it manually.

Both external locking (Xapian isn’t always happy with multiple writers on one database) and the compaction logic are managed via a separate file called xapianactive. The xapianactive looks like this:

% cat /mnt/ssd30/sloti30t01/store23/conf/user/b/brong.xapianactive
temp:264 archive:2 data:37

The first item in the active file is always the writable index – all the others are read-only.

These map to paths on disk according to the config file:

% grep search /etc/cyrus/imapd-sloti30t01.conf
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(the ‘default tier’ is to tell the system where to create a new search item)

So based on these paths, we find.

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
3328 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.264
1520432 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.37
3365336 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2

I haven’t compacted to archive for a while. Let’s watch one of those. I’m selecting all the tiers, and compressing to a single tier. The process is as follows:

  1. take an exclusive lock on the xapianactive file
  2. insert a new default tier database on the front (in this example it will be temp:265) and unlock xapianactive again
  3. start compacting all the selected databases to a single database on the given tier
  4. take an exclusive lock on the xapianactive file again
  5. if the xapianactive file has changed, discard all our work (we lock against this, but it’s a sanity check) and exit
  6. replace all the source databases for the compact with a reference to the destination database and unlock xapianactive again
  7. delete all now-unused databases

Note that the xapianactive file is only locked for two VERY SHORT times. All the rest of the time, the compact runs in parallel, and both searching on the read-only source databases and indexing to the new temp database can continue.

This allows us to only ever have a single thread compacting to disk, so our search drives are mostly idle, and able to serve
customer search requests very quickly.

When holding an exclusive xapianactive lock, it’s always safe to delete any databases which aren’t mentioned in the file – at worst you will race against another task which is also deleting the same databases, so this system is self-cleaning after any failures.

Here goes:

% time sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z archive -t temp,meta,data,archive -u brong
compressing temp:264,archive:2,data:37 to archive:3 for user.brong (active temp:264,archive:2,data:37)
adding new initial search location temp:265
compacting databases
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3.NEW
renaming tempdir into place
finished compact of user.brong (active temp:265,archive:3)

real 4m52.285s
user 2m29.348s
sys 0m13.948s

% du -s /var/run/cyrus/search-sloti30t01/b/user/brong/* /mnt/i30search/sloti30t01/store23/search/b/user/brong/* /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/*
368 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.265
du: cannot access `/mnt/i30search/sloti30t01/store23/search/b/user/brong/*': No such file or directory
4614368 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.3

If you want to look at the code, it’s all open source. I push the fastmail branch to github regularly. The xapianactive code is in imap/search_xapian.c and the C++ wrapper in imap/xapian_wrap.cpp.

Posted in Advent 2014, Technical. Comments Off

FastMail Advent 2014

Welcome to the inaugural FastMail Advent Blog. One post per day for the next 24 days.

The idea came from a reponse I made to a question on reddit about physical locations of servers and their impact on security.

I wrote that up in more detail, with links to blog posts about what FastMail does to address various categories of security risk, and suddenly found myself with something much too long for a single blog post. I wanted to split it up into separate posts, and then started thinking about frequency – and meanwhile my daughters were asking about advent calendars for the year, and it clicked.

We’ve been promising to blog more about some of our technology, and also about some of the less-well-known features – here was a perfect opportunity. You won’t just be hearing from me, I’m going to try to get everyone to write up something about their areas of expertise.

There’s a fine internet tradition in what we’re doing here, check out the perl advent calendar for example. They’ve been doing it for years.

I will be updating this blog post with links to every day as they are posted:

  1. Email Search System
  2. Security – Confidentiality, Integrity and Availability
  3. Push it real good
  4. Standalone Mail Servers
  5. Security – Integrity
  6. User authentication
  7. Automated installation
  8. Squire: FastMail’s rich text editor
  9. Email Authentication
  10. Security – Availability
  11. FastMail Support
  12. FastMail’s MySQL Replication: Multi-Master, Fault Tollerance, Performance. Pick Any Three
  13. FastMail DNS hosting
  14. On Duty!
  15. Putting the fast in FastMail: Loading your mailbox quickly
  16. Security – Confidentiality
  17. Testing
  18. Billing and Payments — a potted history
  19. Mailr
  20. Open-sourcing OvertureJS – the JS lib that powers FastMail
  21. File Storage
  22. CardDAV Beta Release
Posted in Advent 2014, News. Comments Off

Updating our SSL certificates to SHA-256

This is a technical post. The important points to take away are that if, like most of our customers, you’re using FastMail’s web client with a modern, regularly updated browser like Chrome, Firefox, Internet Explorer or Safari, then everything will be fine. If you’re using an old browser or operating system (including long-unsupported mobile devices like old Nokia or WebOS devices), it may start failing to connect to FastMail during December, and you’ll need to make changes to the settings you use to access FastMail. Read on for details.

For many years the standard algorithm used to sign SSL certificates has been SHA-1. Recently, weaknesses have been exposed in that algorithm which make it unsuitable for encryption work. It’s not broken yet, but it’s reasonable to expect that it will be broken within the next year or two.

A replacement algorithm is available, called SHA-256 (sometimes called SHA-2), and its been the recommended algorithm for new certificates for the last couple of years.

Back in April, we updated our certificates with new ones that used SHA-256. This caused problems for certain older clients that didn’t have support for SHA-256. After some investigation, we reverted to SHA-1 certificates.

Recently Google announced that they would start deprecating SHA-1 support this year. Chrome 40 (currently in testing, due for release in January) will start showing the padlock icon on fastmail.com as “secure, with minor errors”. Crucially, it will no longer display the green “EV” badge.

As a result, we are intending to update our certificates to SHA-256 during December. Its something we wanted to do back in April anyway, as we’d much prefer to proactively support modern security best practice rather than scramble frantically to fix things when breaches are discovered.

Unfortunately, this will cause problems for customers using older browsers. Most desktop browsers should not have any problem, though Windows XP users will need to update to Service Pack 3. Many more obscure devices (notably Nokia and WebOS devices) do not support SHA-256 at all, and will not be able to connect to us securely.

We will be attempting to support a SHA-1 certificate on insecure.fastmail.com and insecure.messagingengine.com, but only if our certificate authority will agree to issue one to us. Once we have that information I’ll update this post.

If you have any questions about this change, please contact support.

Further reading:

Posted in Technical. Comments Off

Recent interviews on Rocketship.fm and DomainSherpa.com

I was recently interviewed by two separate sites. Since these interviews cover some of the history of FastMail, the purchase by Opera and re-sale back to the staff, and our recent acquisition of fastmail.com, I thought it might be interesting to some of our users.

Why You Should Charge from Day One

http://rocketship.fm/episodes/ep-79-rob-mueller/

After 15 Years, FastMail Finally Acquires Their .Com – With Rob Mueller

http://www.domainsherpa.com/rob-mueller-fastmail-interview/

Posted in Off Topic. Comments Off

FastMail app for iOS and Android now available

Today we’re proud to announce the release of the FastMail app for your iPhone, iPad, iPod and Android devices. You can get it right now from the App Store (iOS) or the Play Store (Android).

 Download on the App StoreGet it on Google Play

Our apps have been designed to combine our lightning-fast mobile web app with device features normally only available to native apps, most notably push notifications.

iOS notificationAndroid notification

On Android, you’ll even find support for your smartwatch!

Android Wear notificationPebble notification

More information about the FastMail app is available in our help.

Posted in News. Comments Off

About our first fastmail.com customer

Since we rolled out our new fastmail.com domain last week, we’ve had 10,000’s of users use the domain to rename, signup and create aliases.

We decided to have a quick look through our logs and find the first customer to use fastmail.com and ask them a few questions about themselves and FastMail. Thanks for taking some time out to answer questions for us Joe D!


What country & city do you live in?

Newcastle, Australia

What do you do for a living?

I’m a software developer at a services company for the mining and energy industries in Australia.

How long have you used FastMail?

I signed up to FastMail when I was in high school, back in 2002. I’ve been pretty happy since then. My home address changes more often than my email address.

How did you find out about FastMail?

It was so long ago, I honestly don’t remember!

Why do you use FastMail?

I love FastMail for the power-user features. I like being able to set up Personalities to send from different email addresses, and being able to control every aspect of my email filtering through Sieve scripts. Plus I seem to get way less spam than at addresses I’ve tried at other providers.

What domain was your previous FastMail address at?

I have a few addresses on the mailbolt.com domain, which I’ll continue to use for certain things like store memberships, news subscriptions, banking and bills. I also have a "junk" address on this domain, since every website and his dog requires you to sign up and provide an email address these days.

Why did you want a fastmail.com address?

It’s nice and simple. It looks great written, and I can tell it to someone over the phone without having to repeat bits of it.

Were you actively waiting for the opening up of fastmail.com on the day?

I’m a little embarrassed to admit I had some help on this one. I had a program monitoring the FastMail website for the exact moment fastmail.com became available, so that I could register my address straight away. Is this taking email too seriously?

Do you plan to use your new @fastmail.com address as your primary address? Have you told people about it yet?

It’ll be my new personal address to give out to people online and in person. I’ve only told a few people so far, but it will get more and more use over time.


We’ve sent Joe a T-Shirt from our RedBubble store for his time.

Of course there’s always going to be a rush for the most popular names, so we hope everyone managed to get the fastmail.com address they wanted. If not, remember you can also signup your own domain (personally, we use gandi.net) and use that for receiving email (Enhanced or higher personal accounts, or any Family/Business accounts required).

Thanks for reading

The FastMail Team

Posted in Marketing. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,842 other followers