"View" link removed from attachments on message read screen in "Public Terminal" mode

When you enable the "Public Terminal" option on the login screen, Fastmail sets the "no-cache" and "no-store" cache control headers on every page. This means that browsers should not store a copy of the pages you visit (e.g. emails you read) to their local disk. Even after you logout of your session and leave the computer, someone comes along and tries to view a page in the browser history, it should re-check with the server first, which of course will return "this user is now logged out, show the login page instead".

However this is a problem with this whole setup related to attachments. When an email has an attachment, the content of the attachment might be in a form the browser doesn’t understand (e.g. Microsoft Word document). In that case, the browser has to save a copy of the attachment to the local disk, and then launch Microsoft Word to open the file.

Now in the case of the "View" link, the saving to disk would be done automatically into a temporary file storage area. However in IE, if you try and download an SSL document with the no-cache or no-store attributes set, IE will explicitly not save the file to disk, and then when it tries to launch Microsoft Word to read the file, you’ll get a "file does not exist" error or the like.


For other browsers, it appears they work around this problem by actually saving a copy to disk in the temporary storage area, but they delete the file when you close the browser (at least that’s what Firefox did when I tested). That still potentially does leave the file on disk for some time.

To ensure the best privacy possible, while still allowing people to view attached documents in "Public Terminal" mode, we’ve decided to do the following:

  • When you login with the "Public terminal" mode, we’ve removed the "View" link next to attachments. This solves two problems; the unexpected "file not found" in IE, and the privacy concern of storing attachments to disk in the temporary file area of other browsers
  • We’ve left the "View" link next to image attachments, because the web browser can display images itself, without launching a separate program, so it can obey the "no-cache"/"no-store" directives
  • With the "Download" link (which automatically brings up a "Save as…" dialog box), we’ve removed the "no-cache" and "no-store" settings, which means that IE will let you download it and save it somewhere so you can open it to view the document.

We like this solution because it makes things clearer to the user. In "Public Terminal" mode, if you want to view an attachment, you have to download it first, explicitly save it somewhere and then view it. The alternative approach of letting the browser do it either fails (IE), or causes an auto-save of the file to a temporary area which leaves it temporarily cached on the machine when the user doesn’t expect it.

Posted in News, Technical. Comments Off on "View" link removed from attachments on message read screen in "Public Terminal" mode

TCP keepalive, iOS 5 and NAT routers

This post contains some very technical information. For users just interested in the summary:

If over the next week you experience an increase in frozen, non-responding or broken IMAP connections, please contact our support team (use the "Support" link at the top of the http://www.fastmail.fm homepage) with details. Please make sure you include your operating system, email software, how you connect to the internet, and what modem/router/network connection you use in your report.

The long story: The IMAP protocol is designed as a long lived connection protocol. That is, your email client connects from your computer to the server, and stays connected for as long as possible.

In many cases, the connection remains open, but completely idle for extended periods of time while your email client is running but you are doing other things.

In general while a connection is idle, no data at all is sent between the server and the client, but they both know the connection still exists, so as soon as data is available on one side, it can send it to the other just fine.

There is a problem in some cases though. If you have a home modem and wireless network, then you are usually using a system called NAT that allows multiple devices on your wireless network to connect to the internet through one connection. For NAT to work, your modem/router must keep a mapping for every connection from any device inside your network to any server on the internet.

The problem is some modems/routers have very poor NAT implementations that "forget" the NAT mapping for any connection that’s been idle for 5 minutes or more (some appear to be 10 minutes or more). What this means is that if an IMAP connection remains idle with no communication for 5 minutes, then the connection is broken.

In itself this wouldn’t be so bad, but the way the connection is broken is that rather than telling the client "this connection has been closed", packets from the client or server just disappear which causes some nasty user visible behaviour.

The effect is that if you leave your email client idle for 5 minutes and the NAT connection is lost, if you then try and do something with the client (e.g. read or move an email), the client tries to send the appropriate command to the server. But the TCP packets that contain the command never arrive at the server, but neither are RST packets sent back that would tell the client that there’s any problem with the connection, the packets just disappear. So the local computer tries to send again after a timeout period, and again a few more times, until usually about 30 seconds later, it finally gives up and marks the connection as dead, and finally sends that information back up to the email client, which shows some "connection was dropped by the server" type message.

From a user perspective, it’s a really annoying failure mode that looks like a problem with our server, even though it’s really because of a poor implementation of NAT in their modem.

However this is a workaround for this. At the TCP connection level, there’s a feature called keepalive that allows the operating system to send regular "is this connection still open?" type packets back and forth between the server and the client. By default keepalive isn’t turned on for connections, but it is possible to turn it on via a socket option. nginx, our frontend IMAP proxy, allows you to turn this on via a so_keepalive configuration option.

However even after you’ve enabled this option, the default time between keepalive "ping" packets is 2 hours. Fortunately again, there’s a Linux kernel tuneable net.ipv4.tcp_keepalive_time that lets you control this value.

By lowering this value to 4 minutes, it causes TCP keepalive packets to be sent over open but idle IMAP connections from the server to the client every 4 minutes. The packets themselves don’t contain any data, but what they do do is cause any existing NAT connection to be marked as "alive" on the users modem/router. So poor routers with NAT connections that would normally timeout after 5 minutes of inactivity are kept alive, so the user doesn’t see the nasty broken connection problem described above, and neither is there a visible downside to the user either.

So this is how things have been for the last 4-5 years, which has worked great.

Unfortunately, there’s a new and recent problem that has now appeared.

iOS 5 now uses long lived persistent IMAP connections (apparently previous versions only used short lived connections). The problem is that our ping packets every 4 minutes mean that the device (iPhone/iPad/iPod) is "woken up" every 4 minutes as well. This means the device never goes into a deeper sleep mode, which causes significantly more battery drain when you setup a connection to the Fastmail IMAP server on iOS 5 devices.

Given the rapid increase in use of mobile devices like iPhones, and the big difference in battery life it can apparently cause, this is a significant issue.

So we’ve decided to re-visit the need for enabling so_keepalive in the first place. Given the original reason was due to poor NAT routers with short NAT table timeouts, that was definitely an observed problem a few years back, but we’re not sure how much of a problem it is now. It’s possible that the vast majority of modems/routers available in the last few years have much better NAT implementations. Unfortunately there’s no way to easily test this, short of actually disabling keepalive, and waiting for users to report issues.

So we’ve done that now on mail.messagingengine.com, and we’ll see over the next week what sort of reports we get. Depending on the number, there’s a few options we have:

  1. If there’s lots of problem reports, we’d re-enable keepalive by default, but setup an alternate server name like mail-mobile.messagingengine.com that has keepalive disabled, and tell mobile users to use that server name instead. The problem with this is many devices now have auto configuration systems enabled, so users don’t even have to enter a server name, so we’d have to work out how to get that auto configuration to use a different server name
  2. If there’s not many problem reports, we’d leave keepalive off by default, but setup an alternative server name like mail-keepalive.messagingengine.com that has keepalive enabled, and for users that report connection "freezing" problems, we’d tell them to switch to using that server name instead
  3. Ideally, we’d detect what sort of client was connecting, and turn on/off keepalive as needed. This might be possible using software like p0f, but integrating that with nginx would require a bit of work, and still leaves you with the problem of an iPhone user that is usually in their office/home all day and uses a wireless network with a poor NAT router, would they prefer the longer battery life, or better connectivity experience.

I’ll update this post in a week or two when we have some more data.

Posted in News, Technical. Comments Off on TCP keepalive, iOS 5 and NAT routers

DKIM signing outgoing email with From address domain

DKIM is an email authentication standard that allows senders of email to sign an email with a particular domain, and for receivers of the email to confirm that the email was signed by that domain and hasn’t been altered. There’s some more information about how DKIM is useful in this previous blog post. We’ve been DKIM signing all email sent via FastMail for the last 2 years.

In the original design of the DKIM, the domain that signed the email had no particular relationship to the domain in the From address of the email. This was particularly useful for large email providers like us. We have 10,000’s of domains, but would sign all email with just our "generic" messagingengine.com domain.

However this state of affairs is beginning to change. Standards like Author Domain Signing Practices explicitly link the domain of the email address in the From header of the email to the DKIM signing domain. Also recently Gmail has changed their web interface so that email sent with a From domain that’s different to the DKIM signing domain may be shown with an extra "via messagingengine.com" notice next to the sender name.

So we’ve now rolled out new code that changes how all emails sent through FastMail are DKIM signed. We always DKIM sign with messagingengine.com (as we always have), but we also now sign with a separate key for the domain used in the From address header where possible (see below for more details).

For most users, there should be no noticeable difference. For users that use virtual domains at FastMail, or have their own domain in a family/business, then when you send via FastMail, Gmail should no longer show "via messagingengine.com" on the received message (if your DNS is correctly setup, see below for more details).

For users that host their DNS with FastMail (eg. nameservers for your domain are ns1.messagingengine.com and ns2.messagingengine.com), this will "just work". We’ve generated DKIM public/private keys for all domains in our database, and automatically do so when new domains are added. We also publish the public keys for all domains via ns1.messagingengine.com/ns2.messagingengine.com.

In general if you can, we highly recommend hosting your DNS with us. For most cases the default settings we provide "just work", and if you need to customise your DNS, our control panel allows you to add any records of any type, without the arbitrary limitations many other DNS providers have.

However for users that host DNS for their domains externally and want to continue to do so, you’ll have to explicitly add the DKIM public key using your domain hosters DNS management interface. Unfortunately there’s 100’s of different DNS providers out there, so we can’t give specific directions for each one.

The general steps are:

  1. Login to your FastMail account and go to Options –> Virtual Domains (or Manage –> Domains for a family/business account).
  2. Scroll to the bottom, and you’ll see a new "DKIM signing keys" section. For each domain you have, you’ll see a DKIM public key.
  3. Login to your DNS provider, and create a new TXT record for each domain listed and use the value in the "Public Key" column as the TXT record data to publish.

Important: Note that you have to add the TXT record for the domain name shown in the DKIM signing keys section, which will be mesmtp._domainkey.yourdomain.com, do not add it for the base domain name yourdomain.com, that won’t work.

That should be it.

Note that initially each domain is marked as DKIM disabled (Enabled column = [ ]). While a domain is DKIM disabled, we won’t sign any sent emails. This is to avoid DKIM signing failures when the receiving side tries to lookup the public signature and fails to find it. We regularly check each domain to see if the correct public key TXT record is being published. If it is, we mark the domain in our database as "DKIM enabled" (Enabled column = [*]), and then begin signing sent emails.

So after you setup the records at your DNS provider, you should wait a few hours, then check this table again to see that the domain is now correctly DKIM enabled.

Some other technical notes:

There’s currently no way to change the public/private key used to sign emails or upload new ones. We always generate our own key pair for each domain and use the DKIM selector "mesmtp" to sign emails. This shouldn’t be a problem. If you’re transitioning from another provider to FastMail, you can use our custom DNS to publish the DKIM record of the previous provider with it’s selector as well as our own during the transition. Vice-versa for transitioning away from FastMail. The only other reason to change the selector would be if the private key was compromised, which should never happen as it’s stored securely in FastMail’s systems.

Posted in News, Technical. Comments Off on DKIM signing outgoing email with From address domain

Get every new post delivered to your Inbox.

Join 7,293 other followers

%d bloggers like this: