XMPP security improvements

We’ve just rolled out an update to our XMPP service to give it the same level of TLS encryption support that you’ll find in our IMAP, POP3 and SMTP services. It now supports TLS 1.2 with modern ciphers. The changes mean we now get an A rating on the XMPP security test.

If you didn’t know we had an XMPP service, or if you don’t know what any of this means, then you can ignore it. Everything should just continue to work!

Our XMPP service has lagged behind our other services for a while because our XMPP server, djabberd, has problems with TLS >1.0 due to deficiencies in Perl’s TLS libraries, and has resisted our best efforts to fix it. We’re hoping to replace it with another server in the next year or two so to avoid having to do a bunch of work that we’d eventually throw out, we decided to follow the same model that we use for IMAP, POP3 and SMTP. We added XMPP support to nginx’s mail proxy, and then let it do authentication and encryption termination, both tasks which it excels at.

Most importantly, nginx is well known as a highly stable and secure TLS server and receives a constant stream of updates. Any improvements we roll out in the future will automatically be applied to the XMPP service as well.

More information about the actual implementation in nginx is at http://robn.io/nginx-xmpp/.

We’re quietly working on modernising our XMPP service. If that’s something you’re interested in then keep an eye on this blog over the next few months.

Know how to identify genuine email from FastMail

Recently, we’ve seen an upswing in the number of attempts by criminals to steal FastMail accounts. We’re working hard to maintain our high security and keep them at bay, but we’ve also got three simple tips you can follow to keep your account secure.

1. Know how to identify genuine email from FastMail

All genuine email from FastMail is displayed with a white tick in a green circle next to the sender’s name in both the mailbox list and on the message itself. It looks exactly like this in the mailbox:

Green tick next to sender name in mailbox list

And like this on the message:

Green tick next to sender name in message view

If the email doesn’t have the green tick, it’s not from us.

Please note, we can only do this in our web interface and apps; it will not appear in other email clients. It will also not appear in our classic interface; we recommend users upgrade to our current interface for increased security.

Always look for the green tick before trusting emails supposedly from FastMail.

2. Look for the green badge before logging in

When logging into our webmail, always look for a green badge in the address bar of your browser with the text “FastMail Pty Ltd”. Phishing sites (scam websites that try to steal your login details) can easily clone the look and feel of our website, however they can’t clone the green badge.

The badge looks like this in Google Chrome:

Green EV SSL badge reads FastMail Pty Ltd

And like this in Mozilla Firefox:

Green EV SSL badge reads FastMail Pty Ltd

And like this in Safari:

Green EV SSL badge reads FastMail Pty Ltd

And like this in Internet Explorer:

Green EV SSL badge reads FastMail Pty Ltd

And like this in Opera:

Green EV SSL badge reads FastMail Pty Ltd

If you don’t see the badge, you’re not at the genuine FastMail website.

3. Never reuse your FastMail password at another service

Your email is the key to your digital life. Almost every web service you use, such as Amazon, Facebook or Twitter, allows you to reset their password by sending a link to your email address. It’s vitally important to keep your email password secure, as it provides access to everything else!

When you reuse your FastMail password at other sites, you’re making it much easier for attackers to potentially break in to your account. Other sites often don’t have the same high security measures as FastMail (such as compulsory HTTPS, locked-down servers, etc.), which makes them much easier for criminals to break in to. If they hold your email address and the same password that you use for FastMail, the attacker can then access your email account and get into everything else you use online.

Always use a unique password for FastMail that you don’t use elsewhere.

Follow these three simple tips, and you’ll be protected against the vast majority of attacks we see.

Posted in News. Comments Off

FastMail app for Intel-based Android devices now available

Today I pushed an update of the FastMail app that works on Intel-based Android devices. We’ve had a few requests for this as more and more Android devices are now running on Intel CPUs. Happily, the Crosswalk browser engine we use in the app has had Intel support for a long time, so it was just a matter of adjusting our build system to be able to build two different versions of the app.

As usual, its available from the Google Play Store.

Posted in Feature announcement. Comments Off

Dec 24: Working at FastMail

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 23nd December was the open protocol, JMAP. And this is the end!

Technical level: low

FastMail has been around for 15 years now, via a short detour as part of Opera Software and then back to being our own company again.

Some History

I was hired in 2004 as the fourth member of a small technical team in Melbourne. Rob M was living overseas at the time, so I worked with Jeremy (one of the original founders, he’s moved on to other things now) and Richard. We had no office, but I would catch the train and tram to Port Melbourne and work with Jeremy in his lounge room.

After working for a big corporate where (no joke) I couldn’t have a server to do my work for the 6 months I was seconded to New Jersey, because they needed longer than that to plan things, and where I only managed to wrangle a desktop computer to make into a server because my laptop had been purchased in Australia and wasn’t in their database… it was a breath of fresh air to be asked to specify the laptop that I wanted and have it delivered and waiting for me when I started.

Jeremy also had another company, and we moved in with them when they got some space of their own. We shared a house in Port Melbourne where we set up desks in the bedrooms, and then later a proper office in Melbourne CBD until they were sold in 2008. We moved to a serviced office on the 50th floor of one of the tallest buildings in Melbourne. The view was fantastic, though my ears always popped in the elevator! Jeremy stayed with ODG, so it was just the three of us working together.

After the sale to Opera, we doubled the size of the team and took a larger office on the same floor. I was lucky enough to get a transfer to head office in Norway in 2011-2012, and while I was away the team in Australia grew further and moved to our current office on William St in the Melbourne CBD (interestingly, our datacentre in New York is also on a William St — it hasn’t caused any misdirected mail yet). We have a great office of our own now, with plenty of space.

Office environment

We work in rooms with 2-4 people, with doors that can be closed (though they usually aren’t) and a boardroom that’s big enough for the entire team to get together for our weekly status meeting. If anyone is remote (working from home, travelling, etc) they join via video conference. We’ve been using AppearIn from our friends at Telenor.

IMG_20141223_082104

We have a huge open breakout area with couches, table tennis table and kitchen.

IMG_20141223_120108

The nice thing about working on computers on the other side of the world is that it really doesn’t matter where you are. We don’t treat the office network specially, everybody’s laptop makes its own VPN connection anyway – so we can do our work anywhere. Most of the team have children, and many of us work from home one or two days per week.

When we are in the office together, we frequently gather around whiteboards to nut out ideas. The great thing about smartphones is that everyone has a camera, so we all take a photo of the end result and keep it with us as we go back to our individual tasks.

A Small Business

The great thing about FastMail is that it’s a blend of startup and small business. We have the best bits of startup culture — flexible working hours, free coffee, snacks and drinks in the fridge, table tennis table, cake on Fridays (often shared with our friends at ODG, we still stay in touch). This is matched with the best bits of a profitable company — consistent revenue, existing infrastructure, decent salaries, and people who understand the business side of things as well as the tech.

My first question when I interviewed with FastMail was “do you have someone who knows how to run a business”, because I worked for a dotcom that went bankrupt due to poor business planning. I didn’t want to live through that mess again. FastMail has had steady growth every year for the last 15 years, thanks to our fantastic users who appreciate our product and stay with us.

Jobs at FastMail

As with any business, if the right person appears, sometimes you adjust things to create a role for them. Our tasks aren’t that fixed, we split the work between us to get the required jobs done.

Having said that, we have two specific positions opening up in our Melbourne, Australia office for early 2015:

These two people will be working on both our FastMail product and building the reference open-source implementations for JMAP.

If you have the skills we need, and the right to work in Australia (sorry, we can’t help with visas or sponsorships), then drop us a line at jobs@fastmail.com.

Thanks!

Thank you to everyone who has been following this series, reading what we write. More than anything, people want to know they are bringing value to others. One of the best things about working at FastMail is that code we write is out there making people’s lives better almost immediately — that’s a great feeling. The positive feedback we’ve been receiving has made all the effort worthwhile, even last-minute scramble to get posts finished on the weekends!

Extra special thanks to all our customers. It’s your ongoing support that allows us to continue our passion of building email, calendar and contacts done right.

Wishing everybody a happy and safe holiday season.

Posted in Advent 2014. Comments Off

Dec 23: JMAP — A better way to email

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 22nd December was the long awaited beta of contact syncing. The final post on 24th December is about our work environment.

Technical level: low

You can reproduce the demos by signing up a trial account at FastMail and trying for yourself!

Protocols

FastMail started with IMAP in 1999 when it was still quite new. We have been involved in the standards process, and have watched while non-open protocols led the way on features, usability and reliability. We pride ourselves on the standards compliance of our server, but we have been frustrated by the lack of progress in the third-party client experience available to our users.

The fragmentation of server support for newer IMAP features means that clients either have multiple implementations of everything, with the complexity and bugs that involves, or fall back to lowest-common-denominator behaviour. Extending IMAP further just makes this situation worse.

Having a separate protocol (SMTP) for sending email, running on a different TCP port and with potentially different credentials provides opportunities for partial failures, where a client can send but not receive or vice versa. These issues can be caused by firewalls, by temporary maintenance on the server, or just misconfiguration.

Regardless, for the end user the result is confusion. This is a frequent support problem for anybody with non-technical users, and the addition of CalDAV and CardDAV — everybody wants calendar and contact sync these days — means yet another failure mode in the mix.

FastMail’s protocol

When we built our own client and API we had the benefit of years of experience. I had rewritten the Cyrus IMAPd internals with strong locking to allow reliable QRESYNC support. We benefited from the work done by the LEMONADE working group for mobile email.

We also hooked in to MBOXEVENTS and FUZZY SEARCH, building on standards in the backend to create a great experience for our users. We added non-standard conversations support to our server, and there is discussion on the protocol lists about standardising that in a way which is compatible with gmail and other implementations.

Our servers are in New York, and our developers are in Australia. Even on a good day, the ping times are over 230 milliseconds. Anything more than the absolute minimum number of round trips is felt very keenly.

The end result is a very efficient, stateless, easy to use JSON API which already provides a great experience for our customers.

A proliferation of protocols

We are by no means the only people to have this idea. Many companies are building APIs to access email. Here’s just a few of them:

None of these APIs are designed with a primary goal of enabling 3rd party clients to work efficiently, with few round trips and low bandwidth use, and they are all limited to a single vendor. To inter-operate, everything falls back to speaking IMAP and SMTP.

Enter JMAP

JMAP is FastMail’s protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them – making it easy for developers to work with.

JMAP is friendly to mobile devices. By batching multiple commands in a single HTTP request, and having an efficient update mechanism, the radio works less, and battery life is increased. By using the platform push channels, JMAP avoids having to hold its own connection open.

JMAP is friendly to servers. A stateless protocol, there’s no need for the server to maintain a version of the mailbox view that’s out of sync with the current state, as IMAP does, just so that clients can use integer offsets to refer to messages.

JMAP is friendly to the network. Unlike IMAP which can send an unbounded number of unsolicited updates, in JMAP you explicitly ask for the fields you are interested in, and the update command can contain a limit — if there are too many changes, return an error and let the client load the view from scratch. If you’re not caching the entire mailbox locally, then re-fetching a few pages of index listing is better than getting 100,000 EXPUNGED lines down the wire.

JMAP is friendly to multiple clients. In IMAP, if you rename a folder or move messages, then the client which initiated the action can update their local cache, but every other client has to re-fetch everything – there is no unique message ID which follows the message. JMAP uses globally unique IDs for messages, so immutable data can be cached forever.

JMAP does everything. Instead of separate protocols with different syntax for sending email than for receiving it (and separate protocols again for contacts and calendars) JMAP combines them all under one protocol. Best of all, the push notification includes all tokens for each service, so you can synchronise changes to your entire mail account plus your contacts and your calendar in a single round trip — up to date immediately, the way it should be.

Open everything

The JMAP protocol is totally open and unencumbered.

FastMail commits to provide a reference server and reference client, as well as maintaining the specification document in collaboration with others and keeping an up-to-date test suite to allow verification of implementations.

Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP. If the proxy is run somewhere close (in network terms) to the legacy servers, then JMAP will be a better experience than speaking to the servers natively, particularly if you’re on a slow network or the other side of the world.

As more servers start talking JMAP natively, the baseline for interoperability will be raised beyond IMAP+SMTP.

Come and join us at jmap.io and on the mailing list. Together we can build a better way to email.

Posted in Advent 2014. Comments Off

Dec 22: CardDAV beta release

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 21st December was about our file storage system. The following post on 23rd December promotes our standard for better email.

Technical level: medium

After more than a year of anticipation we’re very happy to announce today that we’re releasing CardDAV support into public beta test.

CardDAV is a protocol for reading, writing and synchronising contact data. It’s built into iOS devices and available on Android with an inexpensive third-party application. If you’ve ever wanted to have your FastMail contacts available on your mobile device (and vice-versa), then this is exactly what you want.

Obviously, since this is a beta, there are still a few pointy edges and non-working bits. The most notable thing is that the beta is currently only available to personal accounts, not to business or family accounts. This is because support for shared contacts is not ready yet and there’s some potential for data loss and inconsistent behaviour if you try to use shared contacts without proper support. We’re working hard on finishing shared contact support and hope to make the beta available to business and family accounts within the next couple of months.

CardDAV is only available to Full account levels and higher. Member, Guest and Lite accounts will need to upgrade to be able to use CardDAV.

So now all the disclaimers are out of the way, you can sign up for the CardDAV beta here: https://www.fastmail.com/go/carddavbeta. Instructions for connecting your client are still under development here: https://beta.fastmail.com/help/clients/applist.html.

The contacts story

An address book is a fundamental component of any mail system and FastMail has had one almost since the beginning. It’s always been stored in the MySQL database and available through the web client. For much of its history it’s been confined to the web client. A few years back we did add a read-only LDAP interface, which is useful for desktop mail clients that could support LDAP address books. It works fine, but being read-only severely limits its usefulness. Some time later mobile devices happened, and it became clear that something else was needed.

In 2011 the CardDAV protocol was published, largely developed at Apple to allow device contacts to be synchronised with a server. The protocol is very similar to the earlier CalDAV protocol (which we also use for our calendar) which is good as it allows us to share a lot of code between our calendar and contacts system.

Towards the end of 2012 we started to seriously appreciate the need for both an integrated calendar and device contacts syncing. We weren’t the only ones, as the Cyrus project had started to add support for CalDAV and CardDAV to the Cyrus mail server. We looked at a few options for calendar and contacts and decided that we would implement both on top of the support being baked into Cyrus, and work began in earnest. We decided that calendar was more important because we already had a contacts system and while it wasn’t perfect we preferred to focus our engineering effort on a clear gap in our product lineup. That work took the best part of a year, and we finally released the calendar to production in June 2014. At this point we were able to focus on CardDAV-based contacts.

The actual CardDAV part of this work is actually fairly simple. Unlike CalDAV, the backend server (Cyrus) doesn’t really need much special knowledge. It mostly just saves and loads contact cards as required. Calendar entries are more complicated; the server needs to know about timezones, recurring events, alarms, etc. CardDAV is much easier and if all we had to do was ship CardDAV support, we probably could have done so months ago.

The thing that made it more difficult came from the fact that we already had a contacts system and plenty of code fairly tightly integrated with it. It’s more than just the two user interfaces. The mail delivery pipeline also makes use of user contacts for spam whitelists and distribution lists, so we needed to teach these systems about a whole new storage system for contacts. Up until this time they had simply hit the database for this information. To make matters worse, we always knew that we’d need to roll out CardDAV to users gradually which meant that both the UI and the delivery code needed to be able to work with either. In short, we needed to abstract away the implementation details of the contacts storage, which took a few months to build, test and deploy. We ended up with a nice abstraction based on the JMAP getContacts/setContacts model, with a database provider behind it.

The next step was to write a CardDAV provider for our contacts abstraction. That was actually pretty easy because most of the code needed to access DAV resources was already available from our CalDAV work.

The last piece of the puzzle was the actual data conversion layer. The existing contacts system has a data model that doesn’t match up perfectly with the vCard format used by CardDAV, so we had to develop a mapping. Most of the fields have a 1:1 mapping (addresses, email addresses and phone numbers). What we call “online” fields, however, do not. Our “online” field group includes URLs, Twitter handles and chat IDs. vCard doesn’t group those the same way but more annoyingly, it doesn’t have a standard set of fields for representing these. It took a long time to develop and test a mapping that works most of the time. It’s going to need improvement as we go but it’s not bad for now.

What’s next

The next few months will include a lot more testing, polishing and responding to user feedback and obviously completing the business and family support. That will bring us to a full release where everyone will be quietly and transparently migrated to the CardDAV backend. We can then start to clean up a lot of old code, always a nice thing to do.

If you’re trying the CardDAV beta test, we’d love to hear what you think. Let us know on twitter or by emailing carddavbeta@fastmail.com.

Posted in Advent 2014, News. Comments Off

Dec 21: File Storage

This blog post is part of the FastMail 2014 Advent Calendar.

The previous post on 20th December saw us open source our core javascript library. The following post on 22nd December is the long awaited beta release of our contact syncing solution.

Technical level: high

Why does an email service need a file storage system?

For many years, well before cloud file storage became an everyday thing, FastMail has had a file storage feature. Like many features, it started as a logical extension of what we were already doing. People were accessing their email anywhere via a web browser, it would be really nice to be able to access important files everywhere as well. And there’s the obvious email integration points as well, being able to save attachments from emails to somewhere more structured, or having files that you commonly want to attach to emails (e.g. your resume) stored at FastMail, rather than having to upload it again each time.

Lots of FastMail’s early features exist because they’re the kind of thing that we wanted for ourselves!
It also turns out that have a generic file storage is useful for other features, as we discovered later.

A generic file service

The first implementation of our file storage system used a real filesystem on a large RAID array. To make the data available to our web servers, we used NFS. While in theory and nice and straight forward solution, unfortunately it all ended up being fairly horrible. We used the NFS server built into the Linux kernel at that time, and although it was supposed to be stable, that was not our experience at all. While all our other servers had many months of uptime, the file storage server running NFS would freeze/crash somewhere between every few days and every week. This was particularly surprising to us because we weren’t actually stressing it much, the workload wasn’t high compared to the IO that some file servers perform.

Having the NFS server go offline and people losing access to their files until it is rebooted was bad enough, but there was a much worse problem. Any process that tried to access a file on the NFS mount would freeze up until the server came back. Since the number of processes handling web requests was limited, all it took was a few 100 requests by users trying to access their file storage, and suddenly there were no processes left to handle other web requests, and ALL web requests would start failing, meaning no one was able to access the FastMail site at all. Not nice. We tried a combination of soft mounts and other flags, but couldn’t find a combination that was both consistently reliable and failure safe.

Apparently I have suppressed the memories — something to do with being woken by both young children AND broken servers, but Rob M remembers, and he says: “In one of those great moments of frustration at being woken up again by a crashed NFS server, Bron wanted to do a complete rewrite, and to use an entirely different way of storing the files. Instead of storing the file system structure in a real filesystem, we decided to use a database. However we didn’t want to store the actual file data in the database, that would result in a massive monolithic database with all user file data in it, not easy to manage. So the approach he came up with is a rather neat hybrid that has worked really well.” So there you go.

One of my first major projects at FastMail was this new file storage service. I was fresh from building data management tools for late-phase clinical trials (drugs, man) for Quintiles in New Jersey (that’s right, I moved from living in New Jersey and working on servers in Melbourne to living in Melbourne and working on servers in New York). I over-engineered our file storage system using many of the same ideas I had used for clinical data.

Interestingly, a lot of the work I’d been doing at Quintiles looked very similar in design to git, though it was years before git came out. Data addressed by digest (sha1), signatures on digests of lists of digests to provide integrity and authenticity checks over large blocks of data. That product doesn’t seem to exist any more though.

The original build of the file service was based on many of the same concepts. File version forking and merging (which was too complex and got scrapped) with very clever cache hierarchy and invalidation scheme. Blobs (file contents themselves) are stored in a key-value pool spread over multiple machines, with push to many copies before the store is successful, and a background cleaning task that ensures they are spread everywhere and garbage collected when no longer referenced.

The blob storage system is very simple – we could definitely build or grab off the shelf something a lot faster and better these days, but it’s very robust, and that matters to us more than speed.

Interestingly enough, while the caching system was great when there was a lower volume of changes and slow database servers, it eventually became faster to remove a layer of caching entirely as our needs and technology changed.

Database backed nodes

The same basic architecture still exists today. The file storage is a giant database table in our central mysql database. Every entry is a “Node”, with a primary key called NodeId, and a “ParentNodeId”. Node number 1 is treated specially, and is of class ‘RootNode’. It’s the top of the tree.

Because there are hundreds of thousands of top level nodes (ParentNodeId == 1), it would be crazy to read the key ‘ND:1′ (node directories, parent 1) for normal operations. Instead, we fetch “UA:$UserId” which is the ACL for the user’s userid, and then walk the tree back up from each ACL which grants the user any access, building a tree that way.

For example:

$ DEBUG_VFS=1 vfs -u brong@fastmail.fm ls /
INIT
Fetching ND:504452
/:
--
Fetching UA:485617
Fetching N:20872929
Fetching N:3
Fetching N:1394099
Fetching N:1394098
Fetching N:2
Fetching N:504452
d---   504452 2005-09-14 04:31:35 brong.fastmail.fm/
d---  1394098 2006-01-20 00:44:04 admin.fastmail.fm/
d---        2 2005-09-13 07:46:04 robm.fastmail.fm/

Whereas if we’re inside an ACL path we walk the tree normally from that ACL (we still need to check the other ACLs to see if they also impact the data we’re looking at…):

$ DEBUG_VFS=1 vfs -u brong@fastmail.fm ls '~/websites'
INIT
Fetching ND:504452
Fetching N:504452
Fetching UA:485617
Fetching N:20872929
Fetching N:3
Fetching N:1394099
Fetching N:1394098
Fetching N:2
Fetching UT:485617
/brong.fastmail.fm/files/websites:
----------------------------------
darw  6548549 2007-03-27 06:03:19 cherubfest/
darw 39907741 2008-11-25 10:27:55 custom-ui/
[...]
darw 335168869 2014-10-19 23:51:38 talks/
Fetching NFZ:504465

This structure also allows us to keep deleted files for a week, because when a node is “deleted”, it’s not actually deleted – it just gets an integer field “IsDeleted” set to the NodeId of the top node being deleted.

Let’s try that too:

[brong@utility2 ~]$ vfs -u brong@fastmail.fm mkdir '~/test/blog'
[brong@utility2 ~]$ echo "hello world" > /tmp/hello.txt
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog' /tmp/hello.txt
Failed to write: Could not open /brong.fastmail.fm/files/test/blog for writing: Is a directory 
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog/hello.txt' /tmp/hello.txt
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog/hello.txt'
hello world
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
-arw 387950177 2014-12-21 00:31:56 hello.txt (12)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm rm '~/test/blog/hello.txt'
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
[brong@utility2 ~]$ vfs -u brong@fastmail.fm lsdel '~/test/blog/'
/brong.fastmail.fm/files/test/blog: (deleted)
---------------------------------------------
-arw 387950177 2014-12-21 00:31:56 hello.txt (12)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm undel 387950177
restored 387950177 (hello.txt) as /brong.fastmail.fm/files/test/blog/hello.txt
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog/hello.txt'
hello world

And the versioning:

[brong@utility2 ~]$ echo "hello world v2" > /tmp/hello.txt 
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog/hello.txt' /tmp/hello.txt
Failed to write: Could not open /brong.fastmail.fm/files/test/blog/hello.txt for writing: File exists
[brong@utility2 ~]$ vfs -u brong@fastmail.fm -f cat '~/test/blog/hello.txt' /tmp/hello.txt
[brong@utility2 ~]$ vfs -u brong@fastmail.fm cat '~/test/blog/hello.txt' 
hello world v2
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
-arw 387950597 2014-12-21 00:35:50 hello.txt (15)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm lsdel '~/test/blog/'
/brong.fastmail.fm/files/test/blog: (deleted)
---------------------------------------------
-arw 387950177 2014-12-21 00:31:56 hello.txt (12)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm undel 387950177 oldhello.txt
restored 387950177 (hello.txt) as /brong.fastmail.fm/files/test/blog/oldhello.txt
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
-arw 387950597 2014-12-21 00:35:50 hello.txt (15)
-arw 387950745 2014-12-21 00:36:53 oldhello.txt (12)

This means that if you delete a folder and all its contents, you can undelete it without also undeleting everything ELSE that was deleted in the same action, just by setting nodes with the same IsDeleted field as the top node back to zero again.

[brong@utility2 ~]$ vfs -u brong@fastmail.fm rm '~/test/blog/oldhello.txt'
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
-arw 387950597 2014-12-21 00:35:50 hello.txt (15)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm lsdel '~/test/blog/'
/brong.fastmail.fm/files/test/blog: (deleted)
---------------------------------------------
-arw 387950177 2014-12-21 00:31:56 hello.txt (12)
-arw 387950745 2014-12-21 00:36:53 oldhello.txt (12)
[brong@utility2 ~]$ vfs -u brong@fastmail.fm rm '~/test/blog'
[brong@utility2 ~]$ vfs -u brong@fastmail.fm lsdel '~/test/'
/brong.fastmail.fm/files/test: (deleted)
----------------------------------------
darw 387950021 2014-12-21 00:38:20 blog/
[brong@utility2 ~]$ vfs -u brong@fastmail.fm undel 387950021
restored 387950021 (blog) as /brong.fastmail.fm/files/test/blog
[brong@utility2 ~]$ vfs -u brong@fastmail.fm ls '~/test/blog/'
/brong.fastmail.fm/files/test/blog:
-----------------------------------
-arw 387950597 2014-12-21 00:35:50 hello.txt (15)

So only the file that was in the folder when it was deleted is now present in the restored copy.

That’s the node tree. Alongside this is “Properties”, which are used not only for storing mime types of files (you can edit these in the filestorage) the expansion state of directories in the left view on the Files screen, but can be used via our WebDAV server to set any generic fields you like. There are ACLs and Locks which can also be used via DAV, and a Websites table which is used to manage our customer static-file web hosting.

The cache module can present any file as a regular Perl filehandle, including an append filehandle which already has the old content pre-loaded. Calling “close” on that filehandle returns a complete filestorage node on which other actions can be taken – and it does quota accounting on the fly. The whole system is object oriented and a joy to work with at the top level, despite all the weird bits underneath.

One thing – if you upload over the same filename again and again (webcams for example) we will only keep the most recent 30 copies, because we were finding that it slowed the directory listings too much otherwise.

File storage as a service

A long time ago, if you were in the middle of composing a draft, and you had uploaded a file, we stored that file in a temporary location on the web server’s filesystem. This was fine normally, but if we had to shut down one of our web servers, the session would move to another server, and the email would be sent without the attachment, because it couldn’t find the file.

We now keep uploaded files in a separate VFS directory for each user – and if you’re composing and we switch our web servers, you don’t even notice:

[brong@imap20 hm]$ vfs -u brong@fastmail.fm ls /brong.fastmail.fm
/brong.fastmail.fm:
-------------------
darw   504453 2014-10-18 16:13:19 files/
d-rw  6825201 2014-12-16 21:36:11 temp/

The home directory that you see in the Files screen is “/username.domain/files/” in the global filesystem. Your temporary files go into “/username.domain/temp/” with a special flag that says when they will be automatically deleted (default, 1 week from upload).

This has been very useful for other things too. We now upload imported calendar/addressbook files into VFS first via our general upload mechanism, and just use a VFS address for everything internally, meaning that the upload and the processing phase can be separate.

Calendar attachments are likewise done with an upload as a temporary file, and then a conversion into a long-lived URL later. Even on Mailr, which doesn’t have a file storage service for users, the file storage engine is used underneath for attachments and calendar.

Multiple ways to access

We use Perl as our backend language. We built both FTP and DAV servers, as well as our Files screen within the app. Another very valuable use of filestorage (indeed, one of the original reasons) is that you can copy attachments from emails into file storage, and attach from file storage into new emails. This is a great way to save having to download and upload giant attachments if you want to send them on to someone else.

As you can see from my example above, where my top level had 3 different directories, it’s possible to share file storage within a business. There’s a very powerful access control system to grant access only to individual folders, and separate read and write roles.

You can create websites, including password protected websites. We even have support for basic photo galleries.

Open Source?

A lot of the tooling on top of the VFS is based on open source modules, but hacked so much that they can’t really be shared easily back to the original authors. I have been more lax with this than I should have, particularly the DAV modules where I took the Net::DAV::Server module and almost entirely rewrote it based on the RFC, including a ton of testing against the clients of the day, and adding things like Apple’s quota support.

I don’t have time during this blog series to release anything nice, but there are no real secrets in the tree, so I’ve put up a tar file at http://opensource.brong.fastmail.fm/ME-VFS.tar.gz which contains the current version of the files I’ve talked about in this post, and send me an email at the obvious address if you have comments or suggestions. The file itself is hosted on a FastMail static website!

A road not taken

All of this was built in 2004/2005. These days there are tons of “filesystem as a service” products out there. We could have built one – our VFS has some nice properties which would have made syncing with clients very viable – but our focus is on email, and we didn’t make the time to learn how to build good clients.

We actually have an experimental Linux FUSE module which can talk directly to our VFS, and I built some experimental Windows and Mac clients as well, but we never took it anywhere — though we did talk about the potential to pivot the company into the file storage business.

These days we have gone the other way, with Dropbox integration on the compose screen and hooks to add other services too. It makes sense to integrate with services which do what they are good at, and concentrate our efforts on being the best at our field, email.

Posted in Advent 2014, Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 6,227 other followers