Copyright © 1999–2018 FastMail Pty Ltd
Ever since the early days of FastMail, we've offered the ability to share email folders between users. When we added Calendar and Contact access via CalDAV and CardDAV, the ability to share between users in your family or business was a key feature of our platform.
Until now, you have only been able to share with users on the same
domain - for example
fastmail.com or your own personal domain. If
you had a business with multiple domains, your options were limited
by the contraints of our system.
From today, all users are equal - you can share with everybody in your family or business, no matter what domain they use.
If you are interested in the technical details behind this change, read on!
NOTE: due to the limitations with the current mail.messagingengine.com service, all '.' characters in domains and folders will appear as '^' via IMAP. They appear correctly in our web interface.
Cyrus IMAP virtual domains
Under the hood, we're using the Cyrus IMAP server. Until recently, it didn't support sharing across domains at all. You may have read about one of the steps during our advent blog series last year. Having a fast reverse-ACL lookup allows us to search the entire mailbox list quickly.
This is important, because the mailbox list is sorted by domain first. The format of a mailbox in the mailboxes.db file is like this
So the LIST command could use a prefix search for
reduce the number of records loaded. This kept things relatively fast
even on busy servers.
For horrible legacy reasons, Cyrus supports two different types of
virtual domains. We use the one called
userid - where domains are
based on the username you log in with rather than the IP address you
connect on. If you're an admin, you could always see users in other
domains, but it used a non-standard way to display the mailboxes,
so my listing would look like:
No normal mail client would cope with that. You can see why it was
chosen though, the alternative of
is totally ambiguous.
Unix Hierarchy Separator
There are two other switches within Cyrus that change LIST behaviour,
unixhierarchysep. If you use
altnamespace, the names of your folders are transplanted up a level,
while shared folders and other users are batched together in specially
It's easiest to explain with some examples. I'll do them without domains for simplicity:
shared.news user.aaearly user.brong user.brong.Trash user.brong.a.deep.folder user.friend user.friend.foo zzlate
altnamespace: false / unixhierarchysep: false
INBOX INBOX.Trash INBOX.a.deep.folder user.aaearly user.friend user.friend.foo shared.news zzlate
altnamespace: false / unixhierarchysep: true
INBOX INBOX/Trash INBOX/a/deep/folder user/aaearly user/friend user/friend/foo shared/news zzlate
altnamespace: true / unixhierarchysep: false
INBOX Trash a.deep.folder Other Users.aaearly Other Users.friend Other Users.friend.foo Shared Folders.shared.news Shared Folders.zzlate
altnamespace: true / unixhierarchysep: true
INBOX Trash a/deep/folder Other Users/aaearly Other Users/friend Other Users/friend/foo Shared Folders/shared/news Shared Folders/zzlate
You'll notice some things there about sorting order of the three namespaces
(INBOX, user, shared) - that the shared folders have no prefix at all (just
user). An exciting number of things to keep right when making any
changes, and I've been working hard to simplify these parts and place them
behind reliable interfaces for a while now.
That pesky dot
Like many things in the IMAP standard, the hierarchy delimiter was a compromise between the netnews style dot and the filesystem style slash. One of the reasons for unixhierarchysep was to allow dots in user names, so I could have for example:
Nice, but how do we store that internally? The answer was to choose a character which was not common in mailbox names, and disallow it - then map the name. So internally you would see:
A way to represent domains
One nice thing about this namespace is that if you allow '@' in names as well, you can have cross domain support! In fact, if you turn off virtdomain support entirely, you can get that:
And that will correctly render as:
But we had virtdomains turned on already, and we needed to keep backwards compatibility.
Exposing the internals
I thought this was all going to be impossible, but then I realised that
I could take advantage of that internal representation. In the
unixhierarchysep world I could just translate all dots in the
individual elements and the result would be consistent, and look good when
we switch to unixhierarchysep in the future.
After that, it was a simple matter of a ton of programming and writing test cases to make sure everything else still works!
Some of these internals are still visible. If you log in via IMAP you will see funny looking names. We do the translation in our web interface to keep the folder names and usernames pretty, but we won't have those via IMAP until we switch on unixhierarchysep for users, which is coming with the security and multi factor authentication changes that are planned for later this year.
So here's what it looks like:
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE XAPPLEPUSHSERVICE AUTH=PLAIN AUTH=LOGIN SASL-IR] sloti30t15 Cyrus IMAP 3.0.0-beta1-git-future-12410 server ready . login firstname.lastname@example.org ... . LIST "" * * LIST (\HasChildren) "." INBOX * LIST (\HasNoChildren) "." INBOX.AICSA * LIST (\HasChildren \Archive) "." INBOX.Archive [...] * LIST (\HasNoChildren) "." INBOX.uidtest6 * LIST (\HasNoChildren) "." user.test2 * LIST (\HasNoChildren) "." user.testcrossdom@gondwana^name.Archive . OK Completed (0.020 secs 78 calls)
(yes, I own gondwana.name)
Again because of backwards compatibility, you can see that email@example.com's INBOX has no domain, because that's how FastMail sharing already works. I had to revert the "always show domain" behaviour which is preferred for consistency.
Show me the code
The main commits are:
- Adding crossdomain support
- Allowing '^' in non-uhs mode
- Supporting existing FastMail same domain bare userids
The end result is simpler for users to understand and consistent. When we have an IMAP mode that uses unix hierarchy separators, it will be very nice to use. There's still work to do on making Alt Namespace better, but I'll write about that another day.