Synchronise IMAP folders on login option

FastMail likes to encourage people to find the best way to access their email. This might mean that at home you setup an email software like Outlook Express to access your email via IMAP, while at work on on the road, you might use the web interface.

One issue with this has been that FastMail keeps a list of folders in the web interface that’s separate to the IMAP server. And while it would synchronise the web listing with the IMAP server when you go to the Options -> Folders screen, it still confused people regularly when they’d create a folder with their IMAP client, and then login to the web interface and couldn’t see the folder!

Part of the reason that we didn’t sychronise the web listing automatically on login was that listing the folders on the IMAP server was a relatively expensive command that could take some time to complete. However recently we came up with some optimisations that made the listing of folders quicker. So we added some code to synchronise the folder list on login, and also to display a short welcome message between logging in and transferring to the mailbox screen.

Despite the optimised sychronisation time, the slight increase in login time has annoyed some users, especially those that don’t use IMAP clients and thus have no need to synchronise the folder list regularly. Because of that we’ve added an option on the Options -> Account Preferences screen, a “Sync folders on login” checkbox. If you uncheck that checkbox, then it disables the folder synchronisation and removes the login redirect message. This returns the login process to exactly as it was previously. You can still manually synchronise the folder list by going to Options -> Folders screen.

Posted in News. Comments Off

Improved photo gallery released

Thanks to some great development work by Neil Jenkins, and after some extensive beta testing, we’ve now released our improved photo gallery feature.

Creating a gallery is easy, just create a directory area in your file storage and upload some photos, then use the Websites feature to create the gallery. You can easily upload photos to your file storage from Windows XP, Windows Vista or Mac OS X (use dav.messagingengine.com as the hostname).

This new photo gallery completely replaces the two existing photo gallery options with a single new clean and modern gallery style. It uses advanced features on modern browsers to quickly allow navigating between photos and switching between grid, filmstrip and image view modes, and includes a slideshow mode to play through a series of photos. It also degrades gracefully on older browsers so it should be accessible anywhere.

To discuss the new gallery, please see this forum thread.

Posted in News. Comments Off

Upgrading to Cyrus version 2.3.10

Cyrus version 2.3.10 is almost ready to be released.  As part of the preparations for release, we’ve been porting our Cyrus patches forwards and testing that it will all work correctly.

http://cyrus.brong.fastmail.fm/

The most exciting part of this new Cyrus release is full SHA1 GUID (unique identifiers) on each message.  This is a development which has grown out of our earlier work to use partial MD5 hashes as the UUID value in previous Cyrus releases instead of a locally calculated value, allowing better integrity checking across our replicas.  The SHA1 will provide much stronger cryptographic guarantees and simplify our backup infrastructure considerably as well (eventually – first there’s the changeover which will cause extra IO load on the backup server too – thankfully it has plenty of spare capacity, both disk and CPU)

We provide the totally non standard “FETCH GUID” command to allow this SHA1 value to be seen through IMAP as well (this is not in upstream Cyrus and is subject to change)

However – there is a downside.  The new index format (an additional 8 bytes per record) will cause every single mailbox to be “upgraded” when it is first accessed after the upgrade.  To ease the load on our servers, we will be phasing this upgrade in slowly, one store at a time.  As each store is upgraded, there will be a short downtime (approx 1 minute) followed by higher IO and hence slower access for all users until the indexes are upgraded.  The first time you access your mailbox after the upgrade will be particularly slow, as all your indexes get upgraded!

We will be running a background task trickling through and pre-upgrading mailboxes as well.

I’ll update you again, here and on the forum, when this process is finished.

Posted in Technical. Tags: , , , , . Comments Off

Emails and dates/times

There’s often some confusion with people regarding emails and the dates/times (datetime) shown on the email. The main problem is that there’s actually two main different datetimes associated with every email. There’s the sent datetime and the received datetime.

In many cases, these values are sufficiently close (a few seconds) that it doesn’t really matter which you see or use. However in some cases (eg delayed email, or email retrieved from another server to FastMail via a Pop Link) the datetimes can be significantly different, causing some confusion.

Mostly this revolves around that emails on the Mailbox screen show and are sorted by the received datetime, but if you view the headers of an email, the “Date:” field that most people see is actually the sent datetime. In fact historically, the received datetime is something that is created and stored internally in the email server itself when the email is delivered, it’s not visible as a field in the email headers at all.

Because the received datetime is only stored internally in the email server separate to the email itself, a number of things can cause this information to be lost, such as restoring emails from a backup, copying emails from one server to another, POP retrieving them from the server, etc.

To avoid the confusion this was causing, we decided to try and come up with a more consistent way of representing the internal date that would be stored with the email itself.

We decided to do this by noting that when an email is delivered to one of our email servers, the final hop in the email “Received:” headers shows this delivery step as something like this:

Received: from compute1.internal (compute1.internal [10.202.2.41])
     by store123m.internal (Cyrus v2.3.9) with LMTPA;
     Sat, 13 Oct 2007 04:27:43 -0400

The final part of that header shows the exact time the email was delivered to the mailbox (as per the definition of Trace fields in RFC2822). By using the time in the final “Received:” header, we basically have something that is constant in the email itself as something to reference when an email is moved, or restored from backup, or POPed from another server. This does have one potentially negative side-effect related to using IMAP to APPEND messages to a mailbox, but this is both a relatively rare case, and also we still believe the consistency of using the last “Received:” as the received datetime reference outweighs the potential negative.

At the same time we did this, we also wanted to allow users the option of when they used POP Links to retrieve emails from another server to use either the datetime the email was actually downloaded from the POP server as the received datetime, or use the “Date:” header on the email (eg sent datetime) as the received datetime. The way we decided to do this was by allowing a special “X-DeliveredInternalDate:” header. If that header is present, then it should be a datetime value in the standard RFC format, and that value will be used as the received datetime instead of looking at the datetime in the last “Received:” header. This provides flexibility in setting the received datetime, but at the same time it ensures that the received datetime won’t be lost by internal movements in and out of the server. The use of this is controlled on the Pop Links screen on each Pop Link by the “Received date” options which can be set to either “When POP occurs” or “Message sent date”.

This setup has been in place for about 6 months now and worked very well, giving a good balance of consistency and flexibility. The only issue that has really surfaced is that some programs use IMAP to APPEND emails with no “Received:” headers at all (eg Bynari Connector).

Posted in Technical. Comments Off

Mobile email/teleflip.com blocked (updated)

Update 11-Oct-2007: teleflip.com have contacted us and changed over to using SSL IMAP connections rather than screen scraping web pages, a much better approach to retrieving emails. Thanks teleflip.

I just noticed that one of our web servers was getting noticeably more loaded than the others. After some investigation, I tracked the problem down to a site called teleflip.com. Basically it appears they provide a way to get and send email on your cellphone in the US. A nice idea, but it seems the way they implement at least the “get” part is horrible and unacceptable.

Rather than using standard email protocols like POP or IMAP which are designed to download emails, especially IMAP which lets you download just the important meta-data for emails such as From address, Subject, Date, etc, and then get the parts of an email you actually want, they seem to be logging into our web interface and paging one page at a time through users mailboxes, some of which are 1000′s of pages long.

In fact from our logs, I see that they appear to be logging in for only 8 different users, but appear to be generating over 1,000,000 web requests a day! This is absolutely insane behavior. And to top it off, rather than using a User-Agent field that identifies them as a bot type site, they’re trying to hide themselves as a standard version of Firefox using “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050518 Firefox/1.0.4″.

Unfortunately because of this activity, I’ve had to block all the teleflip.com IPs (64.74.168.128/25). I don’t intend to unblock them, because using the web interface to scrape email data is completely the wrong approach. They should be using POP or IMAP, and if they had some real sense, they’d use IMAP IDLE to actually get notification of when new emails arrived so they could push just those new emails to your phone.

Until teleflip.com improve their systems, we recommend that you do NOT use them.

For users looking for the best possible mobile email experience, we recommend using ChatterEmail with a Palm Treo. ChatterEmail is a full IMAP client on a phone, including IDLE support to handle push email events. For Windows Mobile users, we recommend looking at FlexMail. It’s not quite as powerful as ChatterEmail, but seems to be the best email client for Windows Mobile phones at the moment.

Posted in Technical. Comments Off

HTML editors and some help testing FCKeditor 2.5

For quite some time now, we’ve wanted to change the HTML editor that the FastMail web interface uses for the compose screen when you’re in HTML editing mode. By default, we’re still using the very old htmlArea, although you can choose to use either Xinha or FCKeditor from the Action menu (you have to switch to text editing mode first to make the change, but then any change you made is sticky for the session, and across logins).

The reason we’ve stayed with htmlArea is that we’ve been waiting for a compelling reason to upgrade. htmlArea continues to work, and while the other HTML editors have been developed and have had features added, they’ve also had additional problems and bugs grafted onto them as well that have in many cases negated the additions.

I’ve been keeping an eye on Xinha and FCKeditor (and also other editors like Tiny MCE), and have been waiting for the right release to upgrade. FCKeditor 2.4 was looking very promising. One of the features I really wanted to use was the new EnterMode option which allows you to control if the Enter key generates either a <br>, a <p> or a <div> tag. This turns out to be surprisingly important because of the way HTML works versus the way people expect WYSIWYG style editors to work.

The problem resolves around two conflicting observations

  1. When people hit Enter, they want the cursor to drop directly to the next line and not leave any vertical gap
  2. When people hit Enter, they expect a paragraph of text to created and treated as a “block” of text for block level formatting items (eg indent, right-to-left display, etc)

The problem is that in HTML, to create a new “block” of text, you use <p> markers around the text. Unfortunately by default browsers includes some vertical spacing between blocks of paragraph text. So if you set EnterMode=p, then when you hit Enter, it does create a new <p> block, but also the cursor moves down what appears to be around 1.5 lines, rather than directly to the next line. Now you can explicitly set the spacing on paragraphs so that it bunches paragraphs directly underneath each other with no gap, but you need to apply that to the whole document, which means that if you reply to existing text that contains <p> tags, those will all be bunched next to each other as well, even if that’s not how the original author intended it and can make it hard to separate out paragraphs. Basically there’s a disconnect between the semantic nature of the HTML and the way it’s displayed.

Going the other way, if you use <br> markers, these don’t create new “blocks” of text, they merely designate points in the big block of text where to insert a line-break. While this might sound fine, it creates problems when you then try and perform actions on blocks of text (eg center, or turn into a list, or change to right-to-left text display, etc). Despite these issues, the fact that inserting a <br> moves the cursor immediately to the next line which is what most people expect, is the reason most online HTML editors use this approach.

The final approach is to use <div> markers. In fact, div markers are an excellent compromise. They do create new HTML blocks making formatting easier, but by default don’t include any extra vertical spacing, so they create the “cursor drops down only one line” effect. Interestingly, the HTML editor in Outlook Express has always used <div> markers when using the Enter key, and in most cases it creates exactly the effect people want.

So when FCKeditor 2.4 added the ability to set EnterMode=div, I thought this was going to be a great solution. Unfortunately it turned out that EnterMode=div was very buggy, making it basically useless in 2.4.

It appears that FCKeditor 2.5 (which is currently in beta testing) has had a lot of work into fixing the text entry and tag generation bugs. I’m hoping that this release will be stable enough to finally replace htmlArea. Additionally FCKeditor 2.5 includes Opera and Safari support, so users of those browsers should be able to do HTML editing in the web interface as well.

I’m hoping people can give the 2.5 beta release a try, and I’ve started a forum thread (http://www.emaildiscussions.com/showthread.php?t=50374) where people can provide feedback so I can get a general sense of good this release is looking.

Posted in Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,716 other followers