Payment Issues

Recently there have been some problems with our payment processor.

One of these is that sometimes there is a significant delay in completing a transaction, so the funds are “reserved” but the transaction isn’t actually completed until a few weeks later.

The other problem was that a number of charges were made to some of our customers’ credit cards, in our name. The majority of these are USD 1.00 test charges that are used to verify that a card is valid. Normally we cancel these charges immediately and the charge doesn’t appear on the card statement. However the problem at our payment processor has resulted in some of these supposedly “cancelled” charges recently being applied to customer cards. A number of other charges which were supposed to be “cancelled” have also been processed. This was done in error and without any instructions from FastMail.

We are currently working with our payment processor to refund these erroneous payments. In some cases, the refunds are unable to be processed directly because the card has expired or been cancelled.

We may need to refund these charges via an alternative method, or, at the user’s option, instead credit the amount towards future FastMail renewals. We will be in touch with the affected customers once we have more information on the scope of the problem.

We would like to assure our users that we will make certain that all erroneous charges are corrected. This may take some time, so thanks for your patience.

Our apologies for any inconvenience or confusion. We will post more information as it becomes available.
– the FastMail team

Posted in Technical. Comments Off

New phishing trick, data: URLs to avoid forgery reporting

This is a technical post about a new and interesting phishing technique seen today. Regular FastMail users can skip this post.

We saw an interesting new phishing attempt today that uses a relatively novel technique to try and hide the source of the attack and avoid it being reported as a web forgery.

Firstly the email itself looks reasonably well done (apart from the year in the subject being completely wrong), certainly it’s not the poor quality you often see. It looked like this (ANZ is an Australian bank):

phishing

Secondly, the email was sent using a compromised gmail account with a .edu address. In fact there were two separate emails, both from different compromised gmail .edu accounts. I imagine compromised gmail .edu accounts aren’t that easy to get, and this significantly reduced the chances of it being caught by any spam filter.

Thirdly, the phishing page itself is interesting in that it:

  1. Uses a standard link shortener for a redirect (http://ow.ly in this case)
  2. Which redirects to the phishing delivery page (a compromised page on http://zerra-performance-center.de)
  3. That page however rather than hosting the HTML phishing login page directly, does this:

<script type="text/javascript">
        window.location="data:text/html;base64,... base64 encoded version of HTML phishing login page ...";
</script>

That data: URL is itself the phishing page content, which includes links to real ANZ website logos to make it look as authentic as possible, but has a form submit action to a compromised page on http://lucinaracosta.com.br.

This approach is interesting because it makes it impossible to report this page as a forgery using the standard Firefox "Report Web Forgery" action because Firefox thinks it’s a data: URL. Neat trick that makes it harder to remove or block in the long run.

I’ve reported this issue as a Firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1032564

Posted in Technical. Comments Off

Errors on classic mailbox screen and pop emails retrieved again for some users

A rollout of some new code today contained some errors that badly affected two separate areas of FastMail.

1. Errors selecting any emails to action on the classic interface would cause a fatal error

An internal misuse of an API meant that selecting any emails on the classic interface Mailbox screen and trying to action those emails would fail with a fatal error. Reading and applying actions to individual emails continued to work fine. The way this manifested itself unfortunately wasn’t picked up in our testing before being rolled out. It has been fixed now and we’ll update our tests to catch this. The error lasted for for about 3 hours.

2. Pop links for some users re-downloaded all emails again

A long term bug in the pop retrieve system resulted in a very rare and intermittent problem where some links for some users would forget all existing downloaded message. This means that in certain cases users that had set the “Leave on server” option might see existing messages that had been downloaded previously downloaded again, possibly several times. This obviously resulted in duplicate copies of the same message appearing in a folder.

Unfortunately a fix to this bug rolled out to one server for a short time actually made things worse, causing the same problem to occur for more users than the original bug.

A correct fix for the original problem and the subsequent bug has now been rolled out everywhere.

Users affected by this bug can find and remove any duplicate messages using the Advanced -> Folders -> Mass delete/Download/Remove duplicates … (button down the bottom) screen. Select the folder with the duplicates at the top, and use the “Remove duplicate messages” section to find and remove any duplicate messages.

Posted in Technical. Tags: . Comments Off

Increased spam getting through for the last few days

Due to an undetected compatibility issue between some software modules we use for detecting spam emails, for the last few days a number of the tests we use to detect spam haven’t been working properly. This means that for the last few days, considerably more spam may have been getting through our filters and into users Inboxes.

We’ve now fixed this issue and have added additional tests to ensure this doesn’t happen again.

For those interested in the technical details: We upgraded to a newer version of Net::DNS and the version of SpamAssassin we use was using some internals from Net::DNS that had changed with the new version. This caused all RBL lookups to fail. Failing RBL lookups wouldn’t cause any email delivery to fail, just all RBL scoring to be ignored.

Posted in News, Technical. Comments Off

New IP addresses in Iceland

We have just transitioned our Iceland servers to a new network range (belonging to us rather than to Opera).

This means that the addresses of ns2.messagingengine.com and in2-smtp.messagingengine.com have changed.

Unless you have hard-coded the old addresses somewhere, you shouldn’t see any difference. The new addresses will propogate over the next hour or so.

Posted in Technical. Comments Off

SSL certificates updated again

A few days ago we updated our SSL certificates. The algorithm used to sign these certificates (SHA256) presented problems with some older clients and operating systems, notably WebOS and Nokia devices. To fix this we got our CA (DigiCert) to re-sign the certificates using the older SHA1 algorithm, which should work pretty much everywhere. These certificates are now live on all of our domains.

Most users should not notice any change. If you are on a device or client where you’ve had to install the DigiCert root certificate in the last few days, you may need to do this again as these certificates are signed from a different root certificate. If that affects you, the root certificate is available from DigiCert’s root certificate page and is called “DigiCert High Assurance EV Root CA”. If you also need the intermediate cert, its available from the same page with the name “DigiCert High Assurance CA-3″.

Posted in Technical. Comments Off

When two-factor authentication is not enough

TL;DR: This is the story of a failed attempt to steal FastMail’s domains.

We don’t publish all attempts on our security, but this one stands out for how much effort was put into the attack, and how far it went.

We’ve had a handful of minor attack attempts recently. Targetted phishing emails to staff trying to steal credentials. An NTP-based DDOS which was quickly mitigated by NYI, our excellent hosting service.

These sorts of attacks are the “background radiation” of the internet. Along with port scans and entries in the web server logs from malware trying us out to see if we’re vulnerable to old PHP bugs (hint, we’re not). It’s the reality of being on the internet.

This blog post was first drafted before the Heartbleed fiasco. Sometimes, no matter how careful you are, you get a nasty surprise. We responded very quickly, as always. Anyway, on with this story.

About a month ago, our hostmaster@fastmail.fm account suddenly wound up subscribed to hundreds of mailing lists. All these mailing lists failed to use double or confirmed opt-in, so someone was simply able to enter the email address into a form and sign us up, no confirmation required. This really is poor practice, but it’s still pretty common out there. A special shout-out goes to government and emergency response agencies in the USA for their non-confirmation signup on mailing lists. Thanks guys.

The upshot was that the hostmaster address was receiving significant noise. Rob Mueller (one of our directors) wasted (so we thought) a bunch of his time removing us from those lists one by one, being very careful to check that none of the ‘opt-out’ links were actually phishing attempts. This turns out to have been time very well spent.

Internet identities

At FastMail, our security is central to the safety of our users. Given access to an email account, an attacker could reset passwords on other sites, including those which allow spending real money.

We take this responsibility very seriously, and we’re always looking for ways to improve our security.

Two factor authentication (2FA)

The Domain Name System is one thing that’s even lower layer and more central to identity security than the email server itself.

Based on recent articles in the tech press, we really wanted to have ALL our domains protected by two factor authentication.

Our domains were historically spread amongst multiple registrars. We chose to consolidate with Gandi because they have a great slogan (“no bullshit”), they support 2FA, and they support all the top level domains we require.

Robert Norris, one of our sysadmins, was in charge of the migration. He set up a corporate account with Gandi to get assistance in transferring the domains, and set up two-factor authentication at the same time.

Gandi uses the popular OATH TOTP (also known as “Google Authenticator”) mechanism. Rob wrote a small TOTP client and placed it with the key on our management servers in the secure storage area where we also keep our SSL certificates. The account password itself was encrypted in our password manager, which is stored separately.

Only a small number of trusted people have access to the credentials for our Gandi account. We are satisfied that this level of security is strong enough.

The attempt

On March 19th, this came to hostmaster@fastmail.fm:

From: Gandi Corporate Team
To: hostmaster@fastmail.fm
Subject: [RN1374-GANDI] Email address update request
Date: Wednesday, March 19, 2014 8:27 PM

Hello,

We received an email update request for the account RN1374-GANDI.

Previous email address: hostmaster@fastmail.fm
New email address: fastmail.fm@qq.com

If you are opposed to this modification, thank you for letting us know
only by replying to this email.

If you can read this message, then you can recover the password of your
account, and thus modify the email address of the handle. In that case,
we won't take care of your request.

Without any reply from your side, we will proceed within 24 hours.

Best regards,

Gandi Corporate

The hostmaster alias actually forwards to three of us, and we were all hyper-alert, so we thankfully noticed this email.

Within twenty four hours.

One day.

Gandi assure us that their fraud detection systems would have detected this, but for the 2 weeks it took from this email until we had full control over our account again, we were worried.

This request had completely bypassed our two-factor protection.

Forged source addresses

There is a well known problem in network security. You can’t trust the source address of an IP packet – they are trivially forged.

It’s the reason why we have source port randomisation, sequence number randomisation… all the things designed to stop an attacker being able to forge both an initial SYN packet and also the response to an ACK packet to bring up a TCP connection.

While they can falsify the source of a request, an attacker without full network control can not receive the response to their forged packet and continue the handshake.

This is why this email was such a surprise. Like the poor quality mailing lists mentioned above, it didn’t require a confirmed opt-in. We had to reply to say that we didn’t want the contact email address changed.

This means that a forged source address was sufficient. Even though the attacker couldn’t read email to hostmaster@fastmail.fm, they didn’t need to. All they needed was for us to not read it.

To Gandi’s credit, they responded very quickly to our “NO, DON’T CHANGE IT” email, and locked our account to stop any further shenanigans while they investigated and collected more documents from us.

Falsified documents

We discovered that Gandi received a paper email change form (pdf) claiming to be from a “Robert NORRIS” (the name which appears on our whois data), along with pictures of a passport of said “Robert NORRIS” and company registration documents also claiming to be for FastMail Pty Ltd.

At the time of writing, we are still in debate with the Gandi Legal Department about whether they can even show us these documents. They claim that French Law forbids them from showing us documents which purport to be from us. This is something to be aware of when choosing an vendor – different companies operate in different jurisdictions. There’s also a certain degree to which the conservatism of legal departments (protect the company as much as possible) conflicts with the corporate motto (“no bullshit”). The first response we got was certainly bullshit – “in order to meet a legal or regulatory obligation”. We challenged them to give an actual legal obligation and were given Article 226-15 of the French Criminal Code, along with rough English translation as follows:

“The act, committed in bad faith, of opening, deleting, delaying or diverting correspondence, whether or not it arrived at its destination, and addressed to a third party, or to fraudulently gain knowledge thereof, is punishable by one year of imprisonment and a fine of 45,000 EUR.”

We don’t believe that law is relevant – it’s the “no interception” law that exists everywhere, and doesn’t forbid anyone from quoting documents in replies to the purported source of those documents. If the law really was as Gandi Legal seem to be interpreting it, it would be illegal to quote an email in your response unless you were certain that the source address hadn’t been faked.

Was this a “security flaw”?

Security is built in layers, and I would definitely say that the fact that we received that email means one of the layers was weaker than it should be. Partly it’s poor choice of wording (Gandi claim that they would not necessarily have changed the email within 24 hours, depending on other investigations).

It still would have been necessary to either disable or reset the two-factor authentication on our account as well for the attacker to get full control. That’s difficult, but not necessarily impossible. After the fact, there’s no way to know how it would have gone down. We certainly weren’t willing to take the risk of doing nothing and seeing what happened!

What we do know is that the attacker was very determined, and willing to go as far as forging documents while simultaneously generating noise to make us less likely to notice the attack. They must have figured they had a chance.

Improving security for the future

A disadvantage of adding something like two-factor authentication after the fact is that you may miss the interactions with your existing processes. Gandi’s paper “email reset” form makes a lot of sense in the world where most of their customers are individuals or small businesses with one or two domains, and using addresses that they may lose access to. With no other factors, if they lose access to the email address and forget their password, there needs to be a process to regain access.

It’s always great to have a consistent process. Having a consistent process means that attackers can’t just try their luck until they find someone who is more trusting than average. Australia has a fantastic system called the 100 point check for authenticating people. We like process, consistently applied.

The problem we have is that we didn’t expect that the account email address could be changed without any reference to our two factors at all. Maybe nobody at Gandi realised either. That’s a security flaw – even if it doesn’t mean everything is totally broken.

We have had some very frank discussions with Gandi over the past week, and they agreed to make all three of the improvements we proposed as a result of these events:

  • the setting “disable password resets via email” was not on the security settings page of their website. Because of this, we hadn’t discovered and enabled it. They are moving it to the security page.
  • if an account has 2FA enabled, a red flag will automatically be raised against the request, meaning significant extra investigation will be done.
  • if an account has 2FA enabled, then an active confirmation will be required from the owner of the account before changing the email address. This means it will be harder to regain access if you lose all your factors, but that’s a good thing! Turning on 2FA means you want it to be hard for anyone who doesn’t have those two factors to gain access.

These steps will make attacks against Gandi accounts even more difficult in future, and we applaud their efforts to improve security and willingness to listen to our concerns.

There is one other measure that we have suggested which is still under discussion. Requiring the TOTP code to be entered on the password reset form, rather than using a secret question. We believe secret questions are bogus security, and we have an appeal to authority to back us up.

Gandi have blogged about this as well, and also given some general advice on keeping your account with them secure.

Conclusion

FastMail came out of this attack unscathed. Our domains are now even more secure, because Gandi has tons of proof on file about who we are and who our company is. Also Gandi’s processes have become more secure as a result of our experiences, so we are confident that we can safely keep our domains with them.

An important lesson learned is that just because a provider has a checkbox labelled “2 factor authentication” in their feature list, the two factors may not be protecting everything – and they may not even realise that fact themselves. Security risks always come on the unexpected paths – the “off label” uses that you didn’t think about, and the subtle interaction of multiple features which are useful and correct in isolation.

Posted in Technical. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 5,809 other followers