Background

crikey.jpgAround 9pm U.S. PST Sunday evening, Atlassian detected a security breach on one of our internal systems. The breach potentially exposed passwords for customers who purchased Atlassian products before July 2008. During July 2008, we migrated our customer database into Atlassian Crowd, our identity management product, and all customer passwords were encrypted. However, the old database table was not taken offline or deleted, and it is this database table that we believe could have been exposed during the breach.

The impact

What’s affected?
If you have an Atlassian account from before July 2008 — you should definitely change your password with us. Also, if you used that username/password combination for any other site, we recommend you change it there as well to prevent people potentially gaining access to other systems.
What’s not affected?

  • If you created your Atlassian account after July 2008, you should not be affected. We notified you via email because we feel it’s important every customer is aware of the situation.
  • If you’re an Atlassian customer running our products behind the firewall, your passwords are not affected. By default, our products store the passwords for the userbase in an encrypted form.
  • If you’re an Atlassian SaaS or hosted customer, your userbase and customer data are not affected. Those are stored in another system.
  • No credit card or payment details were accessible.

Lessons we’ve learned today

Firstly, we made a big error. For this we are, of course, extremely sorry. The legacy customer database, with passwords stored in plain text, was a liability. Even though it wasn’t active, it should have been deleted. There’s no logical explanation for why it wasn’t, other than as we moved off one project, and on to the next one, we dropped the ball and screwed up.
Secondly, in attempting to be as open as possible, we sent a communication to all customers — not just those affected — at once. Our intent was to keep everyone notified of the situation, but we appear to have done more to raise alarm with unaffected customers. Beyond that, with hundreds of thousands of accounts changing passwords simultaneously, our web servers crumpled — causing yet more user alarm. We apologise for the extra consternation this caused — our web servers are now back purring along as normal.
In hindsight, we should have reset passwords for affected users on their behalf. This would have avoided the unexpected transactional load on our web servers, and communicated the problem to the rest of our customers in a different way.
For those affected users who haven’t changed their passwords already, we’re now resetting them for you.

What we’re doing next

  • Research: we are feverishly researching the breach. Once we’ve concluded our investigation, we’ll provide another update. Again, no credit card, financial or SaaS-customer information was accessible or exposed. The worst case here, which we take very seriously, is that the password used by customers that purchased before June 2008 to logon to http://my.atlassian.com was exposed, which is why we encouraged customers to change it immediately.
  • Disclosure: in terms of the security breach itself, we will disclose the various attack vectors and what happened once we have a full picture. Expect this in the coming week.
  • Intent: we’re sure at this time that this was a targeted attack. We’re not sure if it was malicious, or just some kids playing around. Either way — it was a breach. We’re trying to determine the exact purpose of the breach.
  • Openness: even though we suspect that very little was actually compromised, and despite the difficulty of the situation — we strongly believe in our company value of “Open Company, No Bullshit”. Times like these are where your values get tested and we’re determined to live up to them today to the best of our abilities. As such, we’re being as open as possible here, and not bullshit any customers.

In summary — we’ve made mistakes, we’re sorry and we’re fixing them — and we’re going to be honest about what those mistakes are. Half of being a reliable and trustworthy vendor from a security perspective is the technical bits, and even though we erred here, we ultimately pride ourselves on how we handle security. The other half is being open and honest, which we’ll never fail at.

Questions?

If you have a question, you can respond below, contact our Customer Service team or email me directly — we’d love to hear from you — good, bad or ugly.

2010-04-13 Update – More password resets

As we continue our investigation, some Atlassian systems may experience downtime. Additionally, if you are having difficulty accessing an Atlassian system, your password may have been reset per the note below. Please select the “Can’t access your account?” or “Forgot your password?” link on the system you are trying to access.

2010-04-16 Update – Security advisory disclosure

As part of our ongoing investigations, we have posted a JIRA security advisory that details two separate security vulnerabilities that affect all supported JIRA versions, and provides associated fixes. We strongly recommend that customers with instances of JIRA accessible to the public internet immediately follow the recommendations outlined in this advisory. We thank the Apache Foundation and Codehaus for their support in helping us identify and address these vulnerabilities.

2010-04-17 Update – Research findings & further attack update

As the investigation continues with our internal IT team, security consultants and law enforcement agencies – we have new information to share with our customers.

Attack update - As part of the sustained, targeted cyber-attack this week our customer service system was also compromised. We have now determined that the attacker(s) may have taken a copy of this server’s database. This may give the attacker(s) access to read any text contained within customer service cases, as well as usernames and encrypted passwords. It is important to note that no customer data from customer attachments, customer backups or logs was downloaded or accessed by the attacker(s) – nor was any information exposed on the systems where we store our credit card or financial information.

Atlassian actions update - we are taking this incursion extremely seriously, and we have done the following:

  • Pro-actively notified (and are working with) affected and high-risk customers to help them protect their systems
  • Issued JIRA Security Advisory – 2010-04-16, security patches for all supported versions of JIRA and related documentation and guides to help our customers
  • Reset passwords on every one of our systems – including the customer service system – whether affected or not
  • Verified the customer service database itself was not altered, none of our source code repositories were accessed, and no information was exposed from the systems we use to store our credit card or financial information
  • Engaged and are continuing to work with US and Australian law enforcement agencies to investigate the incident
  • Hired incident response security experts (in US and Australia) for deep forensic analysis
  • Increased overall security of our public-facing systems in conjunction with security experts and our research

We take your security very seriously. If you have questions or have information that might further help us or law enforcement, please contact us immediately at security@atlassian.com

Lastly – I must personally say how proud I am of the way our company has reacted to an extremely difficult and trying situation. In the last 96 hours our people have been working around the clock to investigate the incident, address vulnerabilities and secure systems – always putting our customers first. The number of Herculean efforts taking place within the world of Atlassian is simply incredible. To see the steely fight this has brought out in our team, their belief in Atlassian’s values and the absolute dedication to not fucking our customers has on more than one occasion brought me to tears over the last few days. This is a truly rare, special place to work. Thank you everyone.