DNS Outage Report

On July 22nd, at approximately 11:45 PM PST, we were alerted to DNS resolution problems in our infrastructure.

We checked with our DNS provider, Zerigo, and they were indeed undergoing a targeted and sustained distributed denial of service attack (DDoS).

At approximately 1:15 AM PST, DNS issues seemed to be resolving. Unfortunately, this turned out to be a false positive and we quickly pointed our root records at the Zerigo servers not being DDoS’ed.

Most issues cleared by 2:00 AM PST as root DNS servers picked up our changes fairly quickly. af.cm took the longest to update, and unfortunately our AppFog APIs rely on these properties to be able to resolve.

Issues cleared for most users by 6 AM PST as DNS servers across the world picked up our updates. Propagation was reasonably quick for most users – though some users reported slow pick up from their own ISPs.

DDoS against DNS providers is a frequent attack vector and one that needs to be planned and architected for. We are immediately implementing a new, improved redundant (multi-vendor) DNS services to avoid this situation in the future. With the work we have underway to develop redundancy and avoid single points of failure (begun after the recent AWS outages), we feel we are well-positioned to avoid these issues in the future.

Thank you so much for your understanding and support through this tough period. It means a lot to us.

If you have any further questions or concerns and would like to set up a time to talk in more detail please email experience@appfog.com.

Follow @appfogstatus for updates and @appfoghelp for support.

Share this post

Try AppFog: The new PaaS Hackers love