Incident Summary: 07/09/2023

A timeline of the events of the outage and steps for remediation

Summary

Last week, Square experienced a multi-hour outage across our services. We understand that you rely on our systems to power your business and that’s a responsibility we take seriously. We apologise for letting you down and for the length of time it took for us to get our systems back up and running.

Beginning at 13:54 U.S. Eastern Time (ET) on 7 September 2023, Square products and services were unavailable. At 14:05 ET 8th September systems began to recover with merchants able to access restored payment services by 17:19 ET. For sellers on a supported configuration that utilised offline mode, Square completed processing offline payments by 13:57 PM ET on 8th September or, if the device came online at a later time, shortly after the device came online. Square Online websites were available; however, Square Online customers were unable to process payments during the outage.

As we previously shared, this outage was caused by a key part of our infrastructure, our DNS servers. Now that we’ve completed a root cause analysis, we want to share an overview of the incident and steps for remediation.

Service Impact

We’re going to start with an overview of how Square’s systems work together. Square operates in multiple data centre regions. Square services use DNS and mesh-based routing infrastructure to find service dependencies and serve requests. Without DNS, Square products, internal tools and services can’t communicate, which results in service disruption. In this incident, an unrelated change to our host-based firewalls combined with a DNS service upgrade caused unexpected load on our internal DNS servers and caused them to fail. Once node-based DNS caches expired, services couldn’t communicate with their dependencies and caused external requests to fail.

Square’s host-based firewall policy is managed by a central service that pushes firewall policies to nodes in Square data centres, which then expand the policy into firewall rules. This service uses an accelerated rollout strategy to quickly adapt to changing environment state. But, in this case, a small policy change expanded to a much larger ruleset. This large ruleset caused node instability and when combined with the traffic pattern of DNS, caused DNS to start failing requests.

Square uses a microservices environment for services that handle external requests and many internal systems to manage our services. In this case, many services used for troubleshooting and recovery were also impacted, which resulted in an extended outage.

Based on a forensic analysis of the incident, we’ve ruled out a cyberattack as the cause of this incident, and there’s no evidence of a data breach or loss.

Timeline

7 September 2023

• 11:04 U.S. Eastern Time (ET) - Host-based firewall rule change deployed to enable region communication, increasing on-node firewall rule size.
• 13:56 ET - DNS zone change.
• 14:02 ET - Engineers were notified of infrastructure issues and incident response begins starting with DNS investigation.
• 14:47 ET - issquareup.com incident created.
• 14:52 ET - Work begins to recover internal access and tooling.
• 15:56 ET - Shed networking traffic to our DNS servers. Started manual work to bring up new DNS servers.
• 18:00 ET - DNS service capacity increased, but doesn’t help. Started manual deployment of networking changes to re-enable our authorisation and access services.
• 18:29 ET - Internal access services recover. This allows engineers to start working in parallel to recover the authorisation and control plane services.
• 19:00 ET - Started manual deployment of networking changes to all data centres.
• 20:36 ET - Square deployment pipeline recovers.
• 22:06 ET - Rebuild of our DNS servers.
• 23:52 ET - New configuration based on reverted ruleset is built and configuration begins to be pushed to DNS hosts.

8 September 2023

• 12:06 ET - Some DNS hosts are healthy and more internal tooling recovers.
• 12:55 ET - All DNS servers are healthy.
• 1:30  ET - Partial recovery of internal service to service connectivity. Partial recovery of our edge routing infrastructure.
• 2:05 ET - Some Square systems begin recovery.
• 2:40 ET - Payment traffic has fully recovered.
• 3:12 ET - Edge routing infrastructure fully recovered.
• 4:18 ET - Majority of Square products and services are recovered. issquareup.com incident is updated that we’ve implemented a series of fixes.
• 5:19 ET - issquareup.com incident is resolved.
• 6:59 ET - Additional DNS capacity is added.
• 9:52 ET - Background processing of offline payments begins.
• 13:57 ET - Uploaded offline payments have been fully processed.

Service Improvements

The incident has highlighted a number of opportunities to improve our infrastructure, and we're working on making these changes, which are designed to prevent future incidents:

• Transitioning our DNS infrastructure to isolated infrastructure.
• Additional monitoring and optimisations for critical networking infrastructure.
• Optimising dependencies between our deploy and platform infrastructure where feasible.

Many sellers utilised Offline Mode in order to continue accepting payments. As a precautionary measure, we deferred processing offline payments for a number of hours. We are expanding support for and improving our communication regarding the availability of Offline Mode.

In Closing

We apologise for the disruption our outage might have created for you, your customers, and your employees. We know this situation was made more difficult by our communication frequency and the delayed support response some of you experienced. We will learn from this event and improve our systems and processes.

We appreciate your business and we are committed to doing better to regain your trust.