- Cloudflare admits it caused its own outage – it wasn’t a cyberattack
- Fluctuating error reports made the problem challenging to identify at first
- The “unacceptable” outage did lead to some learning opportunities
Cloudflare has shared more details about its November 18 outage – its worst outage since 2019 – confirming that it wasn’t the result of an attack or any other type of malicious activity.
In a blog post, company co-founder and CEO Matthew Prince explained a database permission change triggered the system to generate a ‘feature file’ that doubled in size, before it propagated to all the machines on its network, causing the software to fail.
Because Cloudflare was able to identify what had gone wrong, normal operation resumed a little over three hours after the outage, with full recovery a few hours later.
Cloudflare confirms outage wasn’t an attack
“Core traffic was largely flowing as normal by 14:30,” Prince wrote, as confirmed by a chat showing a big drop-off in 5xx error HTTP status codes right around that time.
However, Cloudflare did need to dig a little deeper to discovery what exactly was going on due to a pretty high fluctuation range of error reports. This was because the problematic file was being generated every five minutes.
“As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period,” Prince added, noting that “large amounts of CPU” were being used across debugging and observability.
Cloudflare’s status page also went down during the attack – a page that’s totally independent of Cloudflare’s infrastructure. Apparently, this was little more than a coincidence.
Nevertheless, the outage did at least serve as a learning opportunity for Cloudflare, which now promises to enable more global kill switches for features.
“An outage like today is unacceptable,” Prince concluded, before putting a slightly positive spin on it: “When we’ve had outages in the past it’s always led to us building new, more resilient systems.”
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form, and get regular updates from us on WhatsApp too.
