A huge Amazon Web Services outage has taken out multiple web services including Alexa, Ring, Reddit, Snapchat and many more.
The issues appeared to start at around 7.40am BST when a large spike on Downdetector showed reported problems with Amazon Web Services – which in turn has taken down many services that rely on its cloud computing power.
The exact cause of the issues isn’t clear, but the Amazon Web Services dashboard shows an “operational issue” in North Virginia. The AWS dashboard says “engineers were immediately engaged and are actively working on both mitigating the issue” and recently posted that “we are seeing significant signs of recovery”.
The knock-on effects appear to be widespread, including many banks – you can follow all of the latest updates here in this liveblog…
The latest news
- A huge Amazon Web Services outage has taken down much of the internet
- Alexa, Snapchat, Ring, Zoom and multiple banks are experiencing problems
- The issues started at around 2.40am ET / 7.40am BST
- AWS says it’s now seeing “significant signs of recovery”
This is an online earthquake
The Amazon Web Services (AWS) outage started at around 7.40am BST, according to Downdetector – and the ripple effect has taken out much of the internet.
Right now, Alexa, Snapchat, Ring, Roblox, Fortnite, Zero, Signal, Canva and countless others are showing huge spikes in reports on Downdetector.
The problems show just how many services rely on Amazon’s cloud computing services. The AWS Health Dashboard is showing an “operational issue” in North Virginia that’s producing “increased error rates and latencies”. This is going to have serious repercussions today for millions…
This explains those Alexa issues
While it’s unlikely to be the most serious issue today, the first time many saw the impact of the Amazon Web Services outage was with its Alexa voice assistant.
This Reddit thread shows many were recently reporting problems with the assistant, with some saying they “had to turn the lights off manually” and couldn’t set their alarms.
Well, AWS is the root cause – and Alexa is far from the only service experiencing problems. Alongside Ring, Snapchat and Zoom, we’re also seeing banking apps like Lloyds and Halifax in the UK and Robinhood in the US showing large spikes on Downdetector…
Your bank may also be hit
While the problems with our Echo devices this morning are annoying, a more serious repercussion from the AWS outage for many will be that multiple banks have been hit.
In the UK, Downdetector is showing large spikes on Lloyds, Halifasx Bank of Scotland and even HM Revenue & Customs. It just shows how deep and wide the Amazon Web Services infrastructure is, and how badly the internet is affected when it runs into rare problems.
The latest from the AWS dashboard at 1.26am PDT / 9.26am BST is that “significant error rates” are happening in the ‘US-EAST-1 Region’ and that “engineers were immediately engaged and are actively working on both mitigating the issue, and fully understanding the root cause”. Let’s hope they do that sooner rather than later…
Who is affected?
The easier question for today’s AWS outage might be ‘who hasn’t been hit’?
Right now on Downdetector, we’re seeing large spikes on dozens of services. On the more serious side, this includes Zoom, Slack, Blink Security, Venmo and, in the UK, Lloyds Bank, Halifax, HM Revenue & Customs, Bank of Scotland and more.
But the AWS earthquake is also rippling through Snapchat, Roblox, Fortnite, Duolingo, PlayStations Network and yes, even Wordle.
Even Downdetector is showing a rare red banner showing the severity of the Amazon Web Services outage in the US-East-1 region. We should get another update on the AWS Dashboard soon…
Ring owners are very confused right now
The AWS outage has rippled far and wide across the internet, but some of the biggest hit have been Amazon’s own services like Alexa and Ring.
For many, Ring cameras have become intertwined with their daily schedule – and that’s now been massively disrupted today, as TechRadar’s Senior Writer AI told me.
“Every morning I go for a coffee before work, taking the dog a walk around the block before sitting at my desk for the day. Normally, my smartphone gets bombarded with Ring doorbell notifications from the busy neighbors moving in and out, as I pay close attention to the app waiting for the postman.
“Today, however, until I realized that AWS services were down I just sat confused as to why the normal morning ruckus of the school run wasn’t happening. I work with headphones on and rely on Ring to know when to open the door for the postie before my dog gets alerted by their presence.
“With Ring struggling this morning, I’m going to have to stay ultra aware as I wait for the knock on my door, fully accepting that my tiny French Bulldog is going to go crazy at the thought of an intruder.
“Millions of people rely on Ring for security and safety concerns, while others just need the video doorbell for simplifying life and knowing when mail has been delivered. My Ring doorbell is one of the best tech purchases I’ve ever made, and that’s only emphasized when it stops working and I can’t go about my day the way I usually would”, he said.
We may have a cause
An update on the AWS Dashboard at 2.01am PDT / 5.01am ET / 10.01am BST may have given us some good news.
It says “we have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region”.
Leaving the technical details aside, the most important part is that AWS says it’s “working on multiple parallel paths to accelerate recovery”.
AWS has promised another update by 2:45am PDT / 5.45am ET / 10.45am BST. Until then, expect to see a lot more error messages on your favorite services…
Tidal is out – Monday morning playlists ruined
Ugh, when your favorite music streaming service is out for the count any way you want it (that’s the way you need it) on a Monday morning.
Becky Scarrott, TechRadar’s audio editor, reports: “I’m TechRadar’s audio editor and the time of writing, Downdetector‘s list of Tidal outage reports has tipped 337 entries, with 63% listing app issues (which makes sense, because that’s how most people access music on the fly – ie. their commute), 15% noting the website isn’t letting them in (it isn’t for me) and 22% mentioning the server connection.
“I initially got the wheel of death on the app, then a crash. And when I tried to click back into the Tidal app, a ‘504 Gateway Time-out’ message appeared, after I was prompted to log in.
This is of course related to AWS’s “operational issues”, and let me tell you, my operational issues are also affected when I can’t get my Monday morning playlist going…”
‘Significant signs of recovery’
Some good news, potentially – Amazon Web Services has just provided another promising update.
Its latest post, from 2.27am PT / 5.27am ET / 10.27am BST, says: “We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information”.
My takeaway from that (and the evidence on Downdetector) is that we’re past the worst of the earthquake, but aftershocks and problems could still continue for a while yet…
Strava hits the AWS wall
Fitness app Strava is also experiencing server issues. Strava users might be seeing the following message: “The Strava server has encountered an error. Please contact support if the problem persists.” Unfortunately, as Strava runs on AWS servers, contacting support won’t do very much at all.
Matt Evans, TechRadar’s Senior Fitness & Wearables editor, says: “I can still look through my timeline, view my profile, and interact with friends’ posts, but the app is a little sluggish: 500 errors frequently occur when refreshing pages, and my latest activity, logged on my Garmin watch, hasn’t uploaded to Strava.
“Coincidentally, Garmin Connect also runs on AWS servers, but loading and navigating the app, I haven’t seen any problems so far. It’s hardly critical infrastructure like banks, but concerned athletes should rest assured that this isn’t an issue on Strava’s end.”
The worst may be behind us…
That promising update from AWS (below) is reflected on the Downdetector graph for Amazon Web Services, which is the epicenter of today’s internet earthquake.
At is peak in the US at 3.52am ET / 8.52am BST, the AWS graph had almost 6,000 reports in the US and 4,000 in the UK. But that’s now down to around 2,000 and 1,500 reports respectively, suggesting that those “significant signs of recovery” are being felt in the real world.
Similarly, Downdetector graphs for other services are also showing a dip, which means the end of this major outage could be in sight…
Disaster, now Reddit’s down
The place we all go for refuge when there’s a big internet blackout is sadly also been hit by the AWS storm – now Reddit is showing large spikes on Downdetector.
I’m getting a “too many requests” error on the homepage and so are thousands of others, with over 6,000 reports in the US and over 3,000 in the UK.
In fact, the Reddit graph is bucking the trend of other services, which are showing a downturn in problems, whereas Reddit’s continues to spike upwards. Signs of a different problem? It’s more likely to be Reddit’s core audience waking up to today’s bad news…
Good news, everyone – Wordle is back!
If you’ve been worrying about getting your daily Wordle fix, then fret no more – because the mega-popular online word game is back.
The site, which lives on the New York Times‘ gaming site, was hit by the AWS outage earlier today, with users unable to log in. So, while they could complete the day’s game, it wouldn’t count towards their streak – a feature which is of vital importance to many Wordlers.
Everything appears to be working normally again now, though, so you can go ahead and attempt today’s puzzle. Just be sure to you check out our Wordle today page for hints and discussion of the game first.
Any signs of a cyber attack?
Internet outages on the scale of today’s AWS one usually spark concerns about cyber attacks. But once again, a more boring, infrastructure-related issue appears to be to blame.
The Amazon Web Services dashboard says that “the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1”, which in simpler terms means a problem between the ‘domain name servers’ (which translate IP addresses) and the interface that sits between clients and servers (APIs).
As Rafe Pilling (Director of Threat Intelligence, Sophos) commented: “When anything like this happens the concern that it’s a cyber incident is understandable. AWS has a far reaching and intricate footprint, so any issue can cause a major upset. In this case it looks like it is an IT issue on the database side and they will be working to remedy it as an absolute priority.”
That appears to be the case, with the AWS dashboard stating “we continue to observe recovery across most of the affected AWS Services”. Many sites and services, like Reddit and Ring, do continue to experience major issues.
