Fortinet, Shopify, and others report issues after Lets Encrypt root CA certificate expires


A number of websites and services reported issues on Thursday due to the expiration of a root certificate provided by Let’s Encrypt, one of the largest providers of HTTPS certificates.

Around 10 a.m. ET, IdentTrust DST Root CA X3 expired according to Scott Helme, founder of Security Headers. He followed the issue and explained that millions of websites depend on Let’s Encrypt services and without them some older devices will no longer be able to verify certain certificates.

Let’s Encrypt operates as a free, non-profit organization that makes sure that the connections between your device and the internet are secure and encrypted.

Despite the prior warning that the expiration date would be September 30, the deadline, dozens of users have reported issues with various services and websites.

Helme told ZDNet that he has confirmed issues with Palo Alto, Bluecoat, Cisco Umbrella, Capture point, Guardian Firewall,, PFsense, Google Cloud Monitoring, Azure Application Gateway, OVH, Auth0, Shopify, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, Ledger, Netlify and Cloudflare Pages, but noted there may be more.

“There are several ways to fix this depending on the exact issue, but it boils down to: the service / website needs to update the certificate chain it serves to clients or, the client talking to the website / service needs updating, ”explained Helme.

“For the companies involved, it’s not like everything is down, but they certainly have service issues and incidents are open with the staff working to resolve them. In many ways, I’ve been talking about it since. over a year since the last time, but it’s a difficult problem to identify. It’s like looking for something that could start a fire: it’s really obvious when you see the smoke! “

Some sites have posted notices on their website regarding potential issues and many have fixed the issues. Shopify job a note on its incident page that around 3:30 p.m., merchants and partner businesses who were having trouble getting online had restored their services. Authentication of merchants for interactions with support has also been restored, the company said.

Fortinet told ZDNet that they are aware and have investigated the issue with the expired root CA certificate provided by Lets Encrypt.

“We are communicating with customers directly and have provided a temporary workaround. Additionally, we are working on a longer term solution to resolve this on-board issue directly in our product,” the company said in a statement.

Digital certificate expert Tim Callan said all modern digital systems depend on certificates for their continued operation, including those that secure our cyber and physical environments.

“If the software depends on an expired root to validate a certificate’s chain of trust, the certificate’s trust will fail and in most cases the software will stop functioning properly. The consequences are as wide and varied as our individual systems are, and many times cascading failures or “downstream” failures will cause problems with entirely different systems than the one with the original certificate trust issue. ” , said Callan.

“Computer systems that enforce or monitor security policies may stop functioning. Alerting and reporting systems can fail. Or, if the processes that humans depend on to do our jobs stop functioning, these people will often find fundamentally insecure “workarounds”. “

Callan added that outages can occur when developers integrated into business lines of operations or other skunkworks projects “obtain certificates” without the knowledge of the central IT department, then move on to new tasks or fail to complete. monitor the life cycle of these certificates.

He noted that most systems will be able to withstand a root expiration due to modern root chaining capabilities which allow another root to establish trust.

However, existing systems or those with unresolved (or unknown) certificate management bugs are at risk for failures like these. In the case of a commonly used root of a popular CA, the risk of these failures increases dramatically, ”Callan explained.

TechCrunch reported that devices likely to have issues include older macOS 2016 and Windows XP (with Service Pack 3) as well as older versions of Playstations and any tools that rely on OpenSSL 1.0.2 or earlier.

Other experts have said that PlayStation 4 or earlier devices that have not upgraded firmware will not be able to access the internet. Devices like Android 7.1.1 or earlier will also be affected.

According to Callan, most modern software allows the use of sophisticated chains of trust that allow root transitions without requiring replacement of production certificates. But those that are old or poorly designed or contain chain of trust management bugs may not properly handle this transition, leading to various potential failures.

As many affected companies have since done, Callan suggested that companies take an inventory of systems using certificates and actually used certificates before ensuring that the software has the latest root certificates in its root store.

“By identifying potential points of failure, IT departments can investigate these systems in advance to identify problem areas and implement fixes. If you can set up a version of the system in a sandbox environment, then it’s easy to test the expected behavior once the root expiration occurs, ”Callan said.

“Just set the client system clock to a date after the expiration date to ensure certificate chaining will work properly. You can also uninstall manually or not trust the root that should expire (in the ‘sandbox environment, of course) to make sure that systems only use the most recent roots. “

He added that the popularity of DevOps-enabled architectures like containerization, virtualization and the cloud has dramatically increased the number of certificates the business needs, while dramatically reducing their average lifespan.

“This means a lot more expiration events, a lot more administration time required, and a dramatically increased risk of renewal failure,” he said.

Sean Nikkel, senior cyber threat analyst at Digital Shadows, told ZDNet that Let’s Encrypt warned everyone in May of the root CA expiration today and offered alternatives and workarounds to ensure that the devices would not be affected during the change.

They also kept an open thread on this issue with fairly quick responses, Nikkel added.

“A not very good practice that has already been proposed as a workaround for the problem is to allow untrusted or invalid certificates. Users should be careful before taking any step that potentially opens the door to attackers using compromised certificates.” , said Nikkel.

“Some users have recommended settings to allow expired certificates from trusted issuers; however, these can also have malicious uses. Either way, admins should consider the best solution for them, but also understand the risks of workarounds. Alternatively, administrators can examine alternative trust paths using the intermediate certificate that Let’s Encrypt has configured or by following the configurations suggested in their May newsletter. “

Source link


About Author

Comments are closed.