Salesforce cloud services even niet beschikbaar door DNS-probleem

Salesforce ondervond dinsdag 11 mei een grote storing die de cloud services trof. Oorzaak was een probleem met het domeinnaamsysteem (DNS). Meerdere Salesforce services werden beïnvloed zoals het crm platform en de cloudservices voor marketing, commercie, overheid en experience.

Klanten die probeerden in te loggen werden geleid naar een onderhoudspagina. Inmiddels heeft Salesforce het probleem opgelost.

Salesforce biedt klanten zijn excuses aan: "We bieden onze oprechte excuses aan voor de eventuele gevolgen van dit incident voor u en uw bedrijf. Bij Salesforce is vertrouwen onze nummer 1 waarde en heeft beveiliging onze topprioriteit. We waarderen transparantie en willen van deze gelegenheid gebruik maken om de feiten te schetsen met betrekking tot een recente serviceonderbreking die uw vermogen om meerdere Salesforce-services te gebruiken, zoals we die momenteel begrijpen, mogelijk heeft verstoord. Ons onderzoek is aan de gang en we zullen klanten in de toekomst voorzien van bijgewerkte informatie."

Salesforce meldt op zijn Engelstalige website het volgende over de storing:

The preliminary Root Cause Analysis (RCA) is now available to affected customers as a Knowledge Base article. Further root cause analysis and investigation are underway by the Salesforce Technology team. We will provide an update as soon as available. Please refer: https://help.salesforce.com/articleView?id=000358392&type=1&mode=1

User Impact:

On May 11, 2021, at approximately 21:08 UTC, the Salesforce Technology team became aware of a service disruption across Salesforce production instances. The disruption impacted the ability for users to log into their Salesforce environments within the core Salesforce services, Marketing cloud, Commerce Cloud, Government cloud, Experience cloud, Heroku, Pardot, and Vlocity. In addition, the status.salesforce.com Trust site was also unavailable, and customers were unable to log support cases. Some customers may have also experienced issues with Multi-Factor Authentication (MFA) during the incident. Initial Impact & Remediation The Salesforce change to the Domain Name System (DNS) impacted multiple data centers, affecting the Salesforce service for customers, the status.salesforce.com site, and Salesforce authenticator. Over the following hours, the team worked to restore service on the data centers through manual action taken to the Domain Name System (DNS). The Subject Matter Experts (SMEs) were involved in the restoration work with access control and data center engineers to obtain access to the Domain Name System (DNS) server systems. They restarted the impacted Domain Name System (DNS) services. Marketing cloud was declared "all-clear" at 00:53 UTC on May 12, 2021. Commerce Cloud and Heroku were also "all-clear" at this time, with significant portions of the Government cloud, Experience Cloud, Pardot, and Vlocity infrastructure also restored. As the SMEs restored each impacted Domain Name System (DNS) service, access to impacted clouds was recovered for individual instances and PODs. Following the instance and POD recovery, the Technology team received alerts of connectivity issues to the database tier across several locations. The Salesforce Technology team performed a rolling restart for the application tier across multiple instances and PODs to restore connectivity. The Salesforce Technology team declared an all-clear at 02:20 UTC on May 12, 2021. As a result of this issue affecting multiple clouds and services, including the Trust site, this incident was posted as a General Message. Instance/POD level impact times are posted on your respective instance-specific Trust status pages. We are currently ascertaining additional impact to instances/POD and will continue to post individual Trust pages after confirming the impact start and end times.

Update 20-5-2021: Interne fout veroorzaakte storing. De wereldwijde DNS storing werd veroorzaakt door een technicus die een snelle oplossing probeerde uit te rollen.