Defending against cyber threats is like guarding a fortress in a digital realm – every byte counts in the battle for security.
Unknown
Update: When I wrote this post, Microsoft had not completed their investigation of the cyber attack. Specifically, they hadn’t figured out how the key was stolen. A couple days ago they announced they had discovered how the key was stolen and boy its a doozy. The system where the key was stored, was in fact a secure system. Employee’s were background checked, required special permissions to access the system, and overall it was tightly controlled. At some point the system crashed and created a crash dump. A engineer investigating the issue, did not have the tools they needed on system and the dump was moved to an insecure system. That engineers credentials were eventually stolen and were used to access where the crash dump lived. The attackers discovered the keys in the crash dump and the rest is history. Just goes to show you what you are up against, when you join cyber security.
On July 11th, Microsoft announced that customers in their Azure and Office365 environments had been breached. The affected customers were mostly US Federal Agencies, and per Microsoft, the attacker was most likely a nation-state actor from China. As more and more information came out, articles were written by countless organizations stating “Countless Problems” and how “grossly irresponsible” Microsoft was in terms of Security. I’ve written about my qualms with Microsoft in the past, but as someone who worked there, the amount of blowback isn’t warranted. The attack certainly deserves an external review and an internal one, but certainly not to the level it’s currently getting. The complex realities are that vulnerabilities exist in everything because they were built by humans and are used by humans.
What Happened?
For those who are unaware, here is what happened:
In July 2023, Microsoft revealed details about a cyber espionage campaign conducted by a threat actor called Storm-0558. This group, believed to be China-based, targeted various organizations with forged authentication tokens to access user email accounts. The attack was focused on government agencies and consumer accounts within the public cloud, with the objective of unauthorized email data access. Microsoft has thwarted this campaign and taken steps to enhance security measures and inform affected customers.
Key Points:
Threat Actor and Objectives: Storm-0558 is a China-based cyber threat actor engaged in espionage. The group’s activities involve targeting diplomatic, economic, and legislative entities in the US, Europe, as well as individuals connected to geopolitical interests like Taiwan and Uyghurs.
Attack Techniques: The group employs tactics like credential harvesting, phishing campaigns, and OAuth token attacks. The use of OAuth applications, token theft, and token replay in Microsoft accounts is noted. In this attack they used a stolen inactive MSA key. At this time Microsoft doesn’t know how they obtain the key. See Token Forgery Section.
Initial Access and Exploits: Once inside, they deploy malware like China Chopper and a shared tool named Cigril.
Compromised System Access: After infiltration, Storm-0558 acquires credentials from sources such as LSASS process memory and Security Account Manager (SAM) registry hive. With valid account credentials, the actors access compromised users’ cloud email accounts to extract information.
Discovery and Analysis: Microsoft detected anomalous data access on June 16, 2023, and attributed it to Storm-0558 based on prior tactics. The initial assumption of token theft was revised when it was found that the actors were forging Azure Active Directory (AD) tokens.
Token Forgery: Storm-0558 exploited an acquired Microsoft account (MSA) consumer signing key to forge authentication tokens. This allowed them to access Exchange Online data. Azure AD keys were not affected. Microsoft has taken corrective measures to prevent this kind of attack.
Access Techniques: The threat actor used forged tokens to authenticate via legitimate client flows. They exploited a flaw in the GetAccessTokenForResource API to obtain new tokens, enabling access to mail messages from the Outlook Web Access (OWA) API.
Actor Tooling: Storm-0558 utilizes PowerShell and Python scripts to make REST API calls against the OWA Exchange Store service. These scripts can download emails, attachments, and conversations. The group employs proxies and disguises its activity with various User-Agents.
I would highly recommend anyone who is interested in security to go read the write-up about the event here. In my defense of Microsoft, I have to credit them for releasing this article on the events of how things happened. Many companies will follow Federal disclosure guidelines and then proceed with a ton of hand-wavey bullshit that everything is fine. Crafting this article aids companies and security engineers in detecting and safeguarding against additional cyberattacks, ultimately benefiting the entire cybersecurity community.
In Defense
The most critical article I have seen is this one from Arstechnica. In it, the author talked about a Senator from Oregon putting Microsoft on blast for “negligent cybersecurity practices” and how the CEO of Cyber Security firm Tenable took to Linkedin explaining how they had warned Microsoft this would happen. Tenable had discovered the issue with one of their banking clients, and it scared their security team so much they rushed to get Microsoft to fix the issue. The CEO goes on to explain that Microsoft took their time fixing the issue and was breached in the process. The senator from Oregon, on the other hand, believes that Microsoft should have never deployed systems in the states they were in and failed basic security practices.
That being said, I disagree with what is being said. There are different levels of threats and vulnerabilities, and based on those levels, the most critical should be acted on first. It’s part of the reason we have CVE and CVSS scores. Microsoft is huge, and we have no idea what other vulnerabilities they were dealing with at the time of the disclosure. On top of that, this attack doesn’t look like it could have been carried out without having some level of access to the system already. The attack required a key to chain together with the actual vulnerability, allowing an inactive key to generate tokens that allowed access to all sorts of data. If I am Microsoft and I have a CVE of 9 that I am currently fixing, and a 3rd party company comes in with a CVE of, say, 7.5 and says, with the right key, this API can be abused, what would you do? It’s a priority, but you will keep fixing the higher critical issues. Microsoft’s response might not have been as swift as some expected, resulting in vulnerabilities being exploited. However, it’s unlikely that they remained inactive. Their subsequent actions upon discovering the issue demonstrate their proactive approach.
While I can’t get into too much detail, I will say of all the places I have worked in my career, Microsoft had the best security controls. Segregation of permissions and the network were everywhere. Getting access to business-critical systems or customer data was never permanent. Regarding their Federal Office365 environment, security was taken to a different level. I had to get my fingers printed for a background check, and we worked in a secure room. The amount of controls in place to do anything in that environment was unlike anywhere else I have worked. From an external perspective, Microsoft has enormous legacy debt while also trying to provide the same level of service as AWS and GCP. It looks like in their move to “move fast and break stuff,” they deployed a keystore outside policy and in an unsafe manner. I would be curious to find out how the attacking group got hold of the key, but we may never know.
Unlike certain companies that view security as an impediment, prioritizing progress over protection, Microsoft takes a fundamentally different approach. They don’t engage in mere checkbox security theater or heed advice only post-breach. Microsoft’s proactive stance stands in stark contrast, valuing security as an integral part of their operations rather than an afterthought. At least, that was the case when I worked there.
They Aren’t Perfect
Despite my defense of Microsoft, I do need to state that they did make some mistakes:
- Deploying a keystore with valuable keys where one could leave the company is a big mistake. Maybe it was an inside actor; perhaps it was scraped off an employee’s machine. Either way, a key that could generate that much access should be locked down tight with processes involved in retrieving it.
- Certificates that don’t expire or are valid for years. I’ve worked at multiple companies where this is true. I can’t say I am surprised, but since it’s Microsoft policy never to have certificates that last that long, they should practice what they preach.
- The fact that a customer found out something was wrong before Microsoft is not a good look. I can’t imagine how much data they ingest, but as a security engineer, it always leaves you on the wrong foot when a customer tells you something is wrong before you know.
These challenges extend beyond Microsoft’s domain; however, given their significant presence as a leading cloud provider, they rightly bear a greater responsibility. Their pivotal role in the cloud landscape necessitates a higher standard of security and accountability.
After Thoughts
I readily acknowledge my skepticism toward cloud technology. With the three major US cloud providers collectively hosting countless businesses, vulnerabilities loom large. A single update, misconfigured keystore, or even a single rogue employee can potentially compromise these platforms. Like hunters drawn to a massive whale in the vast ocean, hackers are irresistibly drawn to cloud providers in the United States, lured by the sheer abundance of businesses hosted on their platforms, making them high-value targets for cyber intrusions. Microsoft messed up, they can do better, but this attack isn’t some gross irresponsibility.
Anyways, I hope enjoyed the latest article. Once again good luck to all the engineers out there, and until next time, stay safe!
