On September 28, 2023, a simple mistake caused a chain reaction of security breaches that impacted some of tech’s biggest players, such as Cloudflare and 1Password.
What happened?
An employee at Okta—an identity and access management provider—simply logged into their personal Google account on a company-managed laptop. Seems innocent enough…right?
Without realizing that their Google account or personal device was already compromised, this log-in allowed an attacker access to an Okta service account, which granted permissions to view and update customer support cases. Many of these cases contained HAR (HTTP Archive Format) files, which Okta’s staff used to troubleshoot issues.
Now, things have gotten spicy.
Some of those HAR files contained session tokens, which the attacker used to attempt to move into the Okta customers’ systems. 1Password was the first to notify Okta of suspicious activity, followed by BeyondTrust just days later, and while none of these downstream platforms were affected, this breach was a harrowing reminder of how quickly data loss happens—especially in the SaaS-first environment we all operate in.
Cloudflare made their frustrations clear in a blog post titled, How Cloudflare mitigated yet another Okta compromise, in which they “urged” Okta to take immediate action to prevent further vulnerabilities and recommended all Okta customers upgrade their protections against intrusion and data loss.
Data loss is perhaps the biggest risk for any business. It happens unexpectedly and in unpredictable patterns from sources that are both completely unknown (cyber attackers) and very well-known (the co-worker three cubicles over). But even if publicly traded tech giants like Okta and Cloudflare are vulnerable, how can the rest of us prepare?
What is data loss?
Before we get too deep into the nuances of this conversation, we should define data loss.
You might assume the phrase refers only to unintended data deletion. That’s certainly part of the equation, but equally relevant—and devastating—for many businesses is the breach or exfiltration of confidential data in the form of IP, customer personal identifiable information (PII), and more.
The only difference between deletion and “spillage” is the path to remediation. The former requires robust data protection plans, including automated backups and reliable restores. The latter? You’re probably best served by a crafty PR team and a behemothic law firm on retainer.
The many-headed beast of data loss caused by human error
No matter the type of data loss we’re talking about or its downstream impact on lost time and costly remediation, the damage is invariably bad: In IBM’s Cost of a Data Breach Report 2023, researchers found that the average total cost of a data breach reached an all-time high of $4.45 million in 2023, a 15% increase over the last three years.
That’s all the more reason to debunk the commonly-held myth that external risks, like ransomware, are bigger threats to your data security; a 2020 study between Stanford University and cloud email security company Tessian revealed that 9 in 10 data loss incidents were caused by human error. What common behaviors and pitfalls are most often to blame?
Accidents
An employee might accidentally delete a key project file from Trello or modify configuration settings in Bitbucket, leaving a repository inaccessible to everyone but a departed colleague’s half-deleted account. Or, in the case of GitLab, that employee might mistakenly delete 300GB of user data in a matter of seconds while trying to mitigate performance degradation.
When we give specific individuals wide-ranging permissions to our systems—particularly SaaS ones—we create massive exposure to honest mistakes. At the same time, organizations can’t stay productive if every simple action requires a half-dozen handshakes and sign-offs. It’s a fine balance that often rolls on for years without consequence until just the right sequence of unfortunate events causes a mistake one can’t simply Ctrl+Z on.
Phishing
In the 2022 follow-up to the Stanford+Tessian study mentioned above, researchers found that more than half of employees had fallen for a phishing attack that impersonated a senior executive. In an extreme example, London-based engineering firm Arup recently admitted it was the victim of a $25 million fraud involving AI-generated voices, deep-faked images, and publicly available information.
You might expect such large transactions to have significant safeguards and verifications. If it can happen to them, imagine how a simple phishing email, crafted with information about your executives available publicly on LinkedIn, could trick unsuspecting employees into giving out credentials or confidential details about other SaaS tools you use.
Password reuse
In our 2024 State of SaaS Data and Recovery report, 84% of respondents reported that at least one third of their business-critical data lives inside SaaS applications. Now imagine how frustrating many would find juggling all those usernames and passwords. The more friction, the more likely they will fall back to one familiar password they keep on a Post-It beneath their keyboard.
Password reuse has two dangerous consequences. First, an employee’s common password results in a massive data breach, exposing your SaaS apps to automated attacks. In the second scenario, an attacker leverages more targeted attacks to expose a single password, which they immediately use to make a lateral movement.
User access misconfigurations
Even in a well-governed organization, having 130+ SaaS applications means thousands of settings for your employees’ read/write/update/delete privileges. Identity and access management platforms like Okta, OAuth, and OneLogin provide new control measures, but having complete vision and ownership over this sprawling landscape is a near-impossible task.
In fact, a recent report from Silverfort discovered that a single misconfiguration in Active Directory could create up to 109 new shadow admins. Shadow admins are user accounts that have inadvertently been given full or partial administrative privileges. When an attacker compromises a single shadow admin, they can quickly reset the credentials for a true admin, giving them unfettered access to your SaaS data.
Integration and migration mishaps
A poorly configured integration between GitHub and Trello could easily expose confidential data to a public repository or result in accidental deletion. Tricky migrations, like from an on-premises Jira to their new Cloud offering, often result in data loss that administrators only identify after the work is considered done, leaving folks without a clear remediation path except starting from scratch.
Data loss during these complex technical motions can be caused by unknown bugs or honest mistakes from those responsible, but the root cause is still people. When they lack contingency plans or don’t adequately test integrations and migrations in staging environments, they have only themselves to blame. It might feel unfair to place such a burden on these folks to get the job done right every time, but the buck must stop somewhere.
BYOD policies
Working from home is more popular than ever, and so is using a personal laptop or phone to access company data. In some cases, devices simply get lost (remember the iconic leak of the iPhone 4 due to an Apple employee who enjoyed a Silicon Valley beer garden a little too much?). Still, the more immediate risks are the mixing of saved credentials in a single environment and a lack of security on personal devices.
The Okta story we began with was caused by the opposite of BYOD—an employee used a company-managed machine for personal reasons—but still clearly expresses the challenge involved in mix-and-matching devices and stored credentials in environments that IT administrators or DevOps engineers can’t control.
Shadow IT
Every time an employee deploys a software system without sign-off from IT, they create shadow IT, which puts your business-critical data at new and unique risk. Their intentions are mostly honest and driven by productivity, where they find a viable solution to a meaningful problem and don’t think about the implications of circumventing IT safeguards and policies.
When shadow IT goes wrong—and it often does—it gives expansive permissions to too many people, unknowingly deposits intellectual property on insecure systems, and leaves your organization non-compliant with GDPR, HIPAA, and other strict regulations.
Why do we keep downplaying this pervasive risk?
From our perspective, in working with thousands of companies proactively protecting their SaaS data through automated backups, folks tend to ignore the danger of human error-caused data loss for four reasons.
- Overconfidence.
Most IT administration and DevOps engineering teams have their on-premises data protection scheme figured out by now. They have robust security measures, access management controls, 3-2-1 backup strategies involving adequate offsite storage, and contingency plans for hardware failure.
This familiarity with traditional data protection methods, paired with the promise that cloud environments abstract away all complexity and security fears, leads to a dangerous complacency with protecting new SaaS environments.
- Misdirection around the Shared Responsibility Model in SaaS environments.
A recent report from Oracle and ESG revealed that less than 1 in 10 chief information security officers (CISOs) understood that protecting organization- and user-level data in SaaS apps is entirely their responsibility. Half of their respondents also reported losing data due to that confusion, revealing how large the educational gap still is.
The root cause? SaaS providers do everything in their power not to admit how little protection they offer.
- The unique complexity of data in SaaS environments.
SaaS gives us many valuable shortcuts in day-to-day work, helping us be more collaborative and proactive, but it comes with new costs. Aside from knowing exactly which SaaS apps are being used by employees and properly managing them, it’s extremely difficult to identify when—and with what consequences—sensitive data is stored in external environments you can’t trust. Throw in shadow IT and admins, and you have a very difficult, always-changing landscape to navigate, much less have meaningful ownership.
- Distraction from the fear of new threats.
Given how much more coverage is given to ransomware or the danger of AI assistants with unfettered write access, it’s understandable that most IT admins and DevOps engineers don’t fully recognize the danger of honest mistakes and relatively simple software bugs to the availability of their ever-growing landscape of SaaS data.
What to do (and not do) when you can’t fully control people
Humans, for all their creative problem-solving abilities and unpredictable reactions, can never be fully controlled. Trying to patch over every possible edge case and existential risk would take ages and result in a tightly controlled working environment that employees would, once again, try to circumvent.
Resistance to all things SaaS isn’t an answer either. SaaS apps have done extraordinary good for our collective communication and productivity, and there is simply no taking back the value we’ve all found in untethering data and user experience from the devices in our hands or beneath our keyboards.
From a data continuity and security perspective, your best bet is implementing more proactive solutions over time, focusing on how you respond to and recover from data loss rather than trying to prevent it from ever happening. As time and human ingenuity unveiled new weak points in your SaaS armor, consider these actions (or, in some cases, don’t!) and prioritize based on that core idea of response and recovery.
Do: Automated backups and on-demand restores
You can only recover deleted data that you’ve previously backed up. That’s a relatively simple equation for on-premises deployments, but in our SaaS-based world, it often becomes a convoluted web of custom scripts, .json or .csv files, and processes that work once in testing, but not six months later when data loss strikes.
We encourage you to explore the ecosystem, but just keep in mind that Rewind is designed to protect mission-critical SaaS data. With on-demand restorations that will start reconstituting your data, even in the most catastrophic scenarios, in minutes, you can trust that we’ve done everything possible to prevent every possible opportunity for data loss. With 25,000 organizations storing their SaaS data in our fully encrypted and compliant cloud architecture, you can trust that we’ve done everything possible to prevent every possible opportunity for data loss.
When was the last time your SaaS ecosystem was pentested?
Do: Implement enterprise-grade identity and authentication methods
Yes, Okta has already proven that SaaS-based identity and access management platforms create yet another possible attack vector for your business, but it’s almost always a risk worth taking.
These platforms dramatically simplify your path to single sign-on, multi-factor authentication, hardware-based keys, and more, dramatically reducing the risk of misconfigurations and password reuse. With centralized privileges, you’ll also be far more likely to identify unwanted privilege escalation, shadow admins, or “dangling” permissions after an employee has departed.
Do: Continue your education on how the Shared Responsibility Model affects you
It’s easy to overlook how quickly and thoroughly you let SaaS providers off the liability hook when you agree to their Terms and Conditions.
GitHub, for example, doesn’t sugarcoat its (lack of) liability in the slightest: “You understand and agree that we will not be liable to you or any third party for any loss of profits, use, goodwill, or data, or for any incidental, indirect, special, consequential or exemplary damages.”
Yes, they should be able to restore their service and your data if a meteorite hits their primary data center, but what about all those minute human errors? Your continuing education starts with proactively exploring the liability implications of each mission-critical SaaS provider.
If you’re still trying to understand the depth of your exposure, we sure have written about this model quite a bit over the years.
Don’t: Be a ‘a frog in a well’ of your SaaS data
In a famous Chinese fable, a frog who has spent its entire life at the bottom of the well finds so much joy in their small and simple existence. Its world is limited only to the water beneath and the circle of sky above. When told about the vastness and persistence of the so-called “Eastern Sea,” the frog is flabbergasted and probably a bit heartbroken to realize how narrow its worldview had always been.
SaaS data works similarly. Your most visible SaaS data—contents of Trello tickets, items in your Shopify store, or customer emails in your Mailchimp database—is like that small porthole in the sky. Beyond that, it’s intricately knit together by a far wider world of metadata.
Take a look at any GitHub repository:
- The well? That’s your files, folders, and many lines of code. It gets your services to production, but isn’t the whole story.
- The rest of the world? Your issues, pull requests, active discussions, data-driven insights, project management information, and more—all how your employees communicate, collaborate, and discover new perspectives.
When you look beyond the immediate content your employees create in SaaS apps, you see the network of internal knowledge that’s just as essential to protect.
Don’t: Disconnect your employees’ SaaS access
If they depend on SaaS apps to get their job done, they will simply find another way to do so behind your back. The more shadow IT they create, the fewer opportunities you have to protect the data, metadata, and knowledge they create comprehensively.
Don’t: Assume a SaaS provider’s export files have your back
Most SaaS providers provide some method of creating export files for data portability. You might assume you can always re-import these files back into a hosted SaaS instance, but most often, it is never that simple.
Many imports involve convoluted manual processes like editing .xml files, breaking up large databases into smaller parts for faster uploads, and carefully organizing files and folders into a specific hierarchy your provider can handle. These are not the kinds of edge cases and oddities you want to uncover for the first time when you’re rushing to respond to and recover from your next (sadly, inevitable) data loss incident.
What’s next for protecting your SaaS data against the humble human?
We began with one simple mistake that nearly chain-reacted into breaches at 1Password, Cloudflare, BeyondTrust, and other providers who have, for unknown reasons, chosen to stay silent.
If there is one simple mistake you should not make, it’s taking at least one proactive measure to protect your SaaS data from human error. Naturally, we’re biased toward automated backups, the only true remediation for data loss in the cloud, which you can’t control as carefully as on-premises infrastructure.
On that note, later this year, we’re introducing Rewind for Okta to help you keep your essential identity and access management data safe from loss—whether caused by simple human error or targeted ransomware attack. As always, we use end-to-end encryption with every snapshot or restoration and work with multiple SSO providers (like OAuth, OneLogin, and even Okta!) for secure authentication.
In the days after Okta was first compromised, Cloudflare security engineers found a threat actor trying to move laterally through their systems, including trying to jump into a brand-new data center location in São Paulo, Brazil. They dubbed their all-hands-on-deck remediation effort, some of which took place over the Thanksgiving holiday weekend, “Code Red.”
Today is an excellent day to take a meaningful step toward preventing a Code Red data loss situation of your own tomorrow.
