Vulnerability triage automation: How Nurse Betty keeps Rewind secure

In the world of software development, managing external dependencies is crucial. These dependencies, while offering valuable functionality, can also introduce security vulnerabilities. At Rewind, we’re facing this challenge head-on and developed a powerful tool we call Nurse Betty to streamline our security efforts.

The dependency vulnerability problem

Every software project pulls in some form of external dependencies to take advantage of problems already solved or simplify interactions with an external service. Inevitably, these dependencies have security vulnerabilities, which lead to CVEs (Common Vulnerabilities and Exposures) being reported and patches being produced. With the proliferation of external dependencies, keeping up to date with the sheer number of vulnerabilities and updates to external dependencies can quickly become overwhelming.

There are many commercially available tools and services that help in this area, but they either come at a great cost or are unable to be utilized in certain deployment models. For example, some require kernel level drivers, which cannot be used when deploying services using serverless technologies.

With the sheer number of vulnerabilities being reported, it’s easy to miss critical issues amidst the noise. This is where Nurse Betty steps in to automate vulnerability triage.

Introducing Betty: our automated vulnerability triage nurse

We’ve been using Github’s Dependabot for some time to raise PRs for dependency updates due to vulnerabilities. Combined with Github’s Advanced Security module, it gives a reasonable security posture for external dependency management. However, Dependabot is indiscriminate—it has no idea how we’re using a dependency or what a vulnerability would mean in the context of the Rewind service.

At a local OWASP meetup (shameless plug for the excellent Ottawa chapter!), a talk was given on vulnerability assessment for common criteria evaluations. This contained some excellent information, including a piece on custom scoring models for vulnerabilities. Using this as a seed, our security engineering team discussed if we could apply a similar model at Rewind.  The first step towards doing this was clearly the most important in any new project: we needed a name! And it had to be cool. One of our team members suggested the concept of a Nurse, as they are often  triaging, prioritizing, making decisions and routing. Perfect! Out of this, Nurse Betty was born.

Nurse Betty is a Ruby application integrated with our GitHub Actions workflow. It automates the analysis of Dependabot alerts in GitHub, using a customized scoring matrix to assess the severity of each vulnerability. This matrix allows us to apply Rewind-specific criteria, ensuring that our efforts are focused on the most relevant and impactful issues.

Where did we start?

We started by discussing what factors could contribute to our scoring model. This included things like:

• Is the repository for a service that is customer facing?
• Is this an API-only service?
• Is it only accessed behind a VPN?
• Does it allow ingress/inbound traffic?
• Does it directly process or access customer data?
• Etc.

To see how effective our scoring system may be, we modelled it out in a spreadsheet with the scoring system and a good sampling of our GitHub repositories. This helped to see how the scores would impact the overall Dependabot ranking before we wrote any code. As an example, we determined that if a repository was hosting code that was externally customer facing we would give it a 1.5x multiplier, whereas if the vulnerability was in a development only dependency, we would give it a 0.3x multiplier.

Codifying Betty

Once we had the scoring system established, it was time to codify Betty. We set the following requirements:

• Betty had to be able to read Github Dependabot alerts
• Betty had to be able to raise alerts to our teams via Jira tickets
• Betty had to be able to route the right tickets to the right team

We decided to develop Betty using Ruby as Rewind is a predominantly Ruby shop. Accessing the Dependabot alerts was straightforward using the Github API, and to simplify deployment of Betty, she runs as a Github action on a daily schedule.

To route tickets to the right teams in Jira, we used GitHub topics configured on the repositories.  Betty then uses this mapping to route any tickets she creates to the Jira project for the correct team, thus making sure that the right tickets end up in the right hands. As Rewind manages all of our Github configuration in terraform, topics are an easy way to provide Betty with a simple routing mechanism for triaged tickets.

The final scoring configuration

Nurse Betty is driven by a configuration file that helps her triage vulnerabilities. This is the final scoring methodology we arrived at:

repository:
  base_score: 0.5
  topics:
    external-customer-facing: 1.5
    internal-customer-facing: 1.3
    api-only: 0.8
    behind-vpn: 0.5
    requires-credentials-for-full-access: 0.9
    no-ingress: 0.5
    processes-customer-data: 1.4
    critical: 2
    archived: 0
alert:
  base_score: 1.0
  factors:
    node_dep_in_rails_project: 0.2
    development_dependency: 0.3
    patch_available: 1.2
    exploit_available: 1.5
jira:
  ticket_threshold: 7.0

Any scoring system needs to be individualized to your own organization, but ours may serve as an inspiration. Here are some of the inputs you could give into a scoring matrix for your own scoring system.  

Let’s get into the details now of how Nurse Betty uses this to apply a Rewind-specific score to a vulnerability. Fundamentally, Betty is using the values above to derive a multiplier, which can be applied to a vulnerability’s CVSS (Common Vulnerability Scoring System) score. This score is a standardized scoring mechanism for security vulnerabilities and Dependabot reports the score in its alerts.  

Computing the multiplier

Betty first applies the alert scoring to Dependabot alert. The factors here are fairly straightforward:  Is there a dev dependency? Is there even a patch available? Is it exploitable? The alert score is computed from the base score (currently 1). Betty then computes a multiplier based on the topics assigned to the repository, again using a base score to start with (current 0.5).

The final multiplier is the average of the two multipliers.

Computing the final score

The final score is computed by applying the multiplier to the CVSS score in the alert.  The formula in use here is:

final_score = min(10, max(1, base_score × (1 + (avg_multiplier - 1)/2)))

Let’s break down this formula:

• (multiplier – 1) / 2.0 adjusts the impact of the multiplier to scale the base score.
• Adds that adjustment to 1, then multiplies by base_score.
• The result is:
  • Clamped between 1 and 10.
  • Rounded to 2 decimal places.

Real-world scoring example

Let’s take a Dependabot alert for CVE-2023-48220, which is a CVE against a Ruby dependency called devise_invitable. This has a CVSS of 5.7 and is flagged as having a patch available.

Let’s now look at this in the context of two different repositories:

• 1 repository has the following topics assigned:
  • external-customer-facing
  • Processes-customer-data
  • Critical
• A second repository has the following topics assigned:
  • behind-vpn
  • No-ingress

For repository 1, we would compute a multiplier as follows:

• Alert multiplier = 1.0 (base score) * 1.2 (patch available) = 1.2
• Repository multiplier = 0.5 (base score) * 1.5 (external) * 1.4 (customer data) * 2 (critical) = 4.2
• Average multiplier = (1.2 + 4.2) /2 = 2.7

For repository 2, we would compute a multiplier as follows:

• Alert multiplier = 1.0 (base score) * 1.2 (patch available) = 1.2
• Repository multiplier = 0.5 (base score) * 0.5 (vpn) * 0.5 (ingress) = 0.125
• Average multiplier = (1.2 + 0.125) / 2 = 0.6625

Now let’s score the alert for each of these repositories:

Repository 1

5.7 (CVSS) * (1 + (2.7 – 1) / 2 ) =  10.545 

Repository 2

5.7 (CVSS) * (1 + (0.6625 – 1) / 2 ) =  4.73

This example perfectly illustrates the benefits of Nurse Betty and a context-specific scoring.  She’s not adjusting the CVSS score, but she is applying some context to the score and helping us prioritize which vulnerabilities need attention. In this example, the vulnerability for repository 1 would be flagged and a Jira ticket would be created (since the score is above the ticket creation threshold of 7). The second repository (same vulnerability) would not create a ticket because of the score less than 7. Context has been applied in how the dependency is being used.

The impact of Nurse Betty

Nurse Betty has significantly reduced the burden of managing dependency vulnerabilities. Initially, we were faced with over 900 security notices from Dependabot. Through Nurse Betty’s automated analysis and triage, we’ve narrowed this down to approximately 50 actionable items. 

This represents a greater than 90% reduction in the number of vulnerabilities requiring immediate attention.

The future of Nurse Betty

We’re continuously looking to enhance Nurse Betty’s capabilities. We are currently experimenting with adding some AI-driven analysis to provide even deeper insights and more accurate vulnerability assessments. This will further refine our security processes and ensure we continue to proactively protect our systems. Results so far have been mixed, with this being heavily variable depending on the model being utilized and the complexity of the dependency tree (transitive dependencies specifically are problematic as they are not used directly in our code but are referenced by other dependencies).

Nurse Betty is currently closed-source but we have plans to open source her and allow her triaging expertise to assist others.

By automating the analysis and prioritization of dependency vulnerabilities, Nurse Betty allows our team to focus on the most critical security issues, ensuring our software remains robust and secure.