Security ratings for open source projects

This section describes a security rating for open source projects. The rating is implemented in the OssSecurityRating class.

The rating assesses how well open source projects and their communities care about security. The rating may be used to estimate security risks for an application that uses open source components.

By definition, the security rating produces a score, a label and a confidence score. Here is a list of labels:

1. GOOD: the project implements relatively good security measures and in general cares about security.
2. MODERATE: the security level in the project is not good but definitely not too bad.
3. BAD: looks like the project doesn’t care well about security.
4. UNCLEAR: there is no enough data to reliably calculate a score and a label for the project.

The rating procedure uses thresholds for score and confidence to select one of the labels for an open source project. If a confidence score is lower than a certain value, then the project gets the UNCLEAR label. If a calculated score is higher than a certain threshold, then the project gets GOOD, MODERATE or BAD label.

The thresholds depend on scores for the following projects:

• Popular projects in Apache and Eclipse software foundations
• Popular Spring projects
• Popular FasterXML projects
• A number of other well-known projects such as curl, Netty, OpenSSL and so on.

Here is how the thresholds are calculated:

• First, calculate scores for the projects mentioned above.
• Second, sort the projects by scores (from low to high).
• Then, assign the BAD label to first 30% of the projects with the lowest scores. The highest score in this set becomes the threshold for the BAD label. In other words, a project gets the BAD label if its score is below this threshold.
• Next, assign the MODERATE label to the next 50% of the projects. The highest score in this set becomes the threshold for the MODERATE label accordingly. In other words, a project gets the MODERATE label if its score is below this threshold.
• Finally, the rest 20% of the projects get the GOOD label.

The main goal of this method is to reduce possible bias that may be introduced if experts set the thresholds directly. Instead of setting the thresholds directly, the experts gives a list of well-known in the industry projects and specify a desired fraction for the labels. At the moment, it is 20-50-30 that looks like a normal distribution. As a result, the thresholds don’t set a bar totally based on someone’s opinion. Instead, other open source projects are compared with the real, well-known and trusted ones. The procedure for re-calculating the thresholds is described here.

What the security rating takes into account

The security rating assesses the following factors of an open source project:

1. How well the project implements security testing.
2. Unpatched vulnerabilities.
3. Security reviews.
4. How well the community is aware of security.
5. How the project is active.
6. How the project is popular.
7. How the community commits to support the project.

To assess these factors, the security rating uses the following info about the open source project:

1. Whether the project runs CodeQL checks or not.
2. Whether the project runs CodeQL scans or not.
3. Whether the project uses LGTM checks for commits or not.
4. The worst LGTM grade for the project.
5. Whether the project uses FindSecBugs or not.
6. Whether the project uses Dependabot or not.
7. Whether the project uses OWASP Dependency Check or not.
8. Whether the project has a CVSS threshold for OWASP Dependency Check to fail the build or not.
9. Whether the project uses AddressSanitizer or not.
10. Whether the project uses MemorySanitizer or not.
11. Whether the project uses UndefinedBehaviorSanitizer or not.
12. Whether the project uses OWASP Enterprise Security API (ESAPI) or not.
13. Whether the project uses OWASP Java Encoder or not.
14. Whether the project uses OWASP Java HTML Sanitizer or not.
15. Whether the project uses NoHTTP tool or not.
16. Info about security reviews that have been done for the project.
17. Info about vulnerabilities in the project.
18. Whether the project is included to OSS-Fuzz project or not.
19. Whether the project signs artifacts or not.
20. Whether the project uses signed commits or not.
21. Whether the project has a security policy or not.
22. Whether the project has a security team or not.
23. Whether the project has a bug bounty program or not.
24. Whether the project belongs to Apache Foundation or not.
25. Whether the project belongs to Eclipse Foundation or not.
26. Whether the project is supported by a company or not.
27. Whether the project uses GitHub for development or not.
28. Programming languages used in the project.
29. Package managers used in the project.
30. Number of commits in the last three months.
31. Number of contributors in the last three months.
32. Number of stars in the GitHub repository.
33. Number of watchers in the GitHub repository.

For example, here is a detailed report that shows all sub-scores, structure and data that were used to calculate the security rating for curl.

Security ratings for well-known open source projects

Pre-calculated security ratings for well-known open source projects can be found in this report.

How to improve security ratings

Security ratings may be improved by contributing security enhancements to open source projects. For example:

• Contribute a security policy
• Improve a project’s LGTM grade by fixing issues reported by LGTM
• Enable CodeQL scans

In the bottom of a rating report for a project, you can find a list of advice for improving its rating.

Check out this page that shows how security ratings have been improved for a number of open source projects.

---

Next: Getting the security ratings