This section describes a security rating for open source projects. The rating is implemented in the OssSecurityRating class.
The rating assesses how well open source projects and their communities care about security. The rating may be used to estimate security risks for an application that uses open source components.
By definition, the security rating produces a score, a label and a confidence score. Here is a list of labels:
GOOD: the project implements relatively good security measures and in general cares about security.
MODERATE: the security level in the project is not good but definitely not too bad.
BAD: looks like the project doesn’t care well about security.
UNCLEAR: there is no enough data to reliably calculate a score and a label for the project.
The security rating uses
for score and confidence to assign one of the labels to an open source project.
If a confidence score is lower than a certain value, then the project gets the
If a calculated score is higher than a certain threshold, then the project gets
The thresholds depend on scores for the following projects:
Here is how the thresholds are calculated:
BADlabel to first 30% of the projects with the lowest scores. The highest score in this set becomes the threshold for the
BADlabel. In other words, a project gets the
BADlabel if its score is below this threshold.
MODERATElabel to the next 50% of the projects. The highest score in this set becomes the threshold for the
MODERATElabel accordingly. In other words, a project gets the
MODERATElabel if its score is below this threshold.
The main goal of this method is to reduce possible bias that may be introduced if experts set the thresholds directly. Instead of setting the thresholds directly, the experts gives a list of well-known in the industry projects and specify a desired fraction for the labels. At the moment, it is 20-50-30 that looks like a normal distribution. As a result, the thresholds don’t set a bar totally based on someone’s opinion. Instead, other open source projects are compared with the real, well-known and trusted ones.
The procedure for calculating the thresholds is implemented in SecurityRatingAnalysis notebook.
The security rating assesses the following factors of an open source project:
To assess these factors, the security rating uses the following info about the open source project:
For example, here is a detailed report that shows all sub-scores, structure and data that were used to calculate the security rating for curl.
Pre-calculated security ratings for well-known open source projects can be found in this report.
Security ratings may be improved by contributing security enhancements to open source projects. For example:
In the bottom of a rating report for a project, you can find a list of advice for improving its rating.
Check out this page that shows how security ratings have been improved for a number of open source projects.