This section describes a security rating for open-source projects. The rating is implemented in the OssSecurityRating class.
The rating may be used to assess how well open-source projects and their communities care about security. This may then be used to estimate security risks that may affect an application when it uses open-source components.
By definition, the security rating produces a score, a label and a confidence score. Here is a list of labels:
GOOD: the project implements relatively good security measures and in general cares about security.
MODERATE: the security level in the project is not good but definitely not too bad.
BAD: looks like the project doesn’t care well about security.
UNCLEAR: there is no enough data to reliably calculate a score and a label for the project.
The security rating uses
for score and confidence to assign one of the labels to an open-source project.
If a confidence score is lower than a certain value, then the project gets the
If a calculated score is higher than a certain threshold, then the project gets
The thresholds depend on scores for the following projects:
Here is how the thresholds are calculated:
BADlabel to first 30% of the projects with the lowest scores. The highest score in this set becomes the threshold for the
BADlabel. In other words, a project gets the
BADlabel if its score is below this threshold.
MODERATElabel to the next 50% of the projects. The highest score in this set becomes the threshold for the
MODERATElabel accordingly. In other words, a project gets the
MODERATElabel if its score is below this threshold.
The main goal of the method is to reduce a possible bias that may be introduced if experts set the thresholds directly. Instead of setting the thresholds directly, the experts gives a list of well-known projects in the industry and specify a desired fraction for the labels. At the moment, it is 20-50-30 that looks like a normal distribution. As a result, the thresholds don’t set a bar totally based on someone’s opinion. Instead, other open-source projects are compared with the real, well-known, established and trusted ones.
The procedure for calculating the thresholds is implemented in SecurityRatingAnalysis notebook.
The security rating tries to evaluate the following for an open-source project:
For example, here is a detailed report that shows all sub-scores, structure and data that were used to calculate a rating for curl.
Pre-calculated security ratings for well-known open-source projects can be found in this report.
Security ratings may be improved by contributing security enhancements to open-source projects. For example:
In the bottom of a rating report for a project, you can find a list of advices for improving its rating.
Check out this page that shows how security ratings have been improved for a number of open-source projects.