A framework for defining ratings for open-source projects. In particular, the framework offers a security rating for open-source projects that may be used to assess the security risk that comes with open-source components.
This section describes a security rating for open-source projects. The rating is implemented in the OssSecurityRating class.
The rating may be used to assess how well open-source projects and their communities care about security. This may then be used to estimate security risks that may affect an application when it uses open-source components.
By definition, the security rating produces a score, a label and a confidence score. Here is a list of labels:
GOOD
: the project implements relatively good security measures and in general cares about security.MODERATE
: the security level in the project is not good but definitely not too bad.BAD
: looks like the project doesn’t care well about security.UNCLEAR
: there is no enough data to reliably calculate a score and a label for the project.The security rating uses
thresholds
for score and confidence to assign one of the labels to an open-source project.
If a confidence score is lower than a certain value, then the project gets the UNCLEAR
label.
If a calculated score is higher than a certain threshold, then the project gets GOOD
, MODERATE
or BAD
label.
The thresholds depend on scores for the following projects:
Here is how the thresholds are calculated:
BAD
label to first 30% of the projects with the lowest scores.
The highest score in this set becomes the threshold for the BAD
label.
In other words, a project gets the BAD
label if its score is below this threshold.MODERATE
label to the next 50% of the projects.
The highest score in this set becomes the threshold for the MODERATE
label accordingly.
In other words, a project gets the MODERATE
label if its score is below this threshold.GOOD
label.The main goal of the method is to reduce a possible bias that may be introduced if experts set the thresholds directly. Instead of setting the thresholds directly, the experts gives a list of well-known projects in the industry and specify a desired fraction for the labels. At the moment, it is 20-50-30 that looks like a normal distribution. As a result, the thresholds don’t set a bar totally based on someone’s opinion. Instead, other open-source projects are compared with the real, well-known, established and trusted ones.
The procedure for calculating the thresholds is implemented in SecurityRatingAnalysis notebook.
The security rating tries to evaluate the following for an open-source project:
For example, here is a detailed report that shows all sub-scores, structure and data that were used to calculate a rating for curl.
Pre-calculated security ratings for well-known open-source projects can be found in this report.
Security ratings may be improved by contributing security enhancements to open-source projects. For example:
In the bottom of a rating report for a project, you can find a list of advices for improving its rating.
Check out this page that shows how security ratings have been improved for a number of open-source projects.