A framework for defining ratings for open source projects. In particular, the framework offers a security rating for open source projects that may be used to assess the security risk that comes with open source components.
This page defines quality requirements and a verification procedure for scoring functions and rating procedures. The purpose of a verification procedure is to make sure that a defined rating procedure produces expected and meaningful results.
Tests vectors may be defined for:
Given N
features f[i]
, let’s define a test vector for a feature-based scoring function
as a pair (v, e)
of the following elements:
v
is a vector (v[1], ... , v[N])
where v[i]
contains a value of feature f[i]
.e
is an interval [a, b]
of an expected score
where 0 <= a <= b <= 10
and abs(a - b) != 0
.Given N
scores s[i]
, let’s define a test vector for a score-based scoring function
as a pair (v, e)
of the following elements:
v
is a vector (v[1], ... , v[N])
where v[i]
contains a score s[i]
.e
is an interval [a, b]
of an expected score
where 0 <= a <= b <= 10
and abs(a - b) != 0
.A set of test vectors defines quality requirements for a rating procedure or scoring function.
There are two main strategies for defining test vectors:
v
, and then define the interval e
.v
and the interval e
.The strategies above may be combined.
Define K
test vectors real_test_vector[j]
where j = 1..K
:
K
existing open source projects real_oss_project[j]
where j = 1..K
.real_oss_project[j]
where j = 1..K
:
v
: for each feature f[i]
, gather the value v[i]
where i = 1..N
.e
for the vector v
.real_test_vector[j] = (v, e)
.Notes:
It’s good if the interval e
from test vectors cover the whole [0, 10]
interval.
In other words, union(real_test_vector[j].e)
should be close to [0, 10]
where j = 1..K
.
e
are provided by experts in the domain and by those who have good knowledge about the project.Define L
test vectors abstract_test_vector[j]
where j = 1..L
:
v
: for each feature f[i]
, define v[i]
where i = 1..N
.e
, define an interval e
for the vector v
.abstract_test_vector[j] = (v, e)
.Notes:
It’s good if the intervals e
from test vectors cover the whole [0, 10]
interval.
In other words, union(abstract_test_vector[j].e)
should be closed to [0, 10]
where j = 1..L
.
It’s good if values e
are provided by experts in the domain.
Rating procedures and scoring functions must pass all tests defined by test vectors.
Let’s say we have a rating procedure rating(v)
.
The procedure takes a vector v
that contains values of features, and returns a score.
Then, the following verification procedure can be applied to make sure that the procedure rating(v)
behaves as expected:
verify(test_vectors, rating) {
for each test_vector from test_vectors {
actual_rating = rating(test_vector.v)
if actual_rating does not belong to test_vector.e {
return Failed
}
}
return Passed
}
This procedure may be also applied to a scoring function.
The verification procedure defined above is implemented in com.sap.oss.phosphor.fosstars.model.qa package.
Next: Tuning