fosstars-rating-core

A framework for defining ratings for open-source projects. In particular, the framework offers a security rating for open-source projects that may be used to assess the security risk that comes with open-source components.

View the Project on GitHub SAP/fosstars-rating-core

Quality assurance

This page defines quality requirements and a verification procedure for scores and ratings. The purpose of a verification procedure is to make sure that a defined rating provides expected and adequate results.

Test vectors

Tests vectors may be defined for:

Given N features f[i], let’s define a test vector for a feature-based score as a pair (v, e) of the following elements:

Given N scores s[i], let’s define a test vector for a score-based score as a pair (v, e) of the following elements:

Defining test vectors

A set of test vectors defines quality requirements for a rating.

There are two main strategies for defining test vectors:

  1. By using real open-source projects. This way, we take an existing project, collect data about the project to build the vector v, and then define the interval e.
  2. By using abstract open-source projects. This way, we define both the vector v and the interval e.

The strategies above may be combined.

Test vectors based on real open-source projects

Define K test vectors real_test_vector[j] where j = 1..K:

  1. Select K existing open-source projects real_oss_project[j] where j = 1..K.
  2. For each real_oss_project[j] where j = 1..K:
    1. Build a vector of feature values v: for each feature f[i], gather the value v[i] where i = 1..N.
    2. Based on your knowledge of the project and your expertise, define an interval e for the vector v.
    3. Assign a test vector real_test_vector[j] = (v, e).

Notes:

Test vectors based on abstract open-source projects

Define L test vectors abstract_test_vector[j] where j = 1..L:

  1. Define a vector of feature values v: for each feature f[i], define v[i] where i = 1..N.
  2. Based on your expertise, define an interval e, define an interval e for the vector v.
  3. Assign a test vector abstract_test_vector[j] = (v, e).

Notes:

Verification procedure

An implementation of a rating of a score must pass all tests defined by test vectors.

Let’s say an implementation of a rating is defined as function rating(v). The function takes a vector v that contains values of features, and returns a rating score.

Then, the following verification procedure can be applied to make sure that the function rating(v) behaves as expected:

verify(test_vectors, rating) {
    for each test_vector from test_vectors {
        actual_rating = rating(test_vector.v)
        if actual_rating does not belong to test_vector.e {
            return Failed
        }
    }
    return Passed
}

This procedure may be also applied to a score.

Implementation

The verification procedure defined above is implemented in com.sap.oss.phosphor.fosstars.model.qa package.


Next: Tuning