A framework for defining ratings for open source projects. In particular, the framework offers a security rating for open source projects that may be used to assess the security risk that comes with open source components.
This page contains a comparison between the features provided by Fosstars and OSSF ScoreCard. The comparison is based on the releases available in December 2021 (Fosstars 1.13.0 and OSSF ScoreCard 4.12.0).
Fosstars scores and OSSF Scorecard checks are a bit different and hence can not be mapped one to one. Hence, what is in this comparison mentioned as Features can be (more or less) mapped to a Fosstars Score and Scorecard check.
| Feature | OSSF ScoreCard (Weight) | Fosstars (Weight) | Comments |
|---|---|---|---|
| Binary-Artifacts | ✅ (high) | ✅ (high) | - |
| Branch protection | ✅ (high) | ⛔️ | - |
| Code Review | ✅ (high) | ⛔️ | - |
| Dangerous-Workflow | ✅ (high) | ⛔️ | - |
| Signed-Releases | ✅ (high) | ⛔️ | - |
| Token-Permissions | ✅ (high) | ⛔️ | - |
| LGTM score | ⛔️ | ✅ (high) | - |
| CodeQL use | ⛔️ | ✅ (high) | - |
| OWASP Dependency Check | ⛔️ | ✅ (high) | - |
| Dependency-Update-Tool | ✅ (high) | ✅ (high) | - |
| Maintained | ✅ (high) | ✅ (high) | - |
| Vulnerabilities | ✅ (high) | ✅ (high) | - |
| Project activity | ⛔️ | ✅ (medium) | - |
| Project popularity | ⛔️ | ✅ (medium) | - |
| Community commitment | ⛔️ | ✅ (medium) | - |
| Packaging | ✅ (medium) | ⛔️ | - |
| Pinned-Dependencies | ✅ (medium) | ⛔️ | - |
| Fuzzing | ✅ (medium) | ✅ (medium) | - |
| SAST | ✅ (medium) | ✅ (medium) | - |
| Security-Policy | ✅ (medium) | ✅ (medium) | - |
| License | ✅ (low) | ✅ (low) | - |
| Run CI-Tests | ✅ (low) | ⛔️ | - |
| CII-Best-Practices | ✅ (low) | ⛔️ | - |
| Contributors | ✅ (low) | ⛔️ | - |
Scan the projects to detect any existence of executables (.class, .pyc….). In terms of security aspect it makes sense to scan through the project and identify if there are any security threats. But there are projects which also stores binary executables as part of the project
These projects will get a BAD score.
Checks if the default and release branches are protected by protection settings. To check this an Admin privilege token needs to be used. Some of the main rules and regulations
--force command on public branches, which overwrites code irrevocably. This protection prevents the rewriting of public history without external notice.Also the evaluation is done in Tiers. i.e if all the conditions meet then high value, otherwise lower values.
Check if the project enables mandatory code review before a PR is merged.
This check determines whether the project’s GitHub Action workflows has dangerous code patterns. Some examples of these patterns are untrusted code checkouts, logging github context and secrets, or use of potentially untrusted inputs in scripts. More information here. If no dangerous work flows exist than high score otherwise low.
This check tries to determine if the project cryptographically signs release artifacts. It is currently limited to repositories hosted on GitHub, and does not support other source hosting repositories (i.e., Forges).
Signed releases attest to the provenance of the artifact.
This check looks for the following filenames in the project’s last five releases: *.minisig , *.asc (pgp), *.sig, *.sign.
This check determines whether the project’s automated workflows tokens are set to read-only by default. It is currently limited to repositories hosted on GitHub, and does not support other source hosting repositories (i.e., Forges). If readonly tokens are used more secure. High score.
This check tries to determine if the project is published as a package. It is currently limited to repositories hosted on GitHub, and does not support other source hosting repositories (i.e., Forges).
You can create a package in several ways:
This check tries to determine if the project is an application that has declared and pinned its dependencies. A “pinned dependency” is a dependency that is explicitly set to a specific version instead of allowing a range of versions. It is currently limited to repositories hosted on GitHub, and does not support other source hosting repositories (i.e., Forges). If this project is a library (not an application), this check should automatically pass (but see limitations below).
The check works by looking for:
This ensures that the dependency and the project management is more in control. However, pinning dependencies can inhibit software updates, either because of a security vulnerability or because the pinned version is compromised. Mitigate this risk by:
Checks if the projects runs tests on PR before the merge. The check works by looking for a set of CI-system names in GitHub CheckRuns and Statuses among the recent commits (~30). A CI-system is considered well-known if its name contains any of the following: appveyor, buildkite, circleci, e2e, github-actions, jenkins, mergeable, test, travis-ci.
Checks whether the project has earned CII Best Coding practices followed badge. The criterion To earn the passing badge, the project MUST:
If the badge is present high score.
This check tries to determine if the project has recent contributors from multiple organizations (e.g., companies). It is currently limited to repositories hosted on GitHub, and does not support other source hosting repositories (i.e., Forges).
The check looks at the Company field on the GitHub user profile for authors of recent commits. To receive the highest score, the project must have had contributors from at least 3 different companies in the last 30 commits; each of those contributors must have had at least 5 commits in the last 30 commits.
This check tries to determine if the project uses a dependency update tool, specifically dependabot or renovatebot. If present high score.
This check determines whether the project is actively maintained. If the project is archived, it receives the lowest score. If there is at least one commit per week during the previous 90 days, the project receives the highest score. In rating core we have something similar to determine the
OSSF score card uses OSV, to detect the list of vulnerabilities if present for the project. Rating core uses
Rating core detection of vulnerabilities is not the most efficient way.
This check tries to determine if the project uses fuzzing by checking if the repository name is included in the OSS-Fuzz project list. Fuzz testing is a well-known technique for uncovering programming errors in software. Many of these detectable errors, like buffer overflow, can have serious security implications. Google has found thousands of security vulnerabilities and stability bugs by deploying guided in-process fuzzing of Chrome components, and we now want to share that service with the open source community. If the project is enrolled in OSS-Fuzz then high score.
Check if a SAST plugin of some sort exists. Ensuring that the project is checked for unknown bugs. Rating core has something similar
Check if any security policy exist as part of the project. Usually stored in Security tab of the GitHub repo.
Checks if the project has a Published license. Fosstars rating core has something similar where it check
Checks how is OWASP Dependency Check used and what is the threshold for OWASP Dependency Check.
Checks the programming languages used and the worst LGTM grade of the project.
Evaluates how active a project is. It’s based on number of commits and contributors in the last 3 months.
The project popularity is based on number of stars and watchers. Here is how a number of stars contributes to the score: 0 -> 0.00 (min), 2500 -> 2.50, 5000 -> 5.00, 10000 -> 10.00 (max) Here is how a number of watchers contributes to the score: 0 -> 0.00 (min), 450 -> 1.50, 750 -> 2.50, 3000 -> 10.00 (max)
Checks if the community is backed by a company or a foundation (e.g. Apache, Eclipse, …)