This information is intended to provide guidelines on how data are curated for the OSSelot project and how contributing works. The curator should be familiar with their preferred scanning tool (ours is [[FOSSology|Fossology]]) and have a general understanding of copyright law and in particular knowledge of FOSS licensing.

''Note:'' Whenever information is given that is specific to [[FOSSology|Fossology]], it is prepended with the keyword '''fossy'''.

== Preparation ==
* Obtain the component in source code form.
** Note the download URL.
* Naming convention:
** Try to follow the project’s naming and version convention, e.g. as given by the release’s git tag.
** If this is not consistent, use only lowercase letters.
** [package name]-[version number], e.g. angular-15.1.0.
* Analyze the component with a license scan tool (e.g. [[FOSSology|Fossology]], [[Scancode|Scancode]]).
** '''fossy:''' [[FOSSology|Fossology]] default settings for analysis:
*** 7. Select optional analysis:
**** Upload from file
**** Copyright/Email/URL/Author Analysis
**** Monk License Analysis, scanning for licenses performing a text comparison
**** Nomos License Analysis, scanning for licenses using regular expressions
**** Ojo License Analysis, scanning for licenses using SPDX-License-Identifier
*** 10. ScanCode Toolkit, scan for
**** License
**** Copyright
** [[Scancode|Scancode]] default options for analysis:<syntaxhighlight lang="bash">
scancode -cli --license-text –json [package name-version].json [package]
</syntaxhighlight>c: copyrights; l: licenses; i: file information; --license-text: include full license text

== Data curation ==
* A licensing expert reviews and analyzes the scanning results.
* [[FOSSology|Fossology]] can directly be used to review the results. The [[Scancode]] results must be reviewed with an external tool, e.g. [https://github.com/opossum-tool/opossumUI Opossum].
* Review is done on file level, i.e. every file in the source code tree for which at least one scanner found a result is analyzed.
** '''fossy:''' In [[FOSSology|Fossology]], you can browse through the relevant files by selecting "Go through all files with licenses and no clearing result".
* That means:
** scanner findings are confirmed, or
** scanner findings are corrected.
* If there are no findings for a file, the conclusion is NO ASSERTION (for SPDX tag ''LicenseConcluded'').
** '''fossy:''' In [[FOSSology|Fossology]], this is given by the clearing decision types "No license known" or "Irrelevant" or "Non-functional".

=== ''LicenseComments'' ===
In case a license conclusion is not obvious, the decision is explained.
* This is done with the following heuristic:<blockquote>The information in the file is: "[Quote licensing information in the source code file]" [Give reason for conclusion] Therefore, [license] is concluded.</blockquote>
* Example 1: No version<blockquote>The information in the file is: "This file is GPL'd." As no version of the GPL is given, GPL-1.0-or-later is concluded.</blockquote>
* Example 2: URL for license text<blockquote>The information in the file is: "This file is licensed under License A. You can find the license text at <nowiki>https://www.LicenseTextOfLicenseA.com</nowiki>." The URL contains the license text of License A, therefore License A is concluded. The information was retrieved on [date].</blockquote>
* '''fossy:''' In Fossology, the explanations are given in the "Comment" section which maps to the SPDX tag ''LicenseComments''.

=== Correcting scanner findings ===
The following list includes typical cases where scanner findings have to be corrected and how to do so.

==== Not a license ====
The scanner concludes a license from an expression in a file that is not actually a license expression at all. In this case, the incorrect license finding is removed.
* '''fossy:''' In [[FOSSology|Fossology]], the source of the scanner finding is highlighted when clicking on the number (#1) behind the scanner.
==== Not the file's license ====
The scanner concludes a license from a license expression that is part of the file’s content but not the license of the file itself. In this case, the incorrect license finding is removed.
==== License text ====
Files that contain only a license text (e.g. COPYING) are concluded by the scanners to be licensed under the respective license. This is usually not correct. Most license texts are not explicitly licensed, so the finding is removed. The GNU licenses contain a license statement for the license text itself which is concluded for these cases (''License-of-GNU-licenses'').
==== Imprecise finding ====
The scanner finding might be imprecise, e.g. w.r.t. to the version of a license, e.g. no version number is given. If this is the case, the imprecise finding is removed and the specified license and version is concluded. If no version is given, the lowest existing version with the -or-later extension is concluded.
==== Dual licensing ====
A file might offer a choice of two or more licenses under which it can be used. If the context requires to chose one specific license, this choice must be noted. However, all applicable licenses must be concluded. Also, dual license cases require additional post-processing, see section "Post-processing" below.
* '''fossy:''' In [[FOSSology|Fossology]], add the following text to the "Acknowledgement" section of the "Dual-license" finding to note the license choice, if applicable:<blockquote>To the extend files may be licensed under License A or License B, in this context License B has been chosen. This shall not restrict the freedom of other users to choose either License A or License B. For convenience, all license texts are provided.</blockquote>
==== License exceptions ====
In particular for the GNU licenses, there are a number of license exceptions.
* '''fossy:''' [[FOSSology|Fossology]] notes the license and the exception as separate findings. This is corrected to one finding using the SPDX license expression [License] WITH [exception], e.g. GPL-2.0-or-later WITH GCC-exception-2.0.
* '''fossy:''' If the [[FOSSology|Fossology]] license database does not yet contain these licenses, they have to be added.
==== Generic license texts ====
For some licenses, especially the BSD-type licenses, many variants of the license texts exist. The scanners often provide only the generic license texts. If an individual text differs from the generic text, the individual license text is provided.
* '''fossy:''' In [[FOSSology|Fossology]], click percentage of match to see differences.
* '''fossy:''' The individual text is copied from the file into the "License" section of [[FOSSology|Fossology]].
==== External references ====
Sometimes the file does not contain the name or text of a license but references an external resource such as a COPYRIGHT file in the root directory or a URL. In these cases, the external reference is checked and the detected license is concluded and the process is documented as a ''LicenseComment'' (in case of a URL, the date of access is noted).
==== (Partially) global license assignment ====
Sometimes there is a Readme file or similar that contains a statement assigning a license to several files within the source tree (e.g. all files in a specific directory). As such information is often outdated or does not account for individual licensing of files, it is not used to assign a license to a file here.
==== Acknowledgment ====
If a license has an acknowledgment requirement, the respective acknowledgment text is given. In particular for CC_BY licenses, the acknowledgment must contain the following information (if available): name of the creator, copyright notice, license notice, disclaimer, link to the material.
* '''fossy:''' In [[FOSSology|Fossology]], the acknowledgment text is given in the "Acknowledgement" section.

=== '''fossy:''' Bulk statements ===
In [[FOSSology|Fossology]], scanner findings can be confirmed, removed or corrected with bulk statements.
* When doing so, it is crucial to start with the shorter bulk statements as these can be part of a longer bulk statement which would then be modified by running the short bulk statement after the long one. For example (abbreviated):<blockquote>Short bulk statement: "This file is licensed under GPL version 2.0." Long bulk statement: "This file is licensed under GPL version 2.0. As a special exception, you may..."</blockquote> Here, the short bulk statement will modify the findings for the file with the long bulk statement. It should therefore be run first so that afterwards, the long bulk statement can correct the conclusion for the relevant files.
* Do not limit the scope of bulk statements, rather choose unique bulk statements. When reusing bulk statements for future uploads, the initial scope is not preserved, but they are applied to the entire upload, so it might yield false results.

=== Curating copyright statements ===
* Remove findings that were incorrectly identified as a copyright statement (e.g. license texts, code, etc.).
* Remove content from copyright statements that is not part of the copyright notice (e.g. formatting signs, license notices, comments on content, code, etc.).
* If the source code tree contains an AUTHORS file, the content of this is given as value to the SPDX tag ''PackageCopyrightText'' in the post-processing stage (see section “Post-processing” below).

=== Package license ===
Only If there is a LICENSE or COPYING or similar file in the root directory that states a main license for the package, we give this information as value to the SPDX tag ''PackageLicenseDeclared''.
* '''fossy:''' In [[FOSSology|Fossology]], this is marked as the "main license" by activating the star symbol. Caution: If the main license is a custom text, [[FOSSology|Fossology]] takes the standard template text anyway. This has to be corrected manually in the post-processing stage (see section “Post-processing” below).

Curation guideline

2023-08-03T07:33:50Z

Ckresse: Add section "Data curation" - part 1

This information is intended to provide guidelines on how data are curated for the OSSelot project and how contributing works. The curator should be familiar with their preferred scanning tool (ours is [[FOSSology|Fossology]]) and have a general understanding of copyright law and in particular knowledge of FOSS licensing.

''Note:'' Whenever information is given that is specific to [[FOSSology|Fossology]], it is prepended with the keyword '''fossy'''.

== Preparation ==
* Obtain the component in source code form.
** Note the download URL.
* Naming convention:
** Try to follow the project’s naming and version convention, e.g. as given by the release’s git tag.
** If this is not consistent, use only lowercase letters.
** [package name]-[version number], e.g. angular-15.1.0.
* Analyze the component with a license scan tool (e.g. [[FOSSology|Fossology]], [[Scancode|Scancode]]).
** '''fossy:''' [[FOSSology|Fossology]] default settings for analysis:
*** 7. Select optional analysis:
**** Upload from file
**** Copyright/Email/URL/Author Analysis
**** Monk License Analysis, scanning for licenses performing a text comparison
**** Nomos License Analysis, scanning for licenses using regular expressions
**** Ojo License Analysis, scanning for licenses using SPDX-License-Identifier
*** 10. ScanCode Toolkit, scan for
**** License
**** Copyright
** [[Scancode|Scancode]] default options for analysis:<syntaxhighlight lang="bash">
scancode -cli --license-text –json [package name-version].json [package]
</syntaxhighlight>c: copyrights; l: licenses; i: file information; --license-text: include full license text

== Data curation ==
* A licensing expert reviews and analyzes the scanning results.
* [[FOSSology|Fossology]] can directly be used to review the results. The [[Scancode]] results must be reviewed with an external tool, e.g. [https://github.com/opossum-tool/opossumUI Opossum].
* Review is done on file level, i.e. every file in the source code tree for which at least one scanner found a result is analyzed.
** '''fossy:''' In [[FOSSology|Fossology]], you can browse through the relevant files by selecting "Go through all files with licenses and no clearing result".
* That means:
** scanner findings are confirmed, or
** scanner findings are corrected.
* If there are no findings for a file, the conclusion is NO ASSERTION (for SPDX tag ''LicenseConcluded'').
** '''fossy:''' In [[FOSSology|Fossology]], this is given by the clearing decision types "No license known" or "Irrelevant" or "Non-functional".

=== ''LicenseComments'' ===
In case a license conclusion is not obvious, the decision is explained.
* This is done with the following heuristic:<blockquote>The information in the file is: "[Quote licensing information in the source code file]" [Give reason for conclusion] Therefore, [license] is concluded.</blockquote>
* Example 1: No version<blockquote>The information in the file is: "This file is GPL'd." As no version of the GPL is given, GPL-1.0-or-later is concluded.</blockquote>
* Example 2: URL for license text<blockquote>The information in the file is: "This file is licensed under License A. You can find the license text at <nowiki>https://www.LicenseTextOfLicenseA.com</nowiki>." The URL contains the license text of License A, therefore License A is concluded. The information was retrieved on [date].</blockquote>
* '''fossy:''' In Fossology, the explanations are given in the "Comment" section which maps to the SPDX tag ''LicenseComments''.

=== Correcting scanner findings ===
The following list includes typical cases where scanner findings have to be corrected and how to do so.

==== Not a license ====
The scanner concludes a license from an expression in a file that is not actually a license expression at all. In this case, the incorrect license finding is removed.
* '''fossy:''' In [[FOSSology|Fossology]], the source of the scanner finding is highlighted when clicking on the number (#1) behind the scanner.
==== Not the file's license ====
The scanner concludes a license from a license expression that is part of the file’s content but not the license of the file itself. In this case, the incorrect license finding is removed.
==== License text ====
Files that contain only a license text (e.g. COPYING) are concluded by the scanners to be licensed under the respective license. This is usually not correct. Most license texts are not explicitly licensed, so the finding is removed. The GNU licenses contain a license statement for the license text itself which is concluded for these cases (''License-of-GNU-licenses'').
==== Imprecise finding ====
The scanner finding might be imprecise, e.g. w.r.t. to the version of a license, e.g. no version number is given. If this is the case, the imprecise finding is removed and the specified license and version is concluded. If no version is given, the lowest existing version with the -or-later extension is concluded.
==== Dual licensing ====
A file might offer a choice of two or more licenses under which it can be used. If the context requires to chose one specific license, this choice must be noted. However, all applicable licenses must be concluded. Also, dual license cases require additional post-processing, see section "Post-processing" below.
* '''fossy:''' In [[FOSSology|Fossology]], add the following text to the "Acknowledgement" section of the "Dual-license" finding to note the license choice, if applicable:<blockquote>To the extend files may be licensed under License A or License B, in this context License B has been chosen. This shall not restrict the freedom of other users to choose either License A or License B. For convenience, all license texts are provided.</blockquote>
==== License exceptions ====
In particular for the GNU licenses, there are a number of license exceptions.
* '''fossy:''' [[FOSSology|Fossology]] notes the license and the exception as separate findings. This is corrected to one finding using the SPDX license expression [License] WITH [exception], e.g. GPL-2.0-or-later WITH GCC-exception-2.0.
* '''fossy:''' If the [[FOSSology|Fossology]] license database does not yet contain these licenses, they have to be added.
==== Generic license texts ====
For some licenses, especially the BSD-type licenses, many variants of the license texts exist. The scanners often provide only the generic license texts. If an individual text differs from the generic text, the individual license text is provided.
* '''fossy:''' In [[FOSSology|Fossology]], click percentage of match to see differences.
* '''fossy:''' The individual text is copied from the file into the "License" section of [[FOSSology|Fossology]].
==== External references ====
Sometimes the file does not contain the name or text of a license but references an external resource such as a COPYRIGHT file in the root directory or a URL. In these cases, the external reference is checked and the detected license is concluded and the process is documented as a ''LicenseComment'' (in case of a URL, the date of access is noted).
==== (Partially) global license assignment ====
Sometimes there is a Readme file or similar that contains a statement assigning a license to several files within the source tree (e.g. all files in a specific directory). As such information is often outdated or does not account for individual licensing of files, it is not used to assign a license to a file here.
==== Acknowledgment ====
If a license has an acknowledgment requirement, the respective acknowledgment text is given. In particular for CC_BY licenses, the acknowledgment must contain the following information (if available): name of the creator, copyright notice, license notice, disclaimer, link to the material.
* '''fossy:''' In [[FOSSology|Fossology]], the acknowledgment text is given in the "Acknowledgement" section.

Curation guideline

2023-08-03T06:50:28Z

Ckresse: First section "Preparation"

Main Page

2023-08-03T06:27:39Z

Ckresse:

__NOTOC__
== Welcome to the O Wiki! ==

This Wiki was created to facilitate day-to-day work with the resources of the [https://www.osselot.org O project], especially when accessing them in batch mode, e.g. as part of a software release build.

==[[Search]] for a package==
[[Search|Find out]] whether a particular version of a software package is supported by O and has already been curated. A particular Web script is provided for this purpose, and an interactive Web interface is available to test and use this feature.

==Obtain [[Disclosure_files|disclosure files]]==
A shell script is provided to encapsulate the [[Search|Web search script]] and obtain the related disclosure files. Another interactive Web interface is available to test and use this feature.

==Obtain [[Licenses|licensing information]]==
Another Web script is available that also encapsulates the internal algorithm of the [[Search|Web search script]], but then generates a list of licenses that are used in a given software package. The script accepts as argument either a package name or a package name along with a version. In the former case the licenses of all available versions are listed, whereas in the latter the output is restricted to the specified version.

==Obtain curation data in various format using a [[REST]] interface==
The entire curation data of a software package can be retrieved in [[JSON]] format, curation data of a particular version can be retrieved in [[RDF-XML]], [[SPDX2TV]] or [[YAML]] format.

==Reuse existing material in case of version mismatch==
If a particular version of a software package has not been curated before, but another one that may be close to it has, then FOSSology's reuse feature can be applied. Details are given in the presentation and video material on the O home page: Please check out [https://www.osselot.org/index.php?s=presentations "Use case 2" at the O presentations].

==Contributing to the O project==
How to contribute to the project if a package that is not yet included with O has been externally curated? Contributions are greatly appreciated, and therefore we would like to encourage as many users as possible to contribute. The more versions of more packages that are curated, the more beneficial the O project will be. However, to maintain confidence in the material, a rigorous vetting process was instituted. Volunteers are asked to first contact the O Officer via [mailto:office@osadl.org?subject=OSSelot-volunteer email]. The easiest next step is then probably to arrange a video conference, get to know each other, and understand the basic principles of the O curation process. From that moment, new software packages can be curated and provided in the fork tree. The maintainer will then review the newly provided curation data in close collaboration with the contributor, and once the review is successful, the new curation data will be included and made publicly available through the repository. The contributor's and reviewer's names will be indicated in the README file of the package.

==Best practices==
====FOSSology====
In order to use the [[FOSSology]] Open Source curation administration tool in connection with the O project (when re-using the data and also when contributing to the project), a number of conventions should be observed.
====Scancode====
Normally, the [[Scancode]] Open Source scanning tool is used under the control of [[FOSSology]] in this project; however, to fine-tune or confirm the results it may be necessary to run the tool separately from command line. If this is done, the command line options should match the conventions of the O project.
====How to curate data====
High-quality curation data are the cornerstone of the O project. To ensure that this quality is maintained, every contribution is thoroughly reviewed and only curators with sufficient expertise in FOSS licensing contribute to the database. A [[curation guideline]] on how data are curated for the O project is given here to ensure consistency.

Main Page

2023-08-03T06:24:46Z

Ckresse: Add "How to curate data" section introduction.

__NOTOC__
== Welcome to the O Wiki! ==

This Wiki was created to facilitate day-to-day work with the resources of the [https://www.osselot.org O project], especially when accessing them in batch mode, e.g. as part of a software release build.

==[[Search]] for a package==
[[Search|Find out]] whether a particular version of a software package is supported by O and has already been curated. A particular Web script is provided for this purpose, and an interactive Web interface is available to test and use this feature.

==Obtain [[Disclosure_files|disclosure files]]==
A shell script is provided to encapsulate the [[Search|Web search script]] and obtain the related disclosure files. Another interactive Web interface is available to test and use this feature.

==Obtain [[Licenses|licensing information]]==
Another Web script is available that also encapsulates the internal algorithm of the [[Search|Web search script]], but then generates a list of licenses that are used in a given software package. The script accepts as argument either a package name or a package name along with a version. In the former case the licenses of all available versions are listed, whereas in the latter the output is restricted to the specified version.

==Obtain curation data in various format using a [[REST]] interface==
The entire curation data of a software package can be retrieved in [[JSON]] format, curation data of a particular version can be retrieved in [[RDF-XML]], [[SPDX2TV]] or [[YAML]] format.

==Reuse existing material in case of version mismatch==
If a particular version of a software package has not been curated before, but another one that may be close to it has, then FOSSology's reuse feature can be applied. Details are given in the presentation and video material on the O home page: Please check out [https://www.osselot.org/index.php?s=presentations "Use case 2" at the O presentations].

==Contributing to the O project==
How to contribute to the project if a package that is not yet included with O has been externally curated? Contributions are greatly appreciated, and therefore we would like to encourage as many users as possible to contribute. The more versions of more packages that are curated, the more beneficial the O project will be. However, to maintain confidence in the material, a rigorous vetting process was instituted. Volunteers are asked to first contact the O Officer via [mailto:office@osadl.org?subject=OSSelot-volunteer email]. The easiest next step is then probably to arrange a video conference, get to know each other, and understand the basic principles of the O curation process. From that moment, new software packages can be curated and provided in the fork tree. The maintainer will then review the newly provided curation data in close collaboration with the contributor, and once the review is successful, the new curation data will be included and made publicly available through the repository. The contributor's and reviewer's names will be indicated in the README file of the package.

==Best practices==
====FOSSology====
In order to use the [[FOSSology]] Open Source curation administration tool in connection with the O project (when re-using the data and also when contributing to the project), a number of conventions should be observed.
====Scancode====
Normally, the [[Scancode]] Open Source scanning tool is used under the control of [[FOSSology]] in this project; however, to fine-tune or confirm the results it may be necessary to run the tool separately from command line. If this is done, the command line options should match the conventions of the O project.
====How to curate data====
High-quality curation data are the cornerstone of the O project. To ensure that this quality is maintained, every contribution is thoroughly reviewed and only curators with sufficient expertise in FOSS licensing contribute to the database. A guideline on how data are curated for the O project is given here to ensure consistency.