Merge branch 'develop' into license-detection-improvements-and-review

Author: AyanSinhaMahapatra

CHANGELOG.rst

@@ -2,27 +2,81 @@ Changelog
 =========
 
 v33.0.0 (next next, roadmap)
-
 ----------------------------
 
-
 - We now support new package manifest formats:
 
   - OpenWRT packages.
   - Yocto/BitBake .bb recipes.
 
+- Fallback packages for non-native dependencies of SCTK.
+- Dependencies for 
+- Support for copyright detection objects.
 
-v32.0.0 (next, roadmap)
------------------------
+v32.1.0 (next, roadmap)
+----------------------------
+
+- A new field in packages with the license category for the
+  detected license expression and also an API function to
+  compute license categories from license expressions.
+  See https://github.com/nexB/scancode-toolkit/issues/2897
+
+- More support for tabular output formats: New command-line
+  options for XSLX output, and the old `--csv` command line
+  option is removed.
+  See https://github.com/nexB/scancode-toolkit/issues/830
+
+- `--unknown-licenses` is removed and this is always enabled
+  and only used in case of improper detections automatically.
+  Also tag all license rules with required phrases to improve
+  license detection and reduce false positives.
+  See https://github.com/nexB/scancode-toolkit/issues/3300
+
+- A new `--todo` option is added to show the todo items that
+  should be reviewed, which are ambiguous license/package
+  detections.
+
+- File categorization support added, a post scan plugin tagging
+  files with priority levels for review, and also take advantage
+  of these in other summary plugins.
+  See https://github.com/nexB/scancode-toolkit/issues/1745
+
+
+v32.0.2 - 2023-05-26
+---------------------
+
+This is a minor bugfix release with the following update:
+
+- New release v30.1.1 of license-expression with support for new license keys
+  added. Also fail verbosely in `build_spdx_license_expression` for invalid and
+  deprecated license keys.
+
+
+v32.0.1 - 2023-05-23
+---------------------
+
+This is a minor bugfix release.
+
+There are fixes for two issues in this release:
+- https://github.com/nexB/scancode-toolkit/issues/3407:
+  here in typecode we had an improper import of ctypes.utils
+  and this is fixed in a new release v30.0.1 of typecode
+- https://github.com/nexB/scancode-toolkit/issues/3408 
+  the setup.cfg and setup-mini.cfg was not aligned for plugin
+  entrypoints.
+
+
+v32.0.0 - 2023-05-23
+---------------------
 
 Important API changes:
 ~~~~~~~~~~~~~~~~~~~~~~
 
 This is a major release with major API and output format changes and signicant
 feature updates.
 
-In particular changed to the output format for the licenses and packages, and
-we changed some of the command line options.
+In particular the output format has changed for the licenses and packages, and
+also for some of the command line options.
 
 The output format version is now 3.0.0.
 
@@ -146,7 +200,9 @@ License detection:
     These expressions are parallel to detections.
 
   - The ``declared_license`` attribute is renamed ``extracted_license_statement``
-    and is now a YAML-encoded string.
+    and is now a YAML-encoded string, which can be parsed to recreate the
+    original extracted license statement. Previously this used to be nested
+    python objects lists/dicts/string, but now this is always a YAML string.
 
     See `license updates documentation <https://scancode-toolkit.readthedocs.io/en/latest/reference/license-detection-reference.html#change-in-license-data-format-package>`_
     for examples and details.

ROADMAP.rst

@@ -29,7 +29,8 @@ This is for SCTK first.
 
 Status: This has been completed in SCTK and also included in SCIO. We use
 an updated --summary option and a new license clarity score for this.
-Some work is still in progress as part of 3.) "detections"
+We also have LicenseDetections for resources/packages and a top level
+unique license detections as a summary.
 
 
 2. Package files.
@@ -50,11 +51,15 @@ This has been completed in SCTK and also included in SCIO.
 - package: object of its own, and related set of files, not always in the same
   directory
 
+This is completed in SCTK.
+
 *License*:
 
 - many detections in a file at different locations, could be merged into a single reported license
 - same for primary licenses
 
+This is completed in SCTK.
+
 *Copyright*:
 
 - Copyright and author detection, which are tracked at the line level
@@ -76,6 +81,7 @@ Roadmap
 
 - SCTK: add primary license field in package output and populate this based on
   package-type/ecosystem conventions.
+- SCTK: also populate secondary license fields 
 - SCIO: add primary license field in DiscoveredPackage models and feed it with
   the data from packages
 - SCIO: Do we track secondary? or is this just data aggregated on the fly.
@@ -87,7 +93,7 @@ Roadmap
 
 - This is closely tied to the primary license detection and should focus
   on package manifests and key files. 
-
+- Support copyright parsing from all package ecosystems.
 
 3. Package files
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -122,3 +128,4 @@ Roadmap
 - Revamp how common list of suprrious licenses are detected (this is a bug)
 - Use important key phrases for license detection https://github.com/nexB/scancode-toolkit/issues/2637
 
+This is mostly completed, for follow up see https://github.com/nexB/scancode-toolkit/issues/2878.

docs/source/conf.py

@@ -21,16 +21,25 @@
 copyright = "nexB Inc. and others."
 author = "AboutCode.org authors and contributors"
 
-
 # -- General configuration ---------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
     "sphinx.ext.intersphinx",
+    "sphinx_reredirects",
 ]
 
+
+# Redirects for olds pages
+# See https://documatt.gitlab.io/sphinx-reredirects/usage.html
+redirects = {
+    "explanations/index.html": "../reference/index.html",
+    "explanations/overview.html": "../reference/overview.html",
+    "explanations/license-detection-reference.html": "../reference/license-detection-reference.html"
+}
+
 # This points to aboutcode.readthedocs.io
 # In case of "undefined label" ERRORS check docs on intersphinx to troubleshoot
 # Link was created at commit - https://github.com/nexB/aboutcode/commit/faea9fcf3248f8f198844fe34d43833224ac4a83

docs/source/contribute/contrib_doc.rst

@@ -303,6 +303,12 @@ Style Conventions for the Documentaion
     ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are
     frequently used in multiple files.
 
+8. Redirects
+
+    If layouts of doc pages are being changed and these could be referenced elsewhere, these should
+    be added in the `redirects` mapping in `conf.py`. For examples on using these see
+    https://documatt.gitlab.io/sphinx-reredirects/usage.html
+
 Converting from Markdown
 ------------------------
 

etc/thirdparty/virtualenv.pyz

@@ -1,7 +1,7 @@
 about_resource: virtualenv.pyz
 name: get-virtualenv
-version: 20.16.3
-download_url: https://github.com/pypa/get-virtualenv/blob/20.16.3/public/virtualenv.pyz
+version: 20.23.0
+download_url: https://github.com/pypa/get-virtualenv/raw/20.23.0/public/virtualenv.pyz
 description: virtualenv is a tool to create isolated Python environments.
 homepage_url: https://github.com/pypa/virtualenv
 license_expression: lgpl-2.1-plus AND (bsd-new OR apache-2.0) AND mit AND python AND bsd-new
@@ -10,6 +10,6 @@ copyright: Copyright (c) The Python Software Foundation and others
 redistribute: yes
 attribute: yes
 track_changes: yes
-package_url: pkg:github/pypa/get-virtualenv@20.16.3#public/virtualenv.pyz
+package_url: pkg:github/pypa/get-virtualenv@20.23.0#public/virtualenv.pyz
 notes: this archive has been modified from the original to remove extra
  embedded wheels that are not needed as we support only Python 3.7+

etc/thirdparty/virtualenv.pyz.ABOUT

@@ -9,7 +9,7 @@ chardet==5.0.0
 charset-normalizer==2.1.0
 click==8.1.3
 colorama==0.4.5
-commoncode==31.0.0
+commoncode==31.0.2
 construct==2.10.68
 container-inspector==31.1.0
 cryptography==37.0.4
@@ -34,7 +34,7 @@ jaraco.functools==3.5.1
 javaproperties==0.8.1
 Jinja2==3.1.2
 jsonstreams==0.6.0
-license-expression==30.0.0
+license-expression==30.1.1
 lxml==4.9.2
 MarkupSafe==2.1.2
 more-itertools==8.13.0
@@ -54,7 +54,7 @@ publicsuffix2==2.20191221
 pyahocorasick==2.0.0
 pycparser==2.21
 pygmars==0.7.0
-Pygments==2.12.0
+Pygments==2.13.0
 pymaven-patch==0.3.0
 pyparsing==3.0.9
 pytz==2022.1
@@ -67,7 +67,7 @@ soupsieve==2.3.2.post1
 spdx-tools==0.7.0rc0
 text-unidecode==1.3
 toml==0.10.2
-typecode==30.0.0
+typecode==30.0.1
 typecode-libmagic==5.39.210531
 typing-extensions==4.3.0
 urllib3==1.26.11

requirements.txt

@@ -1,6 +1,6 @@
 [metadata]
 name = scancode-toolkit
-version = 32.0.0rc4
+version = 32.0.2
 license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
 
 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390
@@ -64,13 +64,13 @@ python_requires = >=3.7
 
 install_requires =
     attrs >= 18.1,!=20.1.0;python_version<'3.11'
-    attrs >= 22.1;python_version>='3.11'
+    attrs >= 22.1.0;python_version>='3.11'
     Beautifulsoup4 >= 4.0.0
     boolean.py >= 4.0
     chardet >= 3.0.0
     click >= 6.7, !=7.0
     colorama >= 0.3.9
-    commoncode >= 31.0.0
+    commoncode >= 31.0.2
     container-inspector >= 31.0.0
     debian-inspector >= 31.0.0
     dparse2 >= 0.7.0
@@ -85,7 +85,7 @@ install_requires =
     javaproperties >= 0.5
     jinja2 >= 2.7.0
     jsonstreams >= 0.5.0
-    license_expression >= 30.0.0
+    license_expression >= 30.1.1
     lxml >= 4.9.2
     MarkupSafe >= 2.1.2
     packageurl_python >= 0.9.0
@@ -97,7 +97,7 @@ install_requires =
     pkginfo2 >= 30.0.0
     pip-requirements-parser >= 32.0.1
     pluggy >= 1.0.0
-    plugincode >= 31.0.0
+    plugincode >= 32.0.0
     publicsuffix2
     pyahocorasick >= 2.0.0
     pygmars >= 0.7.0
@@ -111,7 +111,7 @@ install_requires =
     urlpy
     xmltodict >= 0.11.0
     zipp >= 3.0.0; python_version < "3.9"
-    typecode >= 30.0.0
+    typecode >= 30.0.1
 #    typecode[full] >= 30.0.0
 #    extractcode[full] >= 31.0.0
 
@@ -129,15 +129,17 @@ testing =
     pytest >= 6, != 7.0.0
     pytest-xdist >= 2
     aboutcode-toolkit >= 7.0.2
+    pycodestyle >= 2.8.0
     twine
     black
     isort
     vendorize >= 0.3.0
     pytest-rerunfailures
 
 docs =
-    Sphinx >= 3.3.1
-    sphinx-rtd-theme >= 0.5.0
+    Sphinx == 5.1.0
+    sphinx_rtd_theme >= 0.5.1
+    sphinx-reredirects >= 0.1.2
     doc8 >= 0.8.1
 
 # linux-only package handling
@@ -152,6 +154,7 @@ console_scripts =
     scancode = scancode.cli:scancode
     scancode-reindex-licenses = licensedcode.reindex:reindex_licenses
     scancode-license-data = licensedcode.license_db:dump_scancode_license_data
+    regen-package-docs = packagedcode.regen_package_docs:regen_package_docs
 
 # These are configurations for ScanCode plugins as setuptools entry points.
 # Each plugin entry hast this form:
@@ -163,7 +166,6 @@ console_scripts =
 scancode_pre_scan =
     ignore = scancode.plugin_ignore:ProcessIgnore
     facet = summarycode.facet:AddFacet
-    classify = summarycode.classify_plugin:FileClassifier
 
 
 # scancode_scan is the entry point for scan plugins that run a scan after the
@@ -194,6 +196,7 @@ scancode_post_scan =
     filter-clues = cluecode.plugin_filter_clues:RedundantCluesFilter
     consolidate = summarycode.plugin_consolidate:Consolidator
     license-references = licensedcode.licenses_reference:LicenseReference
+    classify = summarycode.classify_plugin:FileClassifier
 
 
 # scancode_output_filter is the entry point for filter plugins executed after

setup-mini.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = scancode-toolkit
-version = 32.0.0rc4
+version = 32.0.2
 license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
 
 # description must be on ONE line https://github.com/pypa/setuptools/issues/1390
@@ -70,7 +70,7 @@ install_requires =
     chardet >= 3.0.0
     click >= 6.7, !=7.0
     colorama >= 0.3.9
-    commoncode >= 31.0.0
+    commoncode >= 31.0.2
     container-inspector >= 31.0.0
     debian-inspector >= 31.0.0
     dparse2 >= 0.7.0
@@ -85,7 +85,7 @@ install_requires =
     javaproperties >= 0.5
     jinja2 >= 2.7.0
     jsonstreams >= 0.5.0
-    license_expression >= 30.0.0
+    license_expression >= 30.1.1
     lxml >= 4.9.2
     MarkupSafe >= 2.1.2
     packageurl_python >= 0.9.0
@@ -111,8 +111,8 @@ install_requires =
     urlpy
     xmltodict >= 0.11.0
     zipp >= 3.0.0; python_version < "3.9"
-    typecode >= 30.0.0
-    typecode[full] >= 30.0.0
+    typecode >= 30.0.1
+    typecode[full] >= 30.0.1
     extractcode[full] >= 31.0.0
 
 
@@ -129,17 +129,18 @@ testing =
     pytest >= 6, != 7.0.0
     pytest-xdist >= 2
     aboutcode-toolkit >= 7.0.2
+    pycodestyle >= 2.8.0
     twine
     black
     isort
     vendorize >= 0.3.0
     pytest-rerunfailures
 
 docs =
-    Sphinx >= 4.3.0
+    Sphinx == 5.1.0
     sphinx_rtd_theme >= 0.5.1
-    docutils < 0.17
-    doc8
+    sphinx-reredirects >= 0.1.2
+    doc8 >= 0.8.1
 
 # linux-only package handling
 packages =