[BUG] NameError: name 'unicode' is not defined

[WilliamHolmes]

When using detect-secrets in our Jenkins pipeline, we started to see this issue (only recently)...

pip install --upgrade "git+https://github.com/ibm/detect-secrets.git@master#egg=detect-secrets"

 detect-secrets scan --update .secrets.baseline
/home/jenkins/.local/lib/python3.12/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.0.1)/charset_normalizer (3.4.4) doesn't match a supported version!
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/core/secrets_collection.py", line 411, in _extract_secrets_from_file
    plugin.analyze(
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/plugins/base.py", line 131, in analyze
    file_lines = tuple(file.readlines())
                       ^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 715, in readlines
  File "<frozen codecs>", line 621, in readlines
  File "<frozen codecs>", line 507, in read
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 16: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/binaryornot/helpers.py", line 103, in is_binary_string
    bytes_to_check.decode(encoding=detected_encoding['encoding'])
TypeError: decode() argument 'encoding' must be str, not None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/detect-secrets", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/main.py", line 57, in main
    baseline_dict = _perform_scan(
                    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/main.py", line 181, in _perform_scan
    new_baseline = baseline.initialize(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/core/baseline.py", line 96, in initialize
    output.scan_file(file, suppress_unscannable_file_warnings)
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/core/secrets_collection.py", line 260, in scan_file
    self._extract_secrets_from_file(
  File "/usr/local/lib/python3.12/site-packages/detect_secrets/core/secrets_collection.py", line 419, in _extract_secrets_from_file
    file_is_binary = is_binary(filename)
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/binaryornot/check.py", line 33, in is_binary
    return is_binary_string(chunk)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/binaryornot/helpers.py", line 106, in is_binary_string
    unicode(bytes_to_check, encoding=detected_encoding['encoding'])  # noqa
    ^^^^^^^
NameError: name 'unicode' is not defined

[briangleeson]

@WilliamHolmes Not sure if this will help, but I just got the same unicode error today while migrating our Travis build to Github Actions, and fixed it by ignoring a binary file from the scan:

detect-secrets scan --update .secrets.baseline --exclude-files 'build/signing\.key\.enc'

Further up in the error was the clue, but it didnt mention which file caused the problem:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 16: invalid continuation byte

[osenbach]

PR: Handle unicode exception while logging details #184

For my group, the exception is always while processing a binary file, but the additional logging will help quickly pinpoint.

[osenbach]

Analysis below. PR won't hurt to more quickly identify the file causing the error / exception, but the actual is related to managing the chardet dependency

Root Cause Identified: chardet Version Incompatibility

Key Difference Between Passing and Failing Builds

Passing Build (Line 91, 107):

• chardet-6.0.0.post1 ✅

Failing Build (Line 123, 139):

• chardet-7.0.1 ❌

The Actual Error

The failing build shows the complete error chain (lines 182-229):

1. Primary Error: UnicodeDecodeError when reading a file with byte 0xfe
2. Secondary Error in binaryornot: TypeError: decode() argument 'encoding' must be str, not None (line 185)
3. Tertiary Error: NameError: name 'unicode' is not defined (line 229)

The final error at line 229 reveals that binaryornot v0.4.4 is using the Python 2 unicode() function, which doesn't exist in Python 3.13.

Why It's Happening

The chardet library version 7.0.1 is returning None for the encoding detection, which causes:

1. binaryornot to fail when trying to decode with encoding=None
2. The fallback code path in binaryornot tries to use unicode() (Python 2 syntax)
3. Python 3.13 doesn't have unicode() built-in, causing a NameError

Solution

Pin chardet to version 6.x in your detect-secrets installation to avoid the breaking changes in chardet 7.x.

Add to your GitHub Actions workflow or requirements file:

- name: Install detect-secrets
  run: |
    pip install 'chardet<7.0.0'
    pip install 'detect-secrets[word_list] @ git+https://github.com/IBM/detect-secrets.git@master'

Or update the detect-secrets setup.py or pyproject.toml to constrain chardet:

'chardet>=3.0.2,<7.0.0'  # instead of 'chardet>=3.0.2'

Why the Passing Build Worked

The passing build happened to get chardet-6.0.0.post1 from the package cache, which is compatible with binaryornot v0.4.4. The failing build got the newer chardet-7.0.1, which has breaking changes.

Long-term Fix

The binaryornot library (v0.4.4, last updated 2019) needs to be updated or replaced with a more modern alternative that's Python 3.13 compatible.

[Wesztman]

We started to see the same errors in our pipelines today, will try to pin chardet until a fix has been merged.

Edit:

For anyone using detect-secrets through pre-commit, you can add the chardet version restrain as an additional_dependencies, which seems to work just fine.

[WilliamHolmes]

I retested today, and I think the #184 has fixed my issue. Thanks @osenbach