UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2090: ordinal not in range(128)

[videlec]

quasar patchbot is running on python3 and experienced some encoding issues, e.g. on this report

========== doctest_continuation ==========
git checkout patchbot/ticket_merged
Already on 'patchbot/ticket_merged'
Traceback (most recent call last):
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/patchbot.py", line 1108, in test_a_ticket
    baseline=baseline, **kwds)
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 426, in doctest_continuation
    msg="Old-style doctest continuation", **kwds)
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 256, in exclude_new
    bad_lines += exclude_new_in_diff(gitdiff, r)
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 284, in exclude_new_in_diff
    for line in gitdiff:
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2090: ordinal not in range(128)
doctest_continuation -- 0 seconds

[fchapoton]

This must be dependant on the locale setting. I am running my patchbots in python3 for a very long time, and never got any problem. Of course, I did the work to make the patchbot "sort of py3 compatible" a long time ago, and it would certainly benefit from a thorough check by Erik or Jeroen..

[videlec]

Note that it works just fine most of the time. It just happens sometime (but I don't know the reason).

[jdemeyer]

There is also this patchbot report at https://patchbot.sagemath.org/log/24863/Ubuntu/16.04/x86_64/4.4.0-116-generic/quasar/2018-02-28%2011:03:12?plugin=non_ascii

========== non_ascii ==========
git checkout patchbot/ticket_merged
Already on 'patchbot/ticket_merged'
Traceback (most recent call last):
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/patchbot.py", line 1108, in test_a_ticket
    baseline=baseline, **kwds)
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 343, in non_ascii
    msg="Non-ascii characters", **kwds)
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 213, in exclude_new_file_by_file
    if file_condition(a_file):
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 339, in not_declared
    return (not(check_unicode_declaration(a_file)) and
  File "/usr/labri/vdelecro/.local/lib/python3.5/site-packages/sage_patchbot/plugins.py", line 322, in check_unicode_declaration
    L0 = regex.split(f.readline())
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3099: ordinal not in range(128)
non_ascii -- 0 seconds
========== end non_ascii ==========

[dkrenn]

A similar problem (however, different lines of code) occurs at https://patchbot.sagemath.org/log/Pending/21202/LinuxMint/19/x86_64/4.15.0-20-generic/pc72-math/2019-04-21%2010:29:34 :

git_rev_list -- 0 seconds
========== end git_rev_list ==========
Traceback (most recent call last):
  File "/local/sage-patchbot/.local/lib/python3.6/site-packages/sage_patchbot/patchbot.py", line 1395, in report_ticket
    report['git_log'] = subprocess.check_output(['git', 'log', '--oneline', '%s..patchbot/ticket_upstream' % ticket_base], universal_newlines=True).strip().split('\n')
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 405, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.6/subprocess.py", line 830, in communicate
    stdout = self.stdout.read()
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11955: ordinal not in range(128)

[dkrenn]

A similar problem (however, different lines of code) occurs at https://patchbot.sagemath.org/log/Pending/21202/LinuxMint/19/x86_64/4.15.0-20-generic/pc72-math/2019-04-21%2010:29:34 :

It seems that my issue can be fixed by calling the patchbot by

LANG=C.UTF-8 python3 -m sage_patchbot.patchbot