badly encoded character in article name causes sync_ftp_files to abort

[JuliusR]

I have experienced a BNN file with a special character in the name of one article. The "bad" article causes the following error:

Mysql2::Error: Incorrect string value: '\xE2\x95\x9Aoro...'
for column `sharedlists`.`articles`.`name` at row 1:
INSERT INTO `articles` (
  `category`, 
  `created_on`, 
  `deposit`, 
  `manufacturer`, 
  `name`, 
  `note`, 
  `number`, 
  `origin`, 
  `price`, 
  `scale_price`, 
  `scale_quantity`, 
  `supplier_id`, 
  `tax`, 
  `unit`, 
  `unit_quantity`, 
  `updated_on`
) VALUES ([...], 'Pomi d╚oro [...]', [...])

The ╚ character looks wrong, but I also see that character when opening the original BNN file using CP 850 encoding. I believe it is supposed to read ' or ` or ’.

The problem is that this one bad file causes the whole sync_ftp_files job to abort.

[1resu]

The cause of problem at the foodcoops.net deployment was a non-utf8 collation of sharedlists' database. After converting the articles table to utf8 (ALTER TABLE articles CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;) the sync job runs without problems.

I'm not sure if this is a problem thats goes beyond our deployment as the default database collation is already utf8.

[JuliusR]

I'm not sure if this is a problem thats goes beyond our deployment as the default database collation is already utf8.

I can reproduce the problem locally, so it is not restricted to the foodcoops.net deployment. It is good to have a manual fix. 👍