Docker suddenly stopped working

[Jesus82]

Hi guys, my Docker stopped working and I cannot find out the way to make it work again.

I'm currently working in the feature/trips branch. Last 2 days I commited some files with no issue at all. Today I started the container, but when going to the browser I get a 502 Bad Gateway error. I checked out to previous commits to see if there was any issue about the changes I had commited, but it doesn't work as well.

Tried to reinstall with no issues:

docker-compose down -v --remove-orphans
docker system prune -f
docker volume rm $(docker volume ls -q)
docker-compose build php

and
make install
But still getting 502

My docker logs from rox_web_1 are:

2021/02/02 11:41:52 [error] 6#6: *1 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.21.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://172.21.0.5:9000", host: "localhost:8080"
172.21.0.1 - - [02/Feb/2021:11:41:52 +0000] "GET / HTTP/1.1" 502 158 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:11:41:52 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:11:41:53 +0000] "GET /service-worker.js HTTP/1.1" 200 239295 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
2021/02/02 12:06:11 [error] 6#6: *3 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.21.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://172.21.0.5:9000", host: "localhost:8080"
172.21.0.1 - - [02/Feb/2021:12:06:11 +0000] "GET / HTTP/1.1" 502 158 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:12:06:12 +0000] "GET /service-worker.js HTTP/1.1" 200 239295 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
2021/02/02 12:08:48 [error] 6#6: *5 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.21.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://172.21.0.5:9000", host: "localhost:8080"
172.21.0.1 - - [02/Feb/2021:12:08:48 +0000] "GET / HTTP/1.1" 502 158 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:12:08:48 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:12:08:49 +0000] "GET /service-worker.js HTTP/1.1" 200 239295 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
2021/02/02 12:14:56 [error] 6#6: *8 connect() failed (113: Host is unreachable) while connecting to upstream, client: 172.21.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://172.21.0.5:9000", host: "localhost:8080"
172.21.0.1 - - [02/Feb/2021:12:14:56 +0000] "GET / HTTP/1.1" 502 158 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:12:14:56 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "http://localhost:8080/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"
172.21.0.1 - - [02/Feb/2021:12:14:57 +0000] "GET /service-worker.js HTTP/1.1" 200 239295 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0" "-"

Any hint will be appreciated. Cheers!

[vincentchalamon]

Looks like the php container is down. I've tried locally, it was due to the sensiolabs/security-checker which is now deprecated and returns a 403 on call. It has been replaced by the symfony binary tool. It is fixed in this project on this PR: #220

Removing the security-check script from composer.json fixes my issue and allows to start the project locally.

Can you share the logs of your php container to ensure this is the same issue? docker-compose logs php

[PLP-GTR]

Hi,

I'm new and can't find a solution here. Before I dive deeper, I wanted to ask about the issue below.

I'm on a fairly clean macOS 14.0 (Intel i9, no ARM yet) and I'm having trouble starting the docker containers.

Fresh git clone, $make install but the php container does not want to stay up (image: bewelcome/php).

The logs of the container are:

2023-11-21 22:24:18 setfacl: build: Not supported
2023-11-21 22:24:18 setfacl: build/forums: Not supported
2023-11-21 22:24:18 setfacl: build/forums/forums.model.php: Not supported
... [571 more lines]
2023-11-21 22:24:19 setfacl: build/members/member.entity.php: Not supported
2023-11-21 22:24:19 setfacl: upload: Not supported
2023-11-21 22:24:19 setfacl: upload/images: Not supported

The changes of #220 are in the code.

$ docker version:

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.25.2 (129061)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker compose version:

Docker Compose version v2.23.0-desktop.1

localhost:8080 -> 502 Bad Gateway | nginx/1.17.10

Thanks :)

[thisismeonmounteverest]

@PLP-GTR The docker setup is kind of outdated (not using Manticore yet). Not sure if my meager docker knowledge will allow me to fix that, but I give it a try.

[thisismeonmounteverest]

Okay, I did a few updates and can create a running docker container locally on Windows with linux subsystem now.

You will still see the setfacl warnings that's likely a problem on Mac OS. but the container should come up after a few minutes. The last line in the php image logs should say something along

rox-php-1 |NOTICE: fpm is running, pid 1
rox-php-1 |NOTICE: ready to handle connections

[PLP-GTR]

@thisismeonmounteverest thanks for looking into it! 🤗

Unfortunately docker still does not want to comply and the php container exists. I cleanly cloned the repo and removed all containers, images and volumes before trying again.

I've waited about 10 mins after docker compose finished its output but the php container did not come up.

If it helps, here's the output of docker compose up: https://app.warp.dev/block/j9328rJAOAdkxvY83EGgN8
And at pastebin with removed setfacl lines: https://pastebin.com/G6DwMV88

[thisismeonmounteverest]

Bringing the php container up takes a while... Did you run make install or docker-compose up -d?

For the second try without setfacl it looks like the php container was just reused, please try deleting it.

[PLP-GTR]

@thisismeonmounteverest oh I'm sorry - same thing happens with make install

The output: https://app.warp.dev/block/6xnAIvGiOIaJFsr86FIkQf

Currently waiting 12 mins, I'll leave it a little longer. But generally it looks like this after starting it with make install or docker compose up:

---

Update:

Half hour later no success.

[thisismeonmounteverest]

As the php-1 is already stopped waiting doesn't really help I assume.

What's the last log entry for php-1? (The warp listing doesn't show it closing down.)

[PLP-GTR]

@thisismeonmounteverest

As the php-1 is already stopped waiting doesn't really help I assume.

That's what I assumed but the "takes some minutes" was not 100% clear in this very case :-)

---

huh, this time web-1 and php-1 didn't come up

What's the last log entry for php-1? (The warp listing doesn't show it closing down.)

Nothing special, really ... https://pastebin.com/rCKVSz4u

Log of web-1:

2023-11-26 14:51:45 2023/11/26 13:51:45 [emerg] 1#1: host not found in upstream "php" in /etc/nginx/conf.d/default.conf:11
2023-11-26 14:51:45 nginx: [emerg] host not found in upstream "php" in /etc/nginx/conf.d/default.conf:11

Maybe I didn't reset everything fully but I'm pretty sure I did. But if I docker compose down and make install again, the web-1 container stays up (although 502 Bad Gateway without the php-1 container at localhost:8080)

---

Also my knowledge is limited here (I'm used to docker, but way simpler setups) and jumping into an unknown application is always kind of tricky imho. I try to help debug as best as I can.

[thisismeonmounteverest]

You really need to delete the container and images in the Docker app. Please try again with the setfacl entries removed. Use make install root=1 if you run into problems with permissions.

[PLP-GTR]

Hi,

thanks for your support! The page renders now :)

You really need to delete the container and images in the Docker app.

I always deleted the containers, images and volumes related to the project (except for some tests).
Also I reseted the repository completely with git clean -dfX && git reset --hard HEAD after each run.

Please try again with the setfacl entries removed.

I have commented out the following lines now
docker/php/docker-entrypoint.sh:17,18 and :40,41:

	setfacl -R -m u:www-data:rwX -m u:"$(whoami)":rwX var build data upload
	setfacl -dR -m u:www-data:rwX -m u:"$(whoami)":rwX var build data upload

			setfacl -R -m u:www-data:rX -m u:"$(whoami)":rwX config/jwt
			setfacl -dR -m u:www-data:rX -m u:"$(whoami)":rwX config/jwt

Use make install root=1 if you run into problems with permissions.

Started with make install (no root option).

The php-1 container now stays up (about 6 mins to start) and the logs look way better now:
https://pastebin.com/TMMkkw1k

Shorted version:

2023-12-04 21:28:17 Generating public / private keys for JWT
2023-12-04 21:28:19 yarn install v1.22.17
2023-12-04 21:28:19 [1/4] Resolving packages...
2023-12-04 21:28:21 [2/4] Fetching packages...
2023-12-04 21:28:23 [3/4] Linking dependencies...
2023-12-04 21:28:18 ...............................................................................................................................................................++++
2023-12-04 21:28:18 ............++++
2023-12-04 21:28:23 warning " > @symfony/webpack-encore@4.5.0" has unmet peer dependency "@babel/preset-env@^7.16.0".
2023-12-04 21:28:23 warning "swagger-cli > @apidevtools/swagger-cli > @apidevtools/swagger-parser@10.1.0" has unmet peer dependency "openapi-types@>=7".
2023-12-04 21:30:12 [4/4] Building fresh packages...
2023-12-04 21:30:15 Done in 116.33s.
2023-12-04 21:30:17 You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3.
2023-12-04 21:30:17 Installing dependencies from lock file (including require-dev)
2023-12-04 21:30:17 Verifying lock file contents can be installed on current platform.
2023-12-04 21:30:17 Package operations: 231 installs, 0 updates, 0 removals
2023-12-04 21:30:17   - Downloading infection/extension-installer (0.1.2)
2023-12-04 21:30:17   - Downloading symfony/flex (v1.21.2)
[...]
2023-12-04 21:30:18   - Downloading twig/intl-extra (v3.7.1)
2023-12-04 21:30:18   - Downloading twig/markdown-extra (v3.7.1)
2023-12-04 21:30:18   - Downloading xemlock/htmlpurifier-html5 (v0.1.11)
2023-12-04 21:30:38   - Installing infection/extension-installer (0.1.2): Extracting archive
2023-12-04 21:30:39   - Installing symfony/flex (v1.21.2): Extracting archive
[...]
2023-12-04 21:30:43   - Installing twig/markdown-extra (v3.7.1): Extracting archive
2023-12-04 21:30:43   - Installing xemlock/htmlpurifier-html5 (v0.1.11): Extracting archive
2023-12-04 21:31:34 Package league/uri-parser is abandoned, you should avoid using it. No replacement was suggested.
2023-12-04 21:31:34 Package sensio/framework-extra-bundle is abandoned, you should avoid using it. Use Symfony instead.
2023-12-04 21:31:34 Package swiftmailer/swiftmailer is abandoned, you should avoid using it. Use symfony/mailer instead.
2023-12-04 21:31:34 Package behatch/contexts is abandoned, you should avoid using it. No replacement was suggested.
2023-12-04 21:31:34 Package php-cs-fixer/diff is abandoned, you should avoid using it. No replacement was suggested.
2023-12-04 21:31:34 Package sebastian/phpcpd is abandoned, you should avoid using it. No replacement was suggested.
2023-12-04 21:31:34 Package webmozart/path-util is abandoned, you should avoid using it. Use symfony/filesystem instead.
2023-12-04 21:31:34 Generating autoload files
2023-12-04 21:31:36 162 packages you are using are looking for funding.
2023-12-04 21:31:36 Use the `composer fund` command to find out more!
2023-12-04 21:31:36 Run composer recipes at any time to see the status of your Symfony recipes.
2023-12-04 21:31:36 infection/extension-installer: No extensions found
2023-12-04 21:31:36 phpstan/extension-installer: Extensions installed
2023-12-04 21:31:36 > nesbot/carbon: installed
2023-12-04 21:31:36 > phpstan/phpstan-doctrine: installed
2023-12-04 21:31:36 > phpstan/phpstan-symfony: installed
2023-12-04 21:31:36 Waiting for db to be ready...
2023-12-04 21:31:50 Creating test database
2023-12-04 21:31:50 ======================
2023-12-04 21:31:50 Dropping the database
2023-12-04 21:31:50 Creating the database
2023-12-04 21:31:50 Created database `bewelcome` for connection named default
2023-12-04 21:31:50 Creating the schema
2023-12-04 21:31:51  ! [CAUTION] This operation should not be executed in a production environment! 
2023-12-04 21:31:51  Creating database schema...
2023-12-04 21:31:56 Adding stored functions
2023-12-04 21:31:56  [OK] Database schema created successfully!                                     
2023-12-04 21:31:56                     Application Migrations                    
2023-12-04 21:31:56 Migrating up to 20200919061813 from 0
[...]
2023-12-04 21:31:56   ++ migrated (took 55.2ms, used 44.5M memory)
2023-12-04 21:31:56   ------------------------
2023-12-04 21:31:56   ++ finished in 211ms
2023-12-04 21:31:56   ++ used 44.5M memory
2023-12-04 21:31:56   ++ 4 migrations executed
2023-12-04 21:31:56   ++ 8 sql queries
2023-12-04 21:31:56 Seeding the database
2023-12-04 21:31:56 
2023-12-04 21:31:57 Importing missing translations
[...]
2023-12-04 21:31:58  [OK] Finished importing the translations into the database                     
2023-12-04 21:31:58  ! [NOTE] clearing cache to force update of translations                        
2023-12-04 21:32:20 Finished have fun.
2023-12-04 21:32:40 infection/extension-installer: No extensions found
2023-12-04 21:32:40 phpstan/extension-installer: Extensions installed
2023-12-04 21:32:40 Run composer recipes at any time to see the status of your Symfony recipes.
2023-12-04 21:33:05 Executing script cache:clear [OK]
2023-12-04 21:33:08 Executing script assets:install public [OK]
2023-12-04 21:33:08 yarn run v1.22.17
2023-12-04 21:33:08 $ /srv/bewelcome/node_modules/.bin/encore dev --mode=development
2023-12-04 21:33:09 Running webpack ...
2023-12-04 21:34:34 webpack compiled with 10 warnings
2023-12-04 21:34:34 asset ./service-worker.js 330 KiB [emitted] 1 related asset
2023-12-04 21:34:34 asset main.js 249 bytes [emitted] (name: main) 1 related asset
2023-12-04 21:34:34 ./assets/js/index.js 26 bytes [built] [code generated]
2023-12-04 21:34:34 LOG from InjectManifest
2023-12-04 21:34:34 <i> The service worker at ./service-worker.js will precache
2023-12-04 21:34:34 <i>         1 URLs, totaling 249 B.
2023-12-04 21:34:34 webpack 5.89.0 compiled successfully in 18275 ms
2023-12-04 21:34:34 Done in 85.47s.
2023-12-04 21:34:34 [04-Dec-2023 20:34:34] NOTICE: fpm is running, pid 1
2023-12-04 21:34:34 [04-Dec-2023 20:34:34] NOTICE: ready to handle connections

It renders nicely at localhost:8080

Login as member-2 also works.

[PLP-GTR]

Calling the profile page lead to
Data subdir "/srv/bewelcome/public/../data/user/avatars" is not writable!

No matter if the make install command was called with root=1 or not.

Fix for me:
$ docker-compose exec php sh -c "chown -R root:root ./data"

But I'm getting a PException when I visit my profile page or the edit page at http://localhost:8080/editmyprofile

I debugged it a bit. Maybe it's not related to the docker setup, so I don't know if offtopic.

The SQL query leading to the error:

SELECT
    specialrelations.Id AS id,
    specialrelations.IdRelation AS IdRelation,
    members.Username,
    specialrelations.Type AS Type,
    specialrelations.Comment AS Comment,
    specialrelations.Confirmed AS Confirmed
FROM
    specialrelations,
    members
WHERE
    specialrelations.IdOwner = 1  AND
    specialrelations.IdRelation = members.Id AND
    members.Status in ('Active', 'ActiveHidden', 'ChoiceInactive', 'OutOfRemind', 'Pending')
ORDER BY members.Username ASC

Unfortunately the error is catched and not given to frontend. I've connected to the DB and executed it manually, leading to the error:

Unknown column 'specialrelations.Type' in 'field list'

The column is missing, I've checked the migrations and can't find it there. Only occurrence of specialrelations is the modernize table comments1 migration (link):

Did I miss some migration command?

Re-applying the test database did not fix it:
$ docker-compose exec php sh -c "php bin/console test:database:create --drop --force"