Skip to content

feat(*): add Dockerfile and docker-compose for self-hosting#16

Open
jcreek wants to merge 10 commits into
Project-Counter:mainfrom
jcreek:feat/dockerfile-and-compose
Open

feat(*): add Dockerfile and docker-compose for self-hosting#16
jcreek wants to merge 10 commits into
Project-Counter:mainfrom
jcreek:feat/dockerfile-and-compose

Conversation

@jcreek

@jcreek jcreek commented May 12, 2026

Copy link
Copy Markdown

Adds a multi-stage Dockerfile (web/worker/frontend targets), a docker-compose.yml that brings up the full stack (web, worker, redis, postgres, c5tools, nginx frontend), an nginx config that serves the Vue SPA and proxies the Django API, an extended .env.example, and a rewritten "Using Docker" section in docs/own-instance.rst.

Resolves the "currently in preparation" note in docs/own-instance.rst. c5tools is pinned to v0.8.3 via build.context git URL - no second submodule needed.

Verified locally end to end: compose up -> validator-web entrypoint runs migrate + collectstatic + optional superuser creation from DJANGO_SUPERUSER_* env -> gunicorn. counter-api-validation POST -> worker -> c5tools -> messages all reachable via http://localhost (nginx -> validator-web) and direct on :8000.

Adds a multi-stage Dockerfile (web/worker/frontend targets), a docker-compose.yml that brings up the full stack (web, worker, redis, postgres, c5tools, nginx frontend), an nginx config that serves the Vue SPA and proxies the Django API, an extended .env.example, and a rewritten "Using Docker" section in docs/own-instance.rst.

  Resolves the "currently in preparation" note in docs/own-instance.rst.
  c5tools is pinned to v0.8.3 via build.context git URL — no second
  submodule needed.

  Verified locally end to end: compose up -> validator-web entrypoint
  runs migrate + collectstatic + optional superuser creation from
  DJANGO_SUPERUSER_* env -> gunicorn. counter-api-validation POST ->
  worker -> c5tools -> messages all reachable via http://localhost
  (nginx -> validator-web) and direct on :8000.
@beda42

beda42 commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Hi @jcreek , thanks for the PR! At first glance, it looks like a solid piece of work 👍

I currently have a lot on my plate, so it will take me some time to review and test it thoroughly. I will get back to you sometime next week.

Anyway, thanks again for the work you put in.

@jcreek

jcreek commented May 15, 2026

Copy link
Copy Markdown
Author

Not a problem at all @beda42 🙂

@beda42 beda42 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for the PR. I tested it and went over the changes. It works quite well. I added some suggestions for changes.

Also, when I tried validating a file in the running instance, I got the following error "[Errno 2] No such file or directory: '/app/media/file_validations/20260522-121200.694669-tLozTumA.json'". Unfortunately, I do not have more time to investigate the issue.

Please let me know if you are willing to incorporate the changes I suggested or if I should finish the PR myself. It is definitely a valuable addition, so I wouldn't like to loose it.

Comment thread .env.example Outdated
# VALIDATION_MODULES_URLS=http://c5tools:8080/

# ---------- User account behaviour ----------
ALLOW_USER_REGISTRATION=True

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to have this False by default. Without proper email setup, user registration would not work well and it seems unnecessary (and potentially dangerous) for simple local one-user setups.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed :)

Comment thread docs/own-instance.rst
it uses Docker Compose to make it easy to spin up all the necessary services.
The repository ships a `Dockerfile` and `docker-compose.yml` that bring up the
validator and all of its dependencies in a single command.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a note about the default setup being intended mostly for local experimentation and the necessity to read Production deployment for anything else. Just to make sure.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed :)

Comment thread Dockerfile Outdated
Comment thread docker-compose.yml Outdated
Comment thread docker-compose.yml Outdated
Comment thread .env.example Outdated
Comment thread docs/own-instance.rst Outdated
Comment thread .env.example Outdated
@jcreek

jcreek commented May 24, 2026

Copy link
Copy Markdown
Author

Thanks again for the PR. I tested it and went over the changes. It works quite well. I added some suggestions for changes.

Also, when I tried validating a file in the running instance, I got the following error "[Errno 2] No such file or directory: '/app/media/file_validations/20260522-121200.694669-tLozTumA.json'". Unfortunately, I do not have more time to investigate the issue.

Please let me know if you are willing to incorporate the changes I suggested or if I should finish the PR myself. It is definitely a valuable addition, so I wouldn't like to loose it.

I believe I've now addressed your comments. If you're able to share an example file that has that error while validating I can debug the issue.

@beda42

beda42 commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

I believe I've now addressed your comments. If you're able to share an example file that has that error while validating I can debug the issue.

Great, thanks for the changes you made.

As for the problem with validating files, I did some investigation and it stems from the fact that the media files are not shared between the web and the worker containers. When a file is uploaded, it is put into the web container, but the worker container expects to find it locally on the same path as was stored in the web container. And of course will not find it.

A similar situation in reverse occurs when you validate COUNTER API. The data from the request is stored by the worker container into its media directory, but is then not accessible on the web when you try to download the validated file.

I think that the best solution for this is to create a separate volume for media files and share it between those two containers. This would have the added benefit of persisting validations between container updates, etc.

This commit makes the self-hosting stack work out of the box by copying CHANGELOG.md into the image so the changelog endpoint no longer 500s, defaulting both web and worker to the console email backend so registration and notification emails work without SMTP, serving the Django admin under /django-admin/ so it no longer shadows the SPA's /admin/* routes, re-resolving validator-web via Docker DNS while forwarding the original Host:port so CSRF checks pass on any published port, and normalising line endings so the container entrypoint runs from Windows checkouts
@jcreek

jcreek commented Jun 30, 2026

Copy link
Copy Markdown
Author

@beda42 I've added a media_data volume shared across validator-web, validator-worker, and validator-frontend, so uploads written by one container are visible to the others. nginx now serves /media/ straight from that volume (rather than proxying to the backend), and as you noted it has the nice side effect of persisting validations across container updates.

Uploading a file for validation now works. For the reverse COUNTER API case you described, I confirmed directly that a file written by the worker into the shared volume is both visible to the web container and downloadable through nginx (/media/... → HTTP 200) so the download path that was previously broken now resolves. The only thing I couldn't exercise was a full live SUSHI fetch, since that needs real endpoint credentials, but the shared-volume mechanism that was failing is confirmed working both ways.

While testing the full stack I also ran into and fixed a few separate issues:

  • Changelog endpoint was returning a 500 - CHANGELOG.md wasn't being copied into the image, but apps/core/changelog.py reads it from BASE_DIR. This is now copied in the build.
  • Registration returned a 500 without an SMTP server (related to your earlier point about registration not working without email setup) - the stack now defaults both web and worker to Django's console email backend, so registration/verification/reset work out of the box and the emails just print to the container logs. Real delivery is still a config switch away (Mailgun, or EMAIL_BACKEND + EMAIL_HOST). I've kept ALLOW_USER_REGISTRATION=False as the default per your suggestion.
  • The SPA's /admin/users page was being shadowed by the Django admin mounted at /admin/ - it returned 404 on refresh/deep-link. I have moved the Django admin to /django-admin/ (via DJANGO_ADMIN_PATH) so the SPA's /admin/* routes work.

I've pushed all of the above. Happy to walk through any of it or split anything out if you'd prefer, and thanks again for the thorough review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants