Architecture

BG upgrades is using the django framework at its core.

The overall architecture should be pretty standard with one central form generated from tuckbox/templates/pattern_form.html.j2

It uses Redis and Celery to process the requests asynchronously (and report on progress).

The server uses uWSGI and Nginx the same way it is documented here

Requirements

• Python 3 (possibly also python3.X-dev on ubuntu)
• imagemagick
• nginx (for full deployment)
• redis
• chromedriver

Development

To setup an environment, use virtualenv and install the packages listed in requirements.txt

    $ virtualenv [--python=/your/path/to/python3] venv
    $ source venv/bin/activate
    $ pip install -r requirements.txt
    $ webdrivermanager chrome --linkpath [folder in your PATH]

To run the tests:

    $ ./manage.py test --parallel

To get coverage report on the tests:

    $ coverage run --source='.' --omit='venv/*' manage.py test --parallel
    $ coverage report

or

    $ coverage html

Test server

    $ redis-server
    $ watchmedo auto-restart --directory=./ --pattern="*.py" --recursive -- celery -A bg_upgrades worker -l info

then use Django's test server

    $ ./manage.py migrate && ./manage.py runserver 0.0.0.0:8000

or through uwsgi

    $ uwsgi --http :8000 --module django_app/bg_upgrades.wsgi

Generate a SSH certificate for local SSH testing:

     openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout localhost.key -out localhost.crt -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"

Startup / Deploy

To deploy, you need to setup those environment variables:

• DJANGO_DEBUG to 'False'
• DJANGO_SECRET_KEY. This can be generated:

    $ python -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())'

• DJANGO_PROJECT_PATH to the folder with the project

Configuration for nginx should be updated with the right paths and placed in the right folder

• /usr/local/etc/nginx/servers/ on mac
• /etc/nginx/servers/ on linux

Start all those different services:

    $ redis-server
    $ celery -A bg_upgrades worker -l info

    $ ./manage.py migrate
    $ ./manage.py collectstatic

    $ uwsgi --ini uwsgi.ini

Start the nginx server

    $ brew services restart nginx

    $ sudo /etc/init.d/nginx restart

Optionally:

    $ flower -A bg_upgrades

The overall deployment framework is using nginx and uWSGI as documented (here)[https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html].

SSL / TLS

Set the environment variables: DOMAIN to bg-upgrades.net and WILDCARD to *.$DOMAIN

    $ sudo add-apt-repository ppa:certbot/certbot
    $ sudo apt install python-certbot-nginx
    $ sudo certbot -d $DOMAIN -d $WILDCARD --nginx --preferred-challenges dns certonly

Instructions also (here)[https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/]

Maintenance

To find packages that need to be updated, you can check this link or run $ pip list --outdated

Then update:

    $ pip install <package_name> --upgrade

Whenever the packages change, record the packages with versions:

    $ pip-chill > requirements.txt

When the reference data needs to change for the graphics tests, run:

    $ python -m tuckbox.test.tests_box

If running into permissions issues for using 'PDF' in ImageMagick, follow those instructions

SSL certificate

Added

0 3 */3 * * cd ~ubuntu/bg_upgrades && perl -e 'sleep int(rand(43200))' && docker compose run --rm certbot renew && docker compose exec nginx nginx -s reload

to /etc/cron.d/certbot/ so that every 3 days at 3am it will check to renew the SSL certificate with certbot

The certificates are being renewed by the certbot images that runs a temporary container. The new certificates are stored in ./certbot/conf/live/www.bg-upgrades.net/ in the home folder of the application. The same folder is shared with the nginx container.

Docker

Start with docker-compose up -d (docker desktop needs to be running).

To test locally, see the procedure to create an SSL certificate locally. Or use the nginx configuration file that does not use SSL.

To run in production, change docker-compose to use Dockerfile.prod for the nginx service (the main change is to point to local SSL certificates) and the django environment variables (change the secret on the real site)

Test server

The test server is configure to run on a host called mysite.test. It is easy to add an entry in /etc/hosts or C:\windows\system32\drivers\etc\hosts on Windows with

127.0.0.1 mysite.test

to redirect queries to that host to localhost. Of course, nothing should be running using port 80.

Dependencies

The libraries this web application uses:

• Django
• Redis
• Celery
• ImageMagick
• Wand
• Bootstrap CSS
• jQuery
• ThreeJS
• Vanilla picker (color picker)
• Name That Color