remove database backend #489 by joshdimanteto · Pull Request #507 · ral-facilities/datagateway-api

joshdimanteto · 2024-11-26T15:50:08Z

This PR will close #489

Description

Remove the database backend functionality from the DataGateway API. It includes significant changes such as deleting database-related files and functionality, updating tests to remove database dependencies, and modifying the configuration to align with the changes.

Testing Instructions

Add a set up instructions describing how the reviewer should test the code

Review code
Check GitHub Actions build
If icatdb Generator Script Consistency Test CI job fails, is this because of a deliberate change made to the script to change generated data (which isn't actually a problem) or is here an underlying issue with the changes made?
Review changes to test coverage
Does this change mean a new patch, minor or major version should be made? If so, does one of the commit messages feature fix:, feat: or BREAKING CHANGE: so a release is automatically made via GitHub Actions upon merge?
{more steps here}

Agile Board Tracking

Connect to #489

notes

The icat models still use SQLAlchemy, this is going to be update in the pydantic V2 upgrade to use pydantic

- remove downgrade of setuptools

Added `# noqa: B907` in several places where manual quoting is required for valid SQL query syntax. Using `!r` would change the format and break query parsing, so explicit quotes are necessary. This ensures tests and API calls remain correct while satisfying linting requirements.

- In the test_query_filter_factory the test don't work on the CI as it is not setting up the db backend.

…-backend-#489

joshdimanteto · 2025-11-17T15:32:34Z

.github/workflows/ci-build.yml

+      - name: Setup Java
+        uses: actions/setup-java@b36c23c0d998641eff861008f374ee103c25ac73 # v4.4.0
+        with:
+          distribution: 'temurin'
+          java-version: 8
+
+       # ICAT Ansible clone and install dependencies
+      - name: Checkout icat-ansible
+        if: success()
+        uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4.2.1
+        with:
+          repository: icatproject-contrib/icat-ansible
+          ref: master
+          path: icat-ansible
+      - name: Install Ansible
+        run: pip install -r icat-ansible/requirements.txt
+     # Prep for running the playbook
+      - name: Create hosts file
+        run: echo -e "[icatdb_minimal_hosts]\nlocalhost ansible_connection=local" > icat-ansible/hosts
+      - name: Prepare vault pass
+        run: echo -e "icattravispw" > icat-ansible/vault_pass.txt
+      - name: Move vault to directory it'll get detected by Ansible
+        run: mv icat-ansible/vault.yml icat-ansible/group_vars/all
+      - name: Replace default payara user with Actions user
+        run: |
+          sed -i -e "s/^payara_user: \"glassfish\"/payara_user: \"runner\"/" icat-ansible/group_vars/all/vars.yml
+
+      # Force hostname to localhost - bug fix for previous ICAT Ansible issues on Actions
+      - name: Change hostname to localhost
+        run: sudo hostname -b localhost
+
+      # Remove existing MySQL installation so it doesn't interfere with GitHub Actions
+      - name: Remove existing mysql
+        run: |
+          sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+          sudo apt-get remove --purge "mysql*"
+          sudo rm -rf /var/lib/mysql* /etc/mysql
+
+      # Create local instance of ICAT
+      - name: Run ICAT Ansible Playbook
+        run: |
+          ansible-playbook icat-ansible/icatdb_minimal_hosts.yml -i icat-ansible/hosts --vault-password-file icat-ansible/vault_pass.txt -vv
+
+      # rootUserNames needs editing as anon/anon is used in search API and required to pass endpoint tests
+      - name: Add anon user to rootUserNames
+        run: |
+          awk -F" =" '/rootUserNames/{$2="= simple/root anon/anon";print;next}1' /home/runner/install/icat.server/run.properties > /home/runner/install/icat.server/run.properties.tmp
+      - name: Apply rootUserNames change
+        run: |
+          mv -f /home/runner/install/icat.server/run.properties.tmp /home/runner/install/icat.server/run.properties
+      - name: Reinstall ICAT Server
+        run: |
+          cd /home/runner/install/icat.server/ && ./setup -vv install


check if this is still needed

it is, as it needs an ICAT server

Just wondering why it was working before without any of this but now it needs all the ICAT stuff?

This work before as it was using the db backend instead of the python icat backend so it didn't need the extra setup

test/integration/conftest.py

joshdimanteto · 2026-02-09T10:08:46Z

test/integration/conftest.py

@@ -45,11 +42,10 @@ def flask_test_app():
 def flask_test_app_db():


This name should change. I have changed in #524 where i had to refactored the tests

VKTB · 2026-02-10T12:27:38Z

.github/workflows/ci-build.yml

+      - name: Setup Java
+        uses: actions/setup-java@b36c23c0d998641eff861008f374ee103c25ac73 # v4.4.0
+        with:
+          distribution: 'temurin'
+          java-version: 8
+
+       # ICAT Ansible clone and install dependencies
+      - name: Checkout icat-ansible
+        if: success()
+        uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4.2.1
+        with:
+          repository: icatproject-contrib/icat-ansible
+          ref: master
+          path: icat-ansible
+      - name: Install Ansible
+        run: pip install -r icat-ansible/requirements.txt
+     # Prep for running the playbook
+      - name: Create hosts file
+        run: echo -e "[icatdb_minimal_hosts]\nlocalhost ansible_connection=local" > icat-ansible/hosts
+      - name: Prepare vault pass
+        run: echo -e "icattravispw" > icat-ansible/vault_pass.txt
+      - name: Move vault to directory it'll get detected by Ansible
+        run: mv icat-ansible/vault.yml icat-ansible/group_vars/all
+      - name: Replace default payara user with Actions user
+        run: |
+          sed -i -e "s/^payara_user: \"glassfish\"/payara_user: \"runner\"/" icat-ansible/group_vars/all/vars.yml
+
+      # Force hostname to localhost - bug fix for previous ICAT Ansible issues on Actions
+      - name: Change hostname to localhost
+        run: sudo hostname -b localhost
+
+      # Remove existing MySQL installation so it doesn't interfere with GitHub Actions
+      - name: Remove existing mysql
+        run: |
+          sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+          sudo apt-get remove --purge "mysql*"
+          sudo rm -rf /var/lib/mysql* /etc/mysql
+
+      # Create local instance of ICAT
+      - name: Run ICAT Ansible Playbook
+        run: |
+          ansible-playbook icat-ansible/icatdb_minimal_hosts.yml -i icat-ansible/hosts --vault-password-file icat-ansible/vault_pass.txt -vv
+
+      # rootUserNames needs editing as anon/anon is used in search API and required to pass endpoint tests
+      - name: Add anon user to rootUserNames
+        run: |
+          awk -F" =" '/rootUserNames/{$2="= simple/root anon/anon";print;next}1' /home/runner/install/icat.server/run.properties > /home/runner/install/icat.server/run.properties.tmp
+      - name: Apply rootUserNames change
+        run: |
+          mv -f /home/runner/install/icat.server/run.properties.tmp /home/runner/install/icat.server/run.properties
+      - name: Reinstall ICAT Server
+        run: |
+          cd /home/runner/install/icat.server/ && ./setup -vv install


Just wondering why it was working before without any of this but now it needs all the ICAT stuff?

datagateway_api/src/common/config.py

joshdimanteto changed the title ~~refactor: remove database backend #469~~ remove database backend #469 Nov 26, 2024

joshdimanteto changed the title ~~remove database backend #469~~ remove database backend #489 Nov 26, 2024

refactor: remove database backend #489

743b34b

joshdimanteto force-pushed the remove-database-backend-#489 branch from 5e5d6c4 to 743b34b Compare November 26, 2024 16:57

fix intergration tests #489

cf349cf

joshdimanteto force-pushed the remove-database-backend-#489 branch 4 times, most recently from 86a347f to c3a28c4 Compare December 2, 2024 13:56

set up icat stack before pip install test #489

2939010

joshdimanteto force-pushed the remove-database-backend-#489 branch from c3a28c4 to 2939010 Compare December 2, 2024 14:17

joshdimanteto added 2 commits December 2, 2024 14:47

ignore vulnerability for werkzeug and cryptography #489

27f8ff9

remove remaining remnants of the backends #489

34d2603

joshdimanteto force-pushed the remove-database-backend-#489 branch from e1afd0a to 34d2603 Compare January 27, 2025 14:02

joshdimanteto added 16 commits September 10, 2025 11:30

update python to use Python 3.9.12 for Ubuntu 24.04 compatibility

482231d

Update lock file

d98b735

update lock file

b1ac7e2

pin python version to 3.10

d841fee

mend

46862fd

upgrade the black verison

b2be8df

reformat code using latest version black #517

2ef73ff

fix linting errors #517

fe50126

upgrade faker #517

c449888

- remove downgrade of setuptools

upgrade the version of poetry to 1.8.0 #517

ac6a5bb

remove install with constraints #517

2433b06

revert faker upgrade #517

53b4507

revert icat_db_generator changes #517

8eb1a43

run black formatter for icat_db_generator.py #517

f3ed2fb

fix linting in icat_db_generator.py #517

137e5de

joshdimanteto added 4 commits October 29, 2025 16:22

change strict=True to False for zip functions #517

c4c9526

Change fixture for db to "function" from "package"

90c8474

- In the test_query_filter_factory the test don't work on the CI as it is not setting up the db backend.

reset ICAT client before each test to avoid shared state issues #517

ec4350a

silence flake8 B909 warnings in query filter factory test #517

04a2567

joshdimanteto changed the base branch from main to remove-older-python-versions-#517 November 4, 2025 16:12

Merge branch 'remove-older-python-versions-#517' into remove-database…

6e5e7f3

…-backend-#489

joshdimanteto force-pushed the remove-older-python-versions-#517 branch from 04a2567 to 8673a50 Compare November 5, 2025 16:39

Merge branch 'remove-older-python-versions-#517' into remove-database…

e280471

…-backend-#489

joshdimanteto force-pushed the remove-database-backend-#489 branch from 94aad25 to e280471 Compare November 17, 2025 12:15

Merge branch 'remove-older-python-versions-#517' into remove-database…

84b9f59

…-backend-#489

joshdimanteto commented Nov 17, 2025

View reviewed changes

joshdimanteto force-pushed the remove-database-backend-#489 branch from 9ac04a6 to 84b9f59 Compare November 17, 2025 15:50

remove unused sql imports #489

5be200a

joshdimanteto marked this pull request as ready for review January 14, 2026 15:57

joshdimanteto requested a review from VKTB January 15, 2026 15:38

joshdimanteto commented Feb 9, 2026

View reviewed changes

test/integration/conftest.py Outdated Show resolved Hide resolved

joshdimanteto commented Feb 9, 2026

View reviewed changes

VKTB reviewed Feb 10, 2026

View reviewed changes

joshdimanteto mentioned this pull request Feb 12, 2026

Use pydantic settings to manage config #526

Open

address review comments #489

9fae76f

joshdimanteto force-pushed the remove-database-backend-#489 branch from a68b552 to 9fae76f Compare February 12, 2026 10:27

joshdimanteto requested a review from VKTB February 12, 2026 10:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove database backend #489#507

remove database backend #489#507
joshdimanteto wants to merge 30 commits intoremove-older-python-versions-#517from
remove-database-backend-#489

joshdimanteto commented Nov 26, 2024 •

edited

Loading

Uh oh!

joshdimanteto Nov 17, 2025

Uh oh!

joshdimanteto Nov 17, 2025

Uh oh!

VKTB Feb 10, 2026

Uh oh!

joshdimanteto Feb 12, 2026

Uh oh!

Uh oh!

joshdimanteto Feb 9, 2026

Uh oh!

VKTB Feb 10, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -45,11 +42,10 @@ def flask_test_app():
		def flask_test_app_db():

Conversation

joshdimanteto commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing Instructions

Agile Board Tracking

notes

Uh oh!

joshdimanteto Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

joshdimanteto Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

VKTB Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

joshdimanteto Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshdimanteto Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

VKTB Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshdimanteto commented Nov 26, 2024 •

edited

Loading