Galaxy server clean-up script along with some changes to galaxykit to support it by ironfroggy · Pull Request #95 · ansible/galaxykit

ironfroggy · 2023-06-27T16:07:20Z

Changes to galaxykit include:

The collection list command gains repository, limit, offset, keywords parameters
- repository: Can specify multiple repositories to list from. Default: "published"
- `limit': How many collections to return at maximum.
- offset: How many collections to skip, to allow paging.
- keywords: Filter results with a keyword search.
Added a new flag to all http methods: include_response. Causes to return both the parsed JSON and the original HttpResponse object.
get() can take a keyword argument "params" to pass querystring parameters easily.
delete_collection() has many changes to support the cleanup script:
- Takes a repositories parameter instead of a repository parameter.
- Takes a boolean dependencies parameter to recursively delete dependencies of the collection before deleting the collection itself.

Finally, of course, added the actual clean up script in scripts/galaxy-cleaner.sh.

galaxy-cleaner.sh
script usage: ./cleanup.sh [-l LIMIT] [-b BATCH_SIZE] [-d]

    -l          Limit. Number of collections to remove.
    -b          Batch size. Number of collections to fetch in each batch
                to then delete. Adjust this for performance and/or rate limit
                problems.
    -d          Debug flag. Enables bash -xv flags and displays extra debug
                information while running.
    -t          Token file. Path to a file to read the access token from.
    -T          Token. Directly pass the access token as a parameter.
    -p          Proxy. Set the HTTP_PROXY value to access the Galaxy API via.
    -w          Wait. Pause between steps to reduce server impact or avoid rate
                limiting.galaxy-cleaner.sh

…the cleanup script.

github-actions · 2023-06-27T16:07:54Z

galaxykit/client.py

+        include_response = kwargs.pop("include_response", False)
+
+        if include_response and not parse_json:
+            raise ValueError("GalaxyClient._http() called with include_response=True only valid when parse_json=True!")


[black] _{reported by reviewdog 🐶}

Suggested change

raise ValueError("GalaxyClient._http() called with include_response=True only valid when parse_json=True!")

raise ValueError(

"GalaxyClient._http() called with include_response=True only valid when parse_json=True!"

)

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py

-def get_collection_list(client):
-    url = "_ui/v1/collection-versions/?limit=999999"
-    return client.get(url)
+def get_collection_list(client, repo="published", limit=None, offset=None, keywords=None):


[black] _{reported by reviewdog 🐶}

Suggested change

def get_collection_list(client, repo="published", limit=None, offset=None, keywords=None):

def get_collection_list(

client, repo="published", limit=None, offset=None, keywords=None

):

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py


 def delete_collection(
-    client, namespace, collection, version=None, repository="published"
+    client, namespace, collection, version=None, repositories=("published",), dependents=False


[black] _{reported by reviewdog 🐶}

Suggested change

client, namespace, collection, version=None, repositories=("published",), dependents=False

client,

namespace,

collection,

version=None,

repositories=("published",),

dependents=False,

github-actions · 2023-06-27T16:07:54Z

galaxykit/collections.py

+            resp = client.delete(delete_url)
+        except GalaxyClientError as e:
+            if e.response.status_code == 404:
+                logger.debug(f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr)


[black] _{reported by reviewdog 🐶}

Suggested change

logger.debug(f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr)

logger.debug(

f"Ignoring (maybe) already deleted {coll_name}", file=sys.stderr

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+                    combined = {
+                        "original": None,
+                        "dependents": []
+                    }


[black] _{reported by reviewdog 🐶}

Suggested change

combined = {

"original": None,

"dependents": []

}

combined = {"original": None, "dependents": []}

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+                    for dep in resp["dependent_collection_versions"]:
+                        dep_coll, dep_ver = dep.split(" ")
+                        dep_ns, dep_name = dep_coll.split(".")
+                        dep_resp = delete_collection(client, dep_ns, dep_name, dep_ver, repositories, dependents=True)


[black] _{reported by reviewdog 🐶}

Suggested change

dep_resp = delete_collection(client, dep_ns, dep_name, dep_ver, repositories, dependents=True)

dep_resp = delete_collection(

client,

dep_ns,

dep_name,

dep_ver,

repositories,

dependents=True,

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/collections.py

+        final_resp["responses"].append({
+            "collection": coll_name,
+            "response": resp,
+        })


[black] _{reported by reviewdog 🐶}

Suggested change

final_resp["responses"].append({

"collection": coll_name,

"response": resp,

})

final_resp["responses"].append(

{

"collection": coll_name,

"response": resp,

}

)

github-actions · 2023-06-27T16:07:55Z

galaxykit/command.py

 def parse_args(parser, args):
    for arg in args:
-        parser.add_argument(arg, **(args[arg]))
+        flag_args=[]


[black] _{reported by reviewdog 🐶}

Suggested change

flag_args=[]

flag_args = []

github-actions · 2023-06-27T16:07:55Z

galaxykit/command.py

+            flag_args.append("--" + args[arg].pop("long").strip("-"))
+        if not flag_args:
+            flag_args.append(arg)
+


[black] _{reported by reviewdog 🐶}

Suggested change

github-actions · 2023-06-27T16:07:56Z

galaxykit/command.py

                    args.version,
-                    args.repository or "published",
+                    args.repository.split(","),
+                    args.dependents


[black] _{reported by reviewdog 🐶}

Suggested change

args.dependents

args.dependents,

himdel · 2023-07-21T02:17:14Z

This is causing a lot of failures in UI tests (https://github.com/ansible/ansible-hub-ui/actions/runs/5617442384/job/15221548810?pr=4013#step:30:464)

but looks like the primary cause is the change to galaxykit collection list:

changes data from list of latest collection versions to a list of collections with a latest_version
no longer includes repository_list
no longer includes collections from other repositories (such as rejected)
default page_size went from 100 to 10

Which makes collection list less straightforward to use for finding and cleaning up all collections.

Is that an intentional change?
(If so, updating tests in ansible/ansible-hub-ui#4013)

(Also, if the goal is to update the API endpoint, shouldn't this be using v3/plugin/ansible/search/collection-versions/?)

Expanding the capability of the collection delete command for use by …

56d8d73

…the cleanup script.

github-actions bot reviewed Jun 27, 2023

View reviewed changes

ironfroggy force-pushed the collection-delete branch from 9a13c57 to 56d8d73 Compare June 27, 2023 16:08

Add help text to cleanup script.

396ebb0

chr-stian requested review from MilanPospisil, appuk, drodowic, hendersonreed, himdel, jctanner and rochacbruno July 20, 2023 08:48

himdel mentioned this pull request Jul 23, 2023

test galaxykit#95 ansible/ansible-hub-ui#4013

Closed

himdel removed request for hendersonreed and himdel January 26, 2024 21:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Galaxy server clean-up script along with some changes to galaxykit to support it#95

Galaxy server clean-up script along with some changes to galaxykit to support it#95
ironfroggy wants to merge 2 commits intomainfrom
collection-delete

ironfroggy commented Jun 27, 2023 •

edited

Loading

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

github-actions bot Jun 27, 2023

Uh oh!

himdel commented Jul 21, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ironfroggy commented Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

himdel commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ironfroggy commented Jun 27, 2023 •

edited

Loading

himdel commented Jul 21, 2023 •

edited

Loading