Skip to content

Change package URL output to Google Maven format #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woocheol-lge merged 1 commit into from
Apr 27, 2026
Merged

Change package URL output to Google Maven format #310

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 22 additions & 23 deletions src/fosslight_dependency/_package_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import subprocess
import shutil
import stat
import requests
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Locate dependency manifests and check for a `requests` declaration.
fd -t f -H -E .git '^(setup\.py|setup\.cfg|pyproject\.toml|requirements.*\.txt|Pipfile|Pipfile\.lock)$' \
  | xargs -I{} sh -c 'echo "=== {} ==="; grep -nE "(^|[^a-zA-Z0-9_])requests([^a-zA-Z0-9_]|$)" "{}" || echo "(no match)"'

Repository: fosslight/fosslight_dependency_scanner

Length of output: 390

🏁 Script executed:

#!/bin/bash

# 1. Check if setup.py exists at root
echo "=== Checking for setup.py ==="
find . -maxdepth 1 -name "setup.py" -type f

# 2. Read the main dependency files
echo -e "\n=== pyproject.toml ==="
cat pyproject.toml 2>/dev/null || echo "(file not found)"

echo -e "\n=== setup.py ==="
cat setup.py 2>/dev/null || echo "(file not found)"

echo -e "\n=== requirements.txt ==="
cat requirements.txt 2>/dev/null || echo "(file not found)"

# 3. Confirm the import statement at line 14 of _package_manager.py
echo -e "\n=== _package_manager.py lines 1-20 ==="
head -20 src/fosslight_dependency/_package_manager.py

Repository: fosslight/fosslight_dependency_scanner

Length of output: 2198

Add requests to the project's runtime dependencies.

requests is imported unconditionally at line 14 of this module. It is not currently declared in pyproject.toml or requirements.txt, which will cause fresh installs to fail with an ImportError. Add requests to the dependencies list in pyproject.toml (and requirements.txt if maintained).

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/fosslight_dependency/_package_manager.py` at line 14, The module contains
an unconditional "import requests" which means the project must declare requests
as a runtime dependency; add "requests" to the dependencies section of
pyproject.toml (with an appropriate version spec, e.g. "requests>=2.28,<3") and
also add it to requirements.txt if your repo maintains one so fresh installs
won't raise ImportError; verify the package is listed under the same environment
(main/runtime) and run a local install to confirm the import resolves.

from packageurl.contrib import url2purl
from askalono import identify
import fosslight_util.constant as constant
Expand Down Expand Up @@ -566,26 +567,24 @@ def parse_gradle_download_lines(stdout_text, package_manager_name=''):

def get_download_location(download_url_map, group_id, artifact_id, version, mvnrepo_url):
actual_key = f"{group_id}:{artifact_id}:{version}"
if download_url_map:
try:
actual_url = download_url_map.get(actual_key)

use_mvnrepo = True
if actual_url:
central_like = ("repo1.maven.org" in actual_url) or ("repo.maven.apache.org" in actual_url)
google_like = (("maven.google.com" in actual_url) or
("dl.google.com/android/maven2" in actual_url) or
("dl.google.com/dl/android/maven2" in actual_url))
if central_like or google_like:
use_mvnrepo = True
else:
use_mvnrepo = False
except Exception as e:
logger.debug(f"Failed to get download location from download_url_map: {e}")
use_mvnrepo = True
else:
use_mvnrepo = True
if use_mvnrepo:
return f"{mvnrepo_url}{group_id}/{artifact_id}/{version}"
else:
return actual_url
actual_url = download_url_map.get(actual_key) if download_url_map else None
if actual_url:
if any(host in actual_url for host in ("repo1.maven.org", "repo.maven.apache.org")):
return f"{mvnrepo_url}{group_id}/{artifact_id}/{version}"
if not any(host in actual_url for host in (
"maven.google.com", "dl.google.com/android/maven2", "dl.google.com/dl/android/maven2")):
return actual_url
return get_google_maven_url(mvnrepo_url, group_id, artifact_id, version)


def get_google_maven_url(mvnrepo_url, group_id, artifact_id, version, ):
group_path = group_id.replace('.', '/')
pom_url = (f"https://dl.google.com/dl/android/maven2"
f"/{group_path}/{artifact_id}/{version}/{artifact_id}-{version}.pom")
try:
resp = requests.head(pom_url, timeout=5, allow_redirects=True)
if resp.status_code == 200:
return f"https://maven.google.com/web/index.html#{group_id}:{artifact_id}:{version}"
except Exception:
logger.debug(f"Failed to check Google Maven URL: {pom_url}")
return f"{mvnrepo_url}{group_id}/{artifact_id}/{version}"
Comment thread
woocheol-lge marked this conversation as resolved.
Comment on lines 568 to +590
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Per‑artifact HEAD probe can substantially slow large scans.

get_download_location is invoked once per oss_item by Android/Gradle/Maven callers. Whenever the mapped URL is None or Google‑hosted, get_google_maven_url issues a synchronous requests.head with a 5 s timeout. For projects with hundreds of dependencies, or when dl.google.com is unreachable (offline / corporate proxy), the cumulative wait time grows linearly and the same (group_id, artifact_id, version) may be probed repeatedly across duplicates. Consider memoizing the probe result and/or short‑circuiting when several consecutive probes have failed.

♻️ Suggested cache to avoid repeated network round‑trips 
+_GOOGLE_MAVEN_CACHE: dict[tuple[str, str, str], str] = {}
+
 def get_google_maven_url(mvnrepo_url, group_id, artifact_id, version, ):
+    cache_key = (group_id, artifact_id, version)
+    if cache_key in _GOOGLE_MAVEN_CACHE:
+        return _GOOGLE_MAVEN_CACHE[cache_key]
     group_path = group_id.replace('.', '/')
     pom_url = (f"https://dl.google.com/dl/android/maven2"
                f"/{group_path}/{artifact_id}/{version}/{artifact_id}-{version}.pom")
+    result = f"{mvnrepo_url}{group_id}/{artifact_id}/{version}"
     try:
         resp = requests.head(pom_url, timeout=5, allow_redirects=True)
         if resp.status_code == 200:
-            return f"https://maven.google.com/web/index.html#{group_id}:{artifact_id}:{version}"
+            result = f"https://maven.google.com/web/index.html#{group_id}:{artifact_id}:{version}"
     except Exception:
         logger.debug(f"Failed to check Google Maven URL: {pom_url}")
-    return f"{mvnrepo_url}{group_id}/{artifact_id}/{version}"
+    _GOOGLE_MAVEN_CACHE[cache_key] = result
+    return result
🧰 Tools
🪛 Ruff (0.15.11)

[warning] 588-588: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/fosslight_dependency/_package_manager.py` around lines 568 - 590,
get_download_location/get_google_maven_url currently issues a synchronous
requests.head per artifact which can cause large cumulative delays; add a simple
in-memory memoization keyed by (group_id, artifact_id, version) in the module
and have get_google_maven_url check that cache before issuing requests.head and
store both positive (Google URL) and negative (fallback) outcomes; additionally
track a short-lived failure counter (or timestamped negative cache) to
short‑circuit repeated probes after several failures and return the mvnrepo
fallback immediately; update get_google_maven_url to use the cache and failure
threshold and keep logging via logger when a probe is skipped.

Loading