Skip to content

Conversation

@artemsafiyulin
Copy link

What does this PR do?

This PR make possible to use custom_queries with auto discovery. Previously it doesn't work.

Motivation

This feature needs for work.

How it has been tested:

  1. created couple databases:
docker exec pg-ad-test psql -U postgres -c "\l"
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 db1       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 db2       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(5 rows)
  1. created environment using ddev (output in file start_output.txt)

  2. prepare test config with custom metric and enabled autodiscovery

init_config: {}

instances:
  - host: 127.0.0.1
    port: 55432
    username: postgres
    password: postgres
    only_custom_queries: true
    database_autodiscovery:
      enabled: true
      include: [".*"]
      global_view_db: postgres
    custom_queries:
      - query: "select current_database(), 1 as dummy_metric"
        columns:
          - name: dbname
            type: tag
          - name: dummy_metric
            type: gauge
  1. Run test on old code wich don't work for custom_queries with autodiscovery (results in file run-old-behavior.txt) - here you can see that metric collected only for postgres db (default db)
/usr/local/ddev/bin/ddev -x env reload postgres latest && /usr/local/ddev/bin/ddev -x env agent postgres latest check postgres
  1. add changes from this PR to code
1. Current code
docker exec dd_postgres_latest bash -lc "nl -ba /opt/datadog-agent/embedded/lib/python3.13/site-packages/datadog_checks/postgres/postgres.py | sed -n '1138,1155p'"
  1138	                    self.statement_samples.run_job_loop(tags)
  1139	                    self.metadata_samples.run_job_loop(tags)
  1140	                if self._config.collect_wal_metrics:
  1141	                    # collect wal metrics for pg < 10, disabled by enabled
  1142	                    self._collect_wal_metrics()
  1143	
  1144	            if self._query_manager.queries:
  1145	                if self.autodiscovery:
  1146	                    for dbname in self.autodiscovery.get_items():
  1147	                        # run custom queries against each discovered database with the correct db tag
  1148	                        db = functools.partial(self.db_pool.get_connection, dbname=dbname)
  1149	                        autodiscovery_tags = list(self.tags_without_db) + [f"db:{dbname}"]
  1150	                        self._query_manager.executor = functools.partial(self.execute_query_raw, db=db)
  1151	                        self._query_manager.execute(extra_tags=autodiscovery_tags)
  1152	                else:
  1153	                    self._query_manager.executor = functools.partial(self.execute_query_raw, db=self.db)
  1154	                    self._query_manager.execute(extra_tags=tags)
  1. run tests on new code (results in file run-new-behavior.txt) - now you can see that custom query metric collected 3 times (for default postgres db and for two auto discovered db1 and db2)

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +1145 to +1151
if self.autodiscovery:
for dbname in self.autodiscovery.get_items():
# run custom queries against each discovered database with the correct db tag
db = functools.partial(self.db_pool.get_connection, dbname=dbname)
autodiscovery_tags = list(self.tags_without_db) + [f"db:{dbname}"]
self._query_manager.executor = functools.partial(self.execute_query_raw, db=db)
self._query_manager.execute(extra_tags=autodiscovery_tags)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid skipping custom queries on secondary DBs

When autodiscovery is enabled, this loop reuses the same QueryManager/Query instances for every database. Query.should_execute() updates its last-execution timestamp the first time a query runs, so any custom query with collection_interval set will be skipped for the remaining databases in the same check cycle and then continue to only run for the first database on subsequent cycles. This means multi-DB setups silently lose custom query metrics for all but the first autodiscovered DB whenever collection_interval is configured. Consider creating per-database query managers or resetting per-query execution state before each DB run so the interval is enforced per database.

Useful? React with 👍 / 👎.

@codecov
Copy link

codecov bot commented Dec 24, 2025

Codecov Report

❌ Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.35%. Comparing base (375fab0) to head (42bd13d).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant