feat: update nodejs and other packages #1340

elaine-mattos · 2025-07-15T21:27:35Z

Hi dear clearlydefined community,

My team is currently evaluating the ClearlyDefined API for integration into our compliance workflows. While testing in our fork, we noticed several deprecated dependencies and decided to modernize the setup.

This PR upgrades the codebase to support Node.js 20 24 and updates a range of dependencies. Along the way, we made necessary refactors due to breaking changes, and introduced improvements in initialization and error handling.

We're still getting familiar with the codebase, so please feel free to comment or guide us if we’ve misunderstood anything. Also, if this PR feels too large, we’re happy to split it into smaller pieces.

Some details

Node and Dependency updates:
- Upgraded project to Node.js 20 24;
- Updated multiple dependencies to their latest compatible versions.
Express 5 migration:
- Removed express-init in favor of direct async initialization;
- Updated middleware and route initialization to match Express 5 syntax;
- Changed app.options('*', cors()) to app.options('*splat', cors()) per new path syntax.
GitHub & GitLab API usage:
- Updated all usages of @octokit/rest and @gitbeaker/node to match new APIs and import styles;
- Replaced deprecated authentication and client initialization patterns.
YAML parsing:
- Replaced deprecated yaml.safeLoad and yaml.safeDump with their respectives yaml.loadand yaml.dump.
Logging:
- Introduced LOGGER_LOG_TO_CONSOLE environment variable to control log output to the console;
- Introduced the ENABLE_REST_VALIDATION_ERRORS environment variable to control REST input validation verbosity. When enabled, ajv will report all validation errors (allErrors: true), which is useful for debugging but may trigger deep object traversal warnings. More info
- Improved debug and error logs for critical operations.

Let us know your thoughts! ^.^

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

…es in the latest mkdip version Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

- Added support for the /splash endpoint - Updated optional route parameters to Express v5 syntax ({/:param}) - Adjusted root path and wildcards for routes like /definitions and /harvest Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

- Removed defaultHeaders due to incompatibility with new version. See: jdalrymple/gitbeaker#3634 Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

…tive AppInsights SDK - Adds support for a new env variable to control console logging output. - The third-party transport has been removed in favor of a direct integration using the SDK. Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

@ts-ignore

- The Redis client accepts as a valid runtime configuration, but the TypeScript definitions expect a different structure. Added @ts-ignore directive to bypass the type check. Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

JamieMagee · 2025-07-16T17:49:40Z

Related: clearlydefined/operations#133

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

henrysachs · 2025-07-21T08:57:19Z

@JamieMagee @qtomlinson @jeffmendoza @elrayle can someone of you approve the workflows? :)

JamieMagee · 2025-07-21T15:56:44Z

@henrysachs approved

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Copilot

Pull Request Overview

This PR modernizes the ClearlyDefined service by upgrading Node.js from v18 to v20 and updating numerous dependencies to their latest versions. The key changes address deprecated APIs and breaking changes introduced by major version updates of Express.js, GitHub/GitLab API clients, testing libraries, and various other packages.

Key changes include:

Node.js upgrade to v20 with dependency updates across the board
Express.js v5 migration with route syntax and initialization changes
Updated GitHub and GitLab API clients with new authentication patterns

Reviewed Changes

Copilot reviewed 48 out of 54 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
package.json	Updated Node.js version constraint and dependency versions
test/*/.js	Updated test assertions to use `deep.equalInAnyOrder` instead of deprecated `equalInAnyOrder`
providers/curation/github.js	Updated GitHub API calls to use new `rest.*` pattern
routes/*.js	Updated Express route patterns to v5 syntax
schemas/validator.js	Updated AJV validation setup for v8 compatibility
app.js	Removed express-init and updated initialization patterns

Comments suppressed due to low confidence (2)

test/routes/harvest.js:28

The error message has changed from 'should NOT have' to 'must NOT have', which may indicate a breaking change in the validation library. Verify this is the expected message format from the updated AJV version.

    expect(response._getData()).to.be.eq('data/0 must NOT have additional properties')

test/routes/harvest.js:37

Similar to above, the error message format has changed from 'should be' to 'must be'. Ensure this aligns with the updated AJV validation error messages.

    expect(response._getData()).to.be.eq('data/0/tool must be string')

test/routes/webhookCrawlerTests.js

providers/logging/winstonConfig.js

providers/caching/redis.js

routes/originMavenGoogle.js

routes/originGo.js

routes/definitions-1.0.0.js

providers/curation/github.js

business/definitionService.js

…t config from getting loaded during tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

…er to correctly pipe logs to appInsights Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

providers/logging/winstonConfig.js

providers/caching/redis.js

qtomlinson

Thanks for the contribution!

qtomlinson · 2025-09-24T15:11:42Z

@pombredanne Your feedback on this PR would be much appreciated. There are some backward compatibility concerns with Redis persistence that I think we should tackle separately, in a follow-up PR by either Elaine or myself.

pombredanne

@elaine-mattos Here are some comments and nit pickings for your consideration. We discussed that in a recent weekly call and the sentiment is that we should merge and do through testing first.

For the future, it would be really great to avoid these kind of big, massive PR that take a long time to review!

For instance you could have done a completely separate PR for each of:

logging enhancements
string formatting
type annotations
minor case changes in test names or comments changes

pombredanne · 2025-11-05T14:38:13Z

providers/caching/redis.js

IMHO we should invalidate an start clean, and not try to keep some compat with existing data which what @elaine-mattos has done here

providers/caching/redis.js

providers/logging/winstonConfig.js

routes/harvest.js

pombredanne · 2025-11-05T14:59:55Z

package.json

@@ -1,9 +1,12 @@
 {
  "name": "service",
  "version": "2.3.0",


We should bump this, likely to 3.0.0?

There is currently no breaking change, how about 2.4.0?

elaine-mattos · 2025-11-07T16:13:26Z

@elaine-mattos Here are some comments and nit pickings for your consideration. We discussed that in a recent weekly call and the sentiment is that we should merge and do through testing first.

For the future, it would be really great to avoid these kind of big, massive PR that take a long time to review!

For instance you could have done a completely separate PR for each of:

logging enhancements

string formatting

type annotations

minor case changes in test names or comments changes

Hi @pombredanne ,

thanks a lot for the review (and the other reviewers as well)!
Yes, I know this is a monster PR and this is why I mentioned in the description that I was willing to break it down if you guys see fit. I should have done this already, it would save us all a lot of time and effort!
However, I have another PR for the crawler coming up and there I'll just break all the changes up into smaller PRs, promise! :)
Looking forward to the meeting next week.

pombredanne · 2025-11-10T09:14:34Z

providers/caching/redis.js

+        this.logger.debug('Attempting to read uncompressed data format')
+        result = typeof cacheItem === 'string' ? cacheItem : cacheItem.toString()
+
+        if (!result.startsWith(objectPrefix)) {


Suggested change

if (!result.startsWith(objectPrefix)) {

if (result.startsWith(objectPrefix)) {

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

qtomlinson · 2025-12-19T23:59:34Z

@elaine-mattos PR1340 has been deployed to the development system. There are several integration test failures. Failure cases 1 and 3 under Verify service functions are related to recent changes in the crawler. Please feel free to conduct additional testing as needed.

Also noticed the following error a couple of times during logging.

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'ClientRequest'
    |     property 'res' -> object with constructor 'IncomingMessage'
    --- property 'req' closes the circle
    at JSON.stringify (<anonymous>)
    at Printf.template (/opt/service/providers/logging/winstonConfig.js:32:80)
    at Printf.transform (/opt/service/node_modules/logform/printf.js:11:26)
    at Format.transform (/opt/service/node_modules/logform/combine.js:20:24)
    at DerivedLogger._transform (/opt/service/node_modules/winston/lib/winston/logger.js:337:29)
    at Transform._read (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:166:10)
    at Transform._write (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:155:83)
    at doWrite (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:390:139)
    at writeOrBuffer (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:381:5)
    at Writable.write (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:302:11)

elaine-mattos · 2025-12-20T18:20:14Z

@elaine-mattos PR1340 has been deployed to the development system. There are several integration test failures. Failure cases 1 and 3 under Verify service functions are related to recent changes in the crawler. Please feel free to conduct additional testing as needed.

Also noticed the following error a couple of times during logging.

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'ClientRequest'
    |     property 'res' -> object with constructor 'IncomingMessage'
    --- property 'req' closes the circle
    at JSON.stringify (<anonymous>)
    at Printf.template (/opt/service/providers/logging/winstonConfig.js:32:80)
    at Printf.transform (/opt/service/node_modules/logform/printf.js:11:26)
    at Format.transform (/opt/service/node_modules/logform/combine.js:20:24)
    at DerivedLogger._transform (/opt/service/node_modules/winston/lib/winston/logger.js:337:29)
    at Transform._read (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:166:10)
    at Transform._write (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:155:83)
    at doWrite (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:390:139)
    at writeOrBuffer (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:381:5)
    at Writable.write (/opt/service/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:302:11)

Hi @qtomlinson ,

I see the issues and it looks like they're getting caused by the express update. I'll take a look at them over the holidays and will keep you guys posted!

elaine-mattos · 2026-01-02T12:53:56Z

@qtomlinson , The query parameter type issue is fixed in PR #1388, please let me know your thoughts!
Also added PR #1389 to address the “Converting circular structure to JSON” issues. Would also love to know what you guys think! 😃

elaine-mattos added 16 commits July 15, 2025 20:48

feat: migrate node

ea75a7f

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

feat: update octokit

b7150d0

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: add defensive checks

f119564

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: use promise-based mkdirp directly without promisify due to chang…

0a6eab2

…es in the latest mkdip version Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: remove old express-init

1e1fe2d

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: encode deflated data as base64 for Redis compatibility

d5c74b8

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: update ajv

5a43000

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: update express

d379475

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: update yaml

1d3d057

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: update gitbeaker

b0c2355

- Removed defaultHeaders due to incompatibility with new version. See: jdalrymple/gitbeaker#3634 Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

chore: improve winston config

3f68325

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix(harvest): instantiate with caching again

4a70108

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: update minimatch

e2572e8

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

chore: improve error handling

22e3502

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

chore: push package updates

ab07192

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

elaine-mattos mentioned this pull request Jul 15, 2025

Migrate off Node 18 #1339

Closed

elaine-mattos added 5 commits July 16, 2025 10:35

fix(tests): update octokit calls, fix deep equalities, initialize logger

59005a0

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: linting issues

9753f70

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: add back comment

cf004fb

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

elaine-mattos marked this pull request as ready for review July 16, 2025 11:15

fix: update node version

0db4a04

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: eslint errors

eb5028b

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

JamieMagee requested a review from Copilot July 21, 2025 18:53

Copilot AI reviewed Jul 21, 2025

View reviewed changes

elaine-mattos added 4 commits August 13, 2025 17:13

Merge branch 'master' into feat/update-nodejs

9d9ac44

fix(tests): add appVersion back to the environment variable to preven…

01ff3c6

…t config from getting loaded during tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

feat: make ajv allErrors control variable more explicit

adf378d

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix(logging): change event listener from info to data on Winston logg…

9bebcb4

…er to correctly pipe logs to appInsights Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

elaine-mattos commented Aug 14, 2025

View reviewed changes

providers/logging/winstonConfig.js Show resolved Hide resolved

qtomlinson reviewed Aug 20, 2025

View reviewed changes

providers/caching/redis.js Outdated Show resolved Hide resolved

qtomlinson approved these changes Sep 10, 2025

View reviewed changes

qtomlinson requested a review from pombredanne September 24, 2025 15:05

Merge branch 'master' into feat/update-nodejs

e2abb12

pombredanne reviewed Nov 5, 2025

View reviewed changes

pombredanne reviewed Nov 10, 2025

View reviewed changes

elaine-mattos added 4 commits November 10, 2025 15:31

fix: simplify cache item retrieval by removing fallback mechanism

7846a10

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

Merge branch 'master' into feat/update-nodejs

d06f958

fix(webhook): handle body parsing for body-parser 2.x compatibility

e38ffa7

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

fix: remove test case for unused cache fallback strategy

8e7294f

Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com>

brettfo mentioned this pull request Nov 17, 2025

update to node 24 packages #1366

Open

qtomlinson and others added 2 commits November 24, 2025 15:53

Merge branch 'master' into feat/update-nodejs

3abb1a3

Merge remote-tracking branch 'origin/master' into feat/update-nodejs

0c1e102

jeffmendoza force-pushed the feat/update-nodejs branch from c31fd28 to 0c1e102 Compare December 16, 2025 22:23

jeffmendoza merged commit 33ca449 into clearlydefined:master Dec 16, 2025
4 checks passed

elaine-mattos deleted the feat/update-nodejs branch December 17, 2025 07:38

elaine-mattos mentioned this pull request Dec 17, 2025

feat(cache): add backward compatibility for pako 1.x to 2.x migration #1383

Open

qtomlinson mentioned this pull request Dec 19, 2025

Update version for MongoDB Node.js driver #1385

Merged

elaine-mattos mentioned this pull request Jan 2, 2026

fix: Add Ajv type coercion for Express 5 query parameter compatibility #1388

Open

elaine-mattos mentioned this pull request Jan 2, 2026

fix(logging): circular JSON errors in Winston logs by summarizing HTTP/Axios objects #1389

Open

	if (!result.startsWith(objectPrefix)) {
	if (result.startsWith(objectPrefix)) {

feat: update nodejs and other packages #1340

feat: update nodejs and other packages #1340

Uh oh!

Conversation

elaine-mattos commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Some details

Uh oh!

JamieMagee commented Jul 16, 2025

Uh oh!

henrysachs commented Jul 21, 2025

Uh oh!

JamieMagee commented Jul 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qtomlinson left a comment

Choose a reason for hiding this comment

Uh oh!

qtomlinson commented Sep 24, 2025

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

pombredanne Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

qtomlinson Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

elaine-mattos commented Nov 7, 2025

Uh oh!

pombredanne Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

elaine-mattos Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qtomlinson commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elaine-mattos commented Dec 20, 2025

Uh oh!

elaine-mattos commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

elaine-mattos commented Jul 15, 2025 •

edited

Loading

qtomlinson commented Dec 19, 2025 •

edited

Loading

elaine-mattos commented Jan 2, 2026 •

edited

Loading