Update `collect` to take input with specified format by davidallendj · Pull Request #130 · OpenCHAMI/magellan

davidallendj · 2026-04-06T22:11:34Z

Pull Request Template

Thank you for your contribution! Please ensure the following before submitting:

Checklist

My code follows the style guidelines of this project
I have added/updated comments where needed
I have added tests that prove my fix is effective or my feature works
I have run make test (or equivalent) locally and all tests pass
DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
[] REUSE Compliance:
- Each new/modified source file has SPDX copyright and license headers
- Any non-commentable files include a <filename>.license sidecar
- All referenced licenses are present in the LICENSES/ directory

Description

This PR changes the collect command to take JSON or YAML as input from the scan command. This allows for the entire discovery process to be completed as a single command without hitting the file system.

Examples:

# scan -> collect
magellan scan --subnet 172.18.0.0/24 --port 5000 -l info -i -F json | magellan collect -f json --show-output -i

# scan -> collect -> send
magellan scan --subnet 172.18.0.0/24 --port 5000 -l info -i -F json | magellan collect -f json --show-output -i | magellan send https://smd.example.com

# collect -> send
magellan collect pdu x3000m0 x3000m1 -u admin -p password | magellan send https://smd.example.com

Fixes #128

Type of Change

Bug fix
New feature
Breaking change
Documentation update

For more info, see Contributing Guidelines.

synackd

Testing with real BMCs.

Scanning:

$ ./magellan scan --subnet 172.16.0.0/24 -l info -i -F json --insecure
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:45:41-06:00","message":"failed to register completion function"}
[
  {
    "host": "https://172.16.0.105",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.948173759-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.104",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947999177-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.102",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947644981-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.101",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947476248-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.103",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947691662-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.199",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.948031377-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.198",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947765703-06:00",
    "service_type": "Redfish"
  }
]

Scanning piping to collect:

$ ./magellan scan --subnet 172.16.0.0/24 -l info -i -F json --insecure | ./magellan collect -f json --show-output -i
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:46:07-06:00","message":"failed to register completion function"}
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:46:07-06:00","message":"failed to register completion function"}
{"level":"error","error":"no file found","time":"2026-05-14T13:46:07-06:00","caller":"/opt/shared/magellan/cmd/collect.go:61","message":"failed to get scanned results from cache"}
{"level":"error","error":"no assets found","time":"2026-05-14T13:46:07-06:00","caller":"/opt/shared/magellan/cmd/collect.go:178","message":"failed to collect data"}
null

davidallendj · 2026-06-01T20:01:40Z

Testing with real BMCs.

Scanning:

$ ./magellan scan --subnet 172.16.0.0/24 -l info -i -F json --insecure
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:45:41-06:00","message":"failed to register completion function"}
[
  {
    "host": "https://172.16.0.105",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.948173759-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.104",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947999177-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.102",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947644981-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.101",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947476248-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.103",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947691662-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.199",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.948031377-06:00",
    "service_type": "Redfish"
  },
  {
    "host": "https://172.16.0.198",
    "port": 443,
    "protocol": "tcp",
    "state": true,
    "timestamp": "2026-05-14T13:45:41.947765703-06:00",
    "service_type": "Redfish"
  }
]

Scanning piping to collect:

$ ./magellan scan --subnet 172.16.0.0/24 -l info -i -F json --insecure | ./magellan collect -f json --show-output -i
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:46:07-06:00","message":"failed to register completion function"}
{"level":"warn","error":"RegisterFlagCompletionFunc: flag 'format' does not exist","time":"2026-05-14T13:46:07-06:00","message":"failed to register completion function"}
{"level":"error","error":"no file found","time":"2026-05-14T13:46:07-06:00","caller":"/opt/shared/magellan/cmd/collect.go:61","message":"failed to get scanned results from cache"}
{"level":"error","error":"no assets found","time":"2026-05-14T13:46:07-06:00","caller":"/opt/shared/magellan/cmd/collect.go:178","message":"failed to collect data"}
null

I think this may be happening because the --cache-path flag is set to a non-empty string by default.

This is where the check happens:
https://github.com/OpenCHAMI/magellan/blob/allend/collect-json-input/cmd/collect.go#L58

And here is where the flag default is set:
https://github.com/OpenCHAMI/magellan/blob/allend/collect-json-input/cmd/root.go#L93

I can run it by setting --cache "" but I'm running to another problem that I'm working to address.

./magellan scan --subnet 172.16.0.0/24 -l info -i -F json | ./magellan collect -f json --show-output -i --cache "" -l debug

davidallendj · 2026-06-01T21:01:34Z

@synackd It should be working now. I had to add some missing logic back to cmd/collect.go and update some of the CLI flags. I don't think I like having to explicitly set --cache "" for this to work though...

This is the command I'm using to test currently:

./magellan scan --subnet 172.16.0.0/24 -l info -i -F json | ./magellan collect -f json --show-output -i --cache "" -l debug

synackd · 2026-06-01T21:57:30Z

That command line worked for me. What if we added a check to see that stdin was open and, if so, used that instead of a cache unless explicitly told to do so?

synackd · 2026-06-23T20:15:57Z

That command line worked for me. What if we added a check to see that stdin was open and, if so, used that instead of a cache unless explicitly told to do so?

@davidallendj Just wanted to check in again with this to see if this was something we wanted to add in this PR.

davidallendj · 2026-06-24T21:49:40Z

That command line worked for me. What if we added a check to see that stdin was open and, if so, used that instead of a cache unless explicitly told to do so?

@davidallendj Just wanted to check in again with this to see if this was something we wanted to add in this PR.

Yes, I think we should. I just haven't gotten back around to add it. If you already have it implemented though, feel free to push to this branch and we can get this merged.

synackd · 2026-06-24T22:29:10Z

Understood. I didn't attempt it in order to avoid possibly duplicating your work. All yours. 🙂

alexlovelltroy · 2026-07-10T17:36:35Z

@synackd seems to have approved. Is that true?

synackd · 2026-07-13T23:11:17Z

I believe we are waiting for @davidallendj to implement this:

That command line worked for me. What if we added a check to see that stdin was open and, if so, used that instead of a cache unless explicitly told to do so?

@davidallendj Just wanted to check in again with this to see if this was something we wanted to add in this PR.

Yes, I think we should. I just haven't gotten back around to add it. If you already have it implemented though, feel free to push to this branch and we can get this merged.

davidallendj · 2026-07-22T23:01:25Z

Sorry this took so long. It was taking a minute to get some of the behaviors correct, but both of these cases should work like expected and --cache "" is no longer required.

./magellan scan --subnet 172.18.0.0/24 -l info -i --port 5000
./magellan collect -f json --show-output -i -l debug

./magellan scan --subnet 172.18.0.0/24 -l info -i -F json --port 5000 | ./magellan collect -f json --show-output -i -l debug

Signed-off-by: David Allen <davidallendj@gmail.com>

Signed-off-by: Polly Labs <pollychen.lab@gmail.com> Signed-off-by: David Allen <davidallendj@gmail.com>

This reverts commit bcb1a01. Signed-off-by: David J. Allen <davidallendj@gmail.com> Signed-off-by: David Allen <davidallendj@gmail.com>

Signed-off-by: David Allen <davidallendj@gmail.com>

synackd · 2026-07-23T15:38:19Z

Just tested again with:

Piping scan output into collect:

./magellan scan --subnet 172.16.0.0/24 -l info -i -F json | ./magellan collect -f json --show-output -i -l debug

Using default cache path:

./magellan scan --subnet 172.16.0.0/24 -l info -i
./magellan collect --show-output -i

Using non-default cache path:

./magellan scan --subnet 172.16.0.0/24 -l info -i --cache ./assets.db
./magellan collect --show-output -i --cache ./assets.db

These all worked as expected.

synackd

One possible code fix and several documentation updates.

Signed-off-by: David Allen <davidallendj@gmail.com>

synackd

I retested all of the mentioned test cases and they worked as expected. However, I found an error when testing YAML with piping commands. There is some other weird behavior I found as well as well as some small documentation updates that are needed.

Signed-off-by: David Allen <davidallendj@gmail.com>

synackd

Looks like most things are fixed, but found one more potential issue. I think after that and #130 (comment) we should be good. Will test after changes are pushed.

Signed-off-by: David Allen <davidallendj@gmail.com>

synackd

Mainly one code fix with some more documentation fixes.

synackd · 2026-07-28T22:21:12Z

 			} else {
 				// input should be a valid JSON
 				var (
 					data  JSONArray
 					input = []byte(arg)
 					err   error
 				)
 				if !json.Valid(input) {
 					log.Error().Msgf("argument %d not a valid JSON", i)
 					continue
 				}
 				err = json.Unmarshal(input, &data)
 				if err != nil {
 					log.Error().Err(err).Msgf("failed to unmarshal input for argument %d", i)
 				}
 				return data
 			}
 		}


This should use format.UnmarshalData() instead of the JSON unmarshaller so inline YAML data passed with -d/--data will honor -f yaml, e.g:

} else { // input should be valid data in the selected input format var ( data JSONArray input = []byte(arg) err error ) err = format.UnmarshalData(input, &data, inputFormat) if err != nil { log.Error().Err(err).Msgf("failed to unmarshal %s input for argument %d", inputFormat, i) } return data }

synackd · 2026-07-28T22:22:20Z

  magellan scan --subnet 10.0.0.0/16 -i

+  // same as above example but output is in JSON without caching
+  magellan scan --subnet 10.0.0.0/16 -i -f json --disable-cache


Shouldn't this be -F?

synackd · 2026-07-28T22:28:33Z

  magellan send -d @inventory.json https://smd.openchami.cluster

-  // send data from multiple files (must specify -f/--format if not JSON)
+  // send data from multiple files (must specify -F/--output-format if not JSON)


Should this be -f?

synackd · 2026-07-28T22:30:05Z

-// assumes subnet without CIDR has a subnet-mask of 255.255.0.0++
+// assumes subnet without CIDR has a subnet-mask of 255.255.0.0


We'll want to keep the ++ otherwise the command will be on the same line as the comment.

synackd · 2026-07-28T22:30:52Z

 magellan scan --subnet 10.0.0.0/16 -i

+// same as above example but output is in JSON without caching
+magellan scan --subnet 10.0.0.0/16 -i -f json --disable-cache


Shouldn't this use -F since we are outputting?

synackd · 2026-07-28T22:34:07Z

 	sendCmd.Flags().StringArrayVarP(&sendDataArgs, "data", "d", []string{}, "Set the data to send to specified host (prepend @ for files)")
-	sendCmd.Flags().VarP(&sendInputFormat, "format", "F", "Set the default data input format (json|yaml) can be overridden by file extension")
-	sendCmd.Flags().BoolVarP(&forceUpdate, "force-update", "f", false, "Set flag to force update data sent to SMD")
+	sendCmd.Flags().VarP(&sendInputFormat, "input-format", "f", "Set the default data input format (json|yaml) can be overridden by file extension")


magellan/man/magellan.1.sc

Line 131 in 9ac3ace

magellan send -d @nodes.yaml -F yaml https://smd.example.com

-F -> -f to match this flag

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj force-pushed the allend/collect-json-input branch from e59ee6e to 92bdb3b Compare April 6, 2026 22:24

davidallendj requested review from alexlovelltroy and synackd April 7, 2026 14:42

synackd reviewed May 14, 2026

View reviewed changes

Comment thread cmd/collect.go

davidallendj force-pushed the allend/collect-json-input branch from cbe75b6 to 7b1e8d4 Compare June 1, 2026 20:32

synackd changed the title ~~Update collect take input with specified format~~ Update collect take input with specified format Jun 1, 2026

synackd changed the title ~~Update collect take input with specified format~~ Update collect to take input with specified format Jun 1, 2026

davidallendj mentioned this pull request Jul 1, 2026

[Bug]: scan command has no output if --format argument missing #134

Closed

1 task

davidallendj force-pushed the allend/collect-json-input branch 2 times, most recently from f817c9a to 98aa0e6 Compare July 22, 2026 23:02

davidallendj added 11 commits July 22, 2026 17:03

chore: updated go deps

c49a5d5

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: minor changes to scan plus added comments

f39f8da

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: reworked cmd/collect.go input and removed token logic

d84955f

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: sqlite driver issue

d4842cd

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: removed unnecessary code

39f4678

Signed-off-by: David Allen <davidallendj@gmail.com>

tests: updated cmd and tests to work with changes

b9a3642

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: make crawl output flag consistent with collect

73d579b

Signed-off-by: David Allen <davidallendj@gmail.com>

docs: updated man pages

4910f23

Signed-off-by: David Allen <davidallendj@gmail.com>

docs: updated man pages

77b6741

Signed-off-by: David Allen <davidallendj@gmail.com>

docs: updated examples in cmd

5aa66fc

Signed-off-by: David Allen <davidallendj@gmail.com>

docs: updated README about workflows

1f2f8d3

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj and others added 13 commits July 22, 2026 17:03

fix: added missing append for incoming assets in cmd/collect.go

273d28e

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: added missing code to append assets from flag

5d6fc90

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: changed asset to pointer for format.UnmarshalData

f04727b

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: default scan output to JSON (#135)

19c55a5

Signed-off-by: Polly Labs <pollychen.lab@gmail.com> Signed-off-by: David Allen <davidallendj@gmail.com>

Revert "fix: default scan output to JSON (#135)" (#140)

5e0c515

This reverts commit bcb1a01. Signed-off-by: David J. Allen <davidallendj@gmail.com> Signed-off-by: David Allen <davidallendj@gmail.com>

fix: added condition to check for stdin in cmd/collect.go

650640c

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: updated the logic for reading stdin in cmd/collect.go

d1c7eba

Signed-off-by: David Allen <davidallendj@gmail.com>

chore: updated go deps

b105623

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: updated IsStdinEmpty impl and added debug messages

0cfd62a

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: changed condition to use IsStdinEmpty

59fb535

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: added missing short description for 'magellan secret retrieve'

0a360df

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: changed config msg from debug -> warn

f2d3d9d

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: updated/corrected IsStdinEmpty to properly check for data in stdin

1a4a3f2

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj force-pushed the allend/collect-json-input branch from 98aa0e6 to 1a4a3f2 Compare July 22, 2026 23:03

synackd requested changes Jul 23, 2026

View reviewed changes

Comment thread cmd/collect.go Outdated

Comment thread README.md

Comment thread man/magellan-collect.1.sc

Comment thread cmd/send.go

Comment thread cmd/list.go

Comment thread cmd/crawl.go

Comment thread cmd/send.go

davidallendj added 2 commits July 23, 2026 10:43

docs: updated references to format flags

11313e0

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: reorder logic to always process --data flag

daaf0c7

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj mentioned this pull request Jul 23, 2026

Explore/magellan serve #132

Open

synackd requested changes Jul 23, 2026

View reviewed changes

Comment thread cmd/collect.go Outdated

Comment thread README.md Outdated

Comment thread cmd/collect.go Outdated

Comment thread cmd/collect.go Outdated

Comment thread cmd/collect.go Outdated

Comment thread cmd/scan.go Outdated

Comment thread README.md Outdated

Comment thread README.md Outdated

refactor/docs: minor fix, changes, and doc updates

394637a

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj requested a review from synackd July 27, 2026 19:05

davidallendj added 2 commits July 28, 2026 08:59

fix: version command and added missing version info

1895b3b

Signed-off-by: David Allen <davidallendj@gmail.com>

refactor: downgraded config warning to debug

b578aea

Signed-off-by: David Allen <davidallendj@gmail.com>

synackd requested changes Jul 28, 2026

View reviewed changes

Comment thread cmd/send.go Outdated

davidallendj added 2 commits July 28, 2026 14:09

docs: fixed spacing for scan example

f3c26d8

Signed-off-by: David Allen <davidallendj@gmail.com>

fix: update processDataArgs to take format as arg

9ac3ace

Signed-off-by: David Allen <davidallendj@gmail.com>

davidallendj mentioned this pull request Jul 28, 2026

chore: Configure Renovate #136

Open

synackd requested changes Jul 28, 2026

View reviewed changes

fix: convert json.Unmarshal -> format.UnmarshalData in cmd/send.go

de2805c

Signed-off-by: David Allen <davidallendj@gmail.com>

		// assumes subnet without CIDR has a subnet-mask of 255.255.0.0++
		// assumes subnet without CIDR has a subnet-mask of 255.255.0.0

Uh oh!

Conversation

davidallendj commented Apr 6, 2026 • edited by synackd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Template

Checklist

Description

Type of Change

Uh oh!

synackd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidallendj commented Jun 1, 2026

Uh oh!

davidallendj commented Jun 1, 2026

Uh oh!

synackd commented Jun 1, 2026

Uh oh!

synackd commented Jun 23, 2026

Uh oh!

davidallendj commented Jun 24, 2026

Uh oh!

synackd commented Jun 24, 2026

Uh oh!

alexlovelltroy commented Jul 10, 2026

Uh oh!

synackd commented Jul 13, 2026

Uh oh!

davidallendj commented Jul 22, 2026

Uh oh!

synackd commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

synackd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

synackd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

synackd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

synackd left a comment

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

synackd Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

davidallendj commented Apr 6, 2026 •

edited by synackd

Loading

synackd commented Jul 23, 2026 •

edited

Loading