Skip to content

feat: Batch collection from URL list file #23

@Snider

Description

@Snider

Summary

Add ability to read URLs from a file and collect them all in batch.

Use Case

Skills have registries with many URLs. Need to collect all of them efficiently.

Proposed Commands

# From file
borg collect batch urls.txt

# From JSON registry
borg collect batch registry.json --jq '.papers[].url'

# Stdin
cat urls.txt | borg collect batch -

File Formats

# urls.txt - one URL per line
https://example.com/paper1.pdf
https://example.com/paper2.pdf

# Or with metadata
https://example.com/paper1.pdf|paper1|cryptography

Options

  • --parallel N - concurrent downloads
  • --delay 1s - delay between requests
  • --output-dir - base output directory
  • --continue - skip already collected

Acceptance Criteria

  • Read URLs from text file
  • Read URLs from JSON with jq filter
  • Support stdin piping
  • Parallel downloads with configurable concurrency
  • Rate limiting
  • Progress reporting

Metadata

Metadata

Assignees

No one assigned

    Labels

    julesFor Jules AI to work on

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions