Restructure robot instructions with hierarchical organization and shared content

[Copilot]

Change Description

Robot instructions for GitHub Copilot and Claude Code contained significant duplication (~80%) and lacked hierarchical organization. This restructure creates a single source of truth for project knowledge while maintaining tool-specific guidance.

Incorporates feedback from PR #656 and #635 to ensure all information is accurate and up-to-date.

Solution Description

New File Structure

• HYRAX_GUIDE.md (319 lines): Shared reference for project knowledge

  • Project overview, design principles, architecture
  • Plugin system (models, datasets, verbs), configuration system
  • Development setup, commands, repository structure
  • Component addition guides, troubleshooting
  • Updated with corrections: Python >= 3.11, Jupyter as primary interface (CLI secondary), config key conventions

• CLAUDE.md (210 lines): Claude Code-specific guidance

  • Command execution timeouts table
  • Sub-agent delegation patterns (explore, task, general-purpose)
  • Working patterns and validation scenarios
  • References HYRAX_GUIDE.md for comprehensive information

• .github/copilot-instructions.md (207 lines): GitHub Copilot-specific guidance

  • Long-running operation warnings
  • Validation workflows and implementation details
  • Quick navigation guides
  • References HYRAX_GUIDE.md for comprehensive information

Key Improvements

• 85%+ deduplication: Common content centralized in HYRAX_GUIDE.md
• Maintainability: Updates required in single location
• Tool-specific focus: Each file contains only relevant guidance
• Scalability: Easy to add support for additional AI tools (GEMINI.md, AGENTS.md)
• Verified structure: All markdown links and anchor references validated
• Accuracy: Incorporated feedback from PRs Add design goals and north stars to Copilot instructions #656 and Robot Guidance #635 including Python version, interface hierarchy, config conventions, and architectural clarifications

Identified Potential Claude Skills

Analysis completed identifying 5 potential skills for future extraction:

1. Hyrax Development Workflow (High Priority) - Setup → code → test → commit patterns
2. Hyrax Testing Strategy (High Priority) - Slow/fast markers, parallel execution, timeouts
3. Adding Hyrax Components (Medium Priority) - Models, datasets, verbs with decorators
4. Hyrax Configuration System (Medium Priority) - ConfigDict, defaults, key=false convention
5. Debugging Hyrax Builds (Low Priority) - Common issues and troubleshooting

Rationales documented for review before implementation.

Example Reference Pattern

# CLAUDE.md
**🔗 For comprehensive project information, see [HYRAX_GUIDE.md](./HYRAX_GUIDE.md)**

**Always refer to [HYRAX_GUIDE.md](./HYRAX_GUIDE.md) for:**
- Design principles and architectural conventions
- Configuration system details → [HYRAX_GUIDE.md](./HYRAX_GUIDE.md#configuration-system)

Code Quality

• I have read the Contribution Guide and agree to the Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• code.claude.com
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• www.humanlayer.dev
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[mtauraso]

@copilot Can you do two things:

1. Read the comments on Add design goals and north stars to Copilot instructions #656 and Robot Guidance #635 to ensure all information is up to date.

2. Identify any claude skills that could be pulled out from the overall corpus, and write a short rationale for each in a comment on this PR. DO NOT write any skills yet.

[Copilot]

@copilot Can you do two things:

1. Read the comments on Add design goals and north stars to Copilot instructions #656 and Robot Guidance #635 to ensure all information is up to ...

Task 1: Information Updates from PR Comments

I've reviewed comments from PR #656 and #635. Key updates needed:

From PR #656 (copilot-instructions):

1. ✅ Primary interface is Jupyter notebook, CLI is secondary (though CLI can do everything notebooks can for HPC contexts)
2. ✅ All config keys must have defaults in TOML (using key=false to mean None/user-must-provide)
3. ✅ Python version: >= 3.11 (not 3.9)
4. ✅ No CHANGELOG file - we use Git for change tracking
5. ✅ Version pinning guidance: Not a pillar, experimental for DataProvider config
6. ✅ Documentation preference: Document current, push users from old to new via error messages/warnings
7. ✅ MLflow: Part of logging system, but results dir is the backbone
8. ✅ Manifest files: Currently minimal working version, may move to standard middleware for 10M-100M scale

From PR #635 (CLAUDE.md):

1. ✅ External libraries can't define new verbs (yet)
2. ✅ Config immutability has been relaxed somewhat
3. ✅ InferenceDataSet/Writer should be treated as black box for storing results with order/ID preservation
4. ✅ Visualize verb is most hacky/specialized with different code maturity
5. ✅ Use linter to correct code style rather than writing perfect code initially
6. ✅ Some sections feel like they should be Claude skills (see Task 2)

Task 2: Potential Claude Skills

Based on review comments and content analysis, here are Claude skills that could be extracted:

1. Hyrax Development Workflow (High Priority)

Rationale: Comments on PR #635 suggest "Many of these sections feel like they should be claude skills" and robots get "bogged down with the amount of dev tooling." This skill would provide structured guidance for the dev setup → code → test → commit workflow.
Scope: Setup commands, validation workflow, pre-commit hooks, timeout expectations
Benefit: Reduces confusion about which tools to use when, prevents premature command cancellation

2. Hyrax Testing Strategy (High Priority)

Rationale: Complex testing setup with slow/fast markers, parallel execution, and specific timeout requirements. This is a discrete, repeatable task that causes confusion.
Scope: When to use @pytest.mark.slow, parallel testing, fixture usage, network retry strategies
Benefit: Clearer testing approach, reduces test failures from premature cancellation

3. Adding Hyrax Components (Medium Priority)

Rationale: Well-defined patterns for adding models, datasets, and verbs with specific decorator/registration requirements. This is a common, structured task.
Scope: Model addition (forward/train_step/prepare_inputs), dataset registration, verb creation with @decorators
Benefit: Ensures correct implementation patterns, reduces registration issues

4. Hyrax Configuration System (Medium Priority)

Rationale: Feedback emphasizes config complexity - all keys need defaults, key=false means None, ConfigDict vs dict, Pydantic validation. This is a common source of errors.
Scope: ConfigDict usage, default requirements, key=false convention, TOML structure, immutability rules
Benefit: Prevents "config key not found" errors, ensures proper config usage

5. Debugging Hyrax Builds (Low Priority)

Rationale: Standard troubleshooting patterns for common issues (import errors, network timeouts, CLI not found). Less about skills, more about reference.
Scope: Common issues checklist, retry strategies, environment validation
Benefit: Faster issue resolution, but could remain in main guidance

Not recommended as skills: Repository structure navigation, architecture overview - these are better as quick reference in main file.