From e8ac2ef8703797f195f3f0662c55072f0e7171aa Mon Sep 17 00:00:00 2001 From: Juniper Alanna <201364921+juniper-shopify@users.noreply.github.com> Date: Thu, 26 Mar 2026 15:35:48 -0400 Subject: [PATCH] Remove remaining legacy docs --- docs/AGENT_STEPS.md | 288 --------------------------------------- docs/INSTRUMENTATION.md | 243 --------------------------------- docs/ITERATION_SYNTAX.md | 147 -------------------- docs/VALIDATION.md | 178 ------------------------ 4 files changed, 856 deletions(-) delete mode 100644 docs/AGENT_STEPS.md delete mode 100644 docs/INSTRUMENTATION.md delete mode 100644 docs/ITERATION_SYNTAX.md delete mode 100644 docs/VALIDATION.md diff --git a/docs/AGENT_STEPS.md b/docs/AGENT_STEPS.md deleted file mode 100644 index 70a02432..00000000 --- a/docs/AGENT_STEPS.md +++ /dev/null @@ -1,288 +0,0 @@ -# Agent Steps in Roast - -Agent steps provide a way to send prompts directly to a coding agent (like Claude Code) without going through the standard LLM translation layer. This document explains when and how to use agent steps effectively. - -## What are Agent Steps? - -Agent steps are denoted by prefixing a step name with `^`. They bypass the normal LLM processing and send your prompt content directly to the CodingAgent tool. - -```yaml -steps: - # File-based prompts - - analyze_code # Regular step - processed by LLM first - - ^implement_fix # Agent step - direct to CodingAgent - - # Inline prompts - - Analyze the code quality and suggest improvements # Regular inline - - ^Fix all ESLint errors and apply Prettier formatting # Agent inline -``` - -Both file-based prompts (with directories like `implement_fix/prompt.md`) and inline prompts (text with spaces) are supported. - -## Agent Step Configuration Options - -Agent steps support two special configuration options: - -### `continue` (boolean, default: false) -When set to `true`, the agent continues from its previous session instead of starting fresh. This is useful for iterative development where you want the agent to maintain context across multiple steps. - -### `include_context_summary` (boolean, default: false) -When set to `true`, the agent receives an AI-generated summary of the workflow context as a system directive. This summary is intelligently tailored to the agent's upcoming task, including only relevant information from previous steps. The summary is generated by analyzing: -- The agent's prompt to understand what context would be helpful -- Previous step outputs and their relevance to the current task -- Workflow description and configuration -- Current working directory - -This helps the agent understand what has been done so far without overwhelming it with irrelevant details. NOTE: Without this option, the agent relies solely on what it is instructed to do either by the prompt or your specific step instructions. - -```yaml -steps: - - analyze_code - - implement_fix: ^Fix the issues identified in the analysis - - add_tests: ^Prepare and publish PR - -implement_fix: - include_context_summary: true # Include a summary of the workflow context so far - -add_tests: - continue: true # does not need context since is continuing from the previous step -``` - -## When to Use Agent Steps - -### Use Agent Steps When: - -1. **You need precise control over tool usage** - - When you want to ensure specific tools are used in a particular way - - When the task requires exact file manipulations or code transformations - -2. **Complex multi-file operations** - - When coordinating changes across multiple files - - When the operation requires specific sequencing of edits - -3. **Performance optimization** - - When you want to skip the LLM interpretation layer - - For well-defined tasks that don't need additional context - -### Use Regular Steps When: - -1. **You need natural language processing** - - When the prompt benefits from LLM interpretation - - When you want the LLM to add context or reasoning - -2. **Flexible, adaptive responses** - - When the exact approach might vary based on context - - When you want the LLM to make judgment calls - -## Practical Examples - -### Example 1: Database Migration - -**Regular Step (analyze_migration/prompt.md):** -```markdown -Look at the user's database schema and determine what migrations might be needed to support the new features they've described. Consider best practices for database design. -``` - -This benefits from LLM interpretation because: -- It needs to understand "best practices" in context -- It should make judgments about schema design -- The approach varies based on the specific features - -**Agent Step (^apply_migration/prompt.md):** -```markdown -Create a new migration file with the following specifications: - -1. Create a new migration file: db/migrate/{{timestamp}}_add_user_preferences.rb -2. The migration must include: - - Add column :users, :preferences, :jsonb, default: {} - - Add index :users, :preferences, using: :gin - - Add column :users, :notification_settings, :jsonb, default: {} -3. Ensure proper up/down methods -4. Follow Rails migration conventions exactly -``` - -This is better as an agent step because: -- The instructions are precise and technical -- No interpretation needed - just execution of steps known beforehand - -### Example 2: Code Refactoring - -**Regular Step (identify_code_smells/prompt.md):** -```markdown -Review the provided code and identify any code smells or anti-patterns. Consider things like: -- Long methods that do too much -- Duplicated code -- Poor naming -- Tight coupling -- Missing abstractions - -Explain why each identified issue is problematic. -``` - -This works well as a regular step because it requires judgment and explanation. - -**Agent Step (^extract_method/prompt.md):** -```markdown -Extract the authentication logic from UserController#create into a separate method: - -1. Read file: app/controllers/user_controller.rb -2. Find the code block from line 15-28 (the authentication logic) -3. Create a new private method called `authenticate_user_params` -4. Move the authentication logic to this new method -5. Replace the original code with a call to the new method -6. Ensure all variables are properly passed - -Use MultiEdit to make all changes in a single operation. -Preserve exact indentation and formatting. -``` - -This is ideal as an agent step because: -- Specific line numbers and method names -- Exact refactoring instructions -- No room for interpretation - -### Example 3: Test Generation - -**Regular Step (plan_test_coverage/prompt.md):** -```markdown -Analyze the {{file}} and determine what test cases would provide comprehensive coverage. Consider: -- Happy path scenarios -- Edge cases -- Error conditions -- Boundary values - -Focus on behavior, not implementation details. -``` - -**Agent Step (^implement_tests/prompt.md):** -```markdown -Create test file: test/models/user_validator_test.rb - -Implement exactly these test cases: -1. test "validates email format" - - Use valid emails: ["user@example.com", "test.user+tag@domain.co.uk"] - - Use invalid emails: ["invalid", "@example.com", "user@", "user space@example.com"] - -2. test "validates age is positive integer" - - Valid: [18, 25, 100] - - Invalid: [-1, 0, 17, 101, "twenty", nil] - -3. test "validates username uniqueness" - - Create user with username "testuser" - - Attempt to create second user with same username - - Assert validation error on :username - -Use minitest assertion style. -Each test must be independent. -Use setup method for common test data. -``` - -### Example 4: API Integration - -**Regular Step (design_api_client/prompt.md):** -```markdown -Design a client for the {{api_name}} API that follows Ruby best practices. Consider: -- Error handling strategies -- Rate limiting -- Authentication patterns -- Response parsing -- Testing approach - -Suggest an architecture that will be maintainable and extensible. -``` - -**Agent Step (^implement_api_client/prompt.md):** -```markdown -Implement the API client with this exact structure: - -1. Create file: lib/external_apis/weather_api/client.rb - ```ruby - module ExternalApis - module WeatherApi - class Client - include HTTParty - base_uri 'https://api.weather.com/v1' - - def initialize(api_key) - @api_key = api_key - @options = { headers: { 'Authorization' => "Bearer #{api_key}" } } - end - - def current_weather(location) - response = self.class.get("/current", @options.merge(query: { location: location })) - handle_response(response) - end - - private - - def handle_response(response) - raise ApiError, response.message unless response.success? - response.parsed_response - end - end - end - end - ``` - -2. Create file: lib/external_apis/weather_api/api_error.rb - Define custom exception class - -3. Update file: config/initializers/weather_api.rb - Add configuration for API endpoint and timeout - -Use exact module structure and method signatures shown. -``` - -## Best Practices - -1. **Be explicit about tool usage in agent steps** - ```markdown - # Good agent step - Use MultiEdit tool to update the following files: - - app/models/user.rb: Add validation - - test/models/user_test.rb: Add test case - ``` - -2. **Include specific line numbers or code markers when possible** - ```markdown - # Good agent step - In app/controllers/application_controller.rb: - - Find method `authenticate_user!` (around line 45) - - Add the following before the redirect_to call: - session[:return_to] = request.fullpath - ``` - -3. **Specify exact formatting requirements** - ```markdown - # Good agent step - Create method with exactly this signature: - def calculate_tax(amount, rate = 0.08) - - Ensure: - - Two-space indentation - - No trailing whitespace - - Blank line before method definition - ``` - -4. **Chain agent steps for complex operations** - ```yaml - steps: - # First understand the system - - analyze_current_architecture - - # Then execute precise changes - - ^create_service_objects - - ^update_controllers - - ^add_test_coverage - - # Finally verify - - verify_all_tests_pass - ``` - -## Summary - -Agent steps are powerful when you need direct control over tool usage and precise execution of technical tasks. They complement regular steps by handling the implementation details while regular steps handle the analysis and planning. - -The `continue` and `include_context_summary` options make agent steps even more powerful for iterative development workflows where maintaining context is important. - -Choose agent steps when precision matters more than interpretation. Choose regular steps when context and judgment are important. \ No newline at end of file diff --git a/docs/INSTRUMENTATION.md b/docs/INSTRUMENTATION.md deleted file mode 100644 index 72311381..00000000 --- a/docs/INSTRUMENTATION.md +++ /dev/null @@ -1,243 +0,0 @@ -# Instrumentation Hooks in Roast - -Roast provides built-in instrumentation hooks using ActiveSupport::Notifications, allowing you to track workflow execution, monitor performance, and integrate with your own monitoring systems. - -## Overview - -The instrumentation system emits events at key points during workflow execution: - -- Workflow lifecycle (start, complete) -- Step execution (start, complete, error) -- Chat completion/AI calls (start, complete, error) -- Tool function execution - -## Configuration - -To add custom instrumentation, create Ruby files in your project's `.roast/initializers/` directory. These files will be automatically loaded during workflow startup. - -Example structure: -``` -your-project/ - ├── .roast/ - │ └── initializers/ - │ ├── logging.rb - │ ├── metrics.rb - │ └── monitoring.rb - └── ... -``` - -## Available Events - -### Workflow Events - -- `roast.workflow.start` - Emitted when a workflow begins - - Payload: `{ workflow_path:, options:, name: }` - -- `roast.workflow.complete` - Emitted when a workflow completes - - Payload: `{ workflow_path:, success:, execution_time: }` - -### Step Events - -- `roast.step.start` - Emitted when a step begins execution - - Payload: `{ step_name:, resource_type:, workflow_name: }` - -- `roast.step.complete` - Emitted when a step completes successfully - - Payload: `{ step_name:, resource_type:, workflow_name:, success: true, execution_time:, result_size: }` - -- `roast.step.error` - Emitted when a step encounters an error - - Payload: `{ step_name:, resource_type:, workflow_name:, error:, message:, execution_time: }` - -### AI/Chat Completion Events - -- `roast.chat_completion.start` - Emitted before an AI API call - - Payload: `{ model:, parameters: }` - -- `roast.chat_completion.complete` - Emitted after successful AI API call - - Payload: `{ success: true, model:, parameters:, execution_time:, response_size: }` - -- `roast.chat_completion.error` - Emitted when AI API call fails - - Payload: `{ error:, message:, model:, parameters:, execution_time: }` - -### Tool Execution Events - -- `roast.tool.execute` - Emitted when a tool function is called - - Payload: `{ function_name:, params: }` - -- `roast.tool.complete` - Emitted when a tool function completes - - Payload: `{ function_name:, execution_time:, cache_enabled: }` - -- `roast.tool.error` - Emitted when a tool execution fails - - Payload: `{ function_name:, error:, message:, execution_time: }` - -## Example Usage - -### Basic Logging - -```ruby -# .roast/initializers/logging.rb -ActiveSupport::Notifications.subscribe(/roast\./) do |name, start, finish, id, payload| - duration = finish - start - puts "[#{name}] completed in #{duration.round(3)}s" -end -``` - -### Performance Monitoring - -```ruby -# .roast/initializers/performance.rb -ActiveSupport::Notifications.subscribe("roast.step.complete") do |name, start, finish, id, payload| - duration = finish - start - if duration > 10.0 - puts "WARNING: Step '#{payload[:step_name]}' took #{duration.round(1)}s" - end -end -``` - -### Integration with External Services - -```ruby -# .roast/initializers/metrics.rb -ActiveSupport::Notifications.subscribe("roast.workflow.complete") do |name, start, finish, id, payload| - duration = finish - start - - # Send to your metrics service - MyMetricsService.track_event("workflow_execution", { - workflow_path: payload[:workflow_path], - duration: duration, - success: payload[:success] - }) -end -``` - -### Internal Shopify Example - -For the internal Shopify version, you can use these instrumentation hooks to track metrics with Monorail: - -```ruby -# .roast/initializers/monorail.rb - -# Track workflow execution -ActiveSupport::Notifications.subscribe("roast.workflow.start") do |name, start, finish, id, payload| - Roast::Support::Monorail.track_command("run", { - "workflow_path" => payload[:workflow_path], - "options" => payload[:options], - "name" => payload[:name] - }) -end - -ActiveSupport::Notifications.subscribe("roast.workflow.complete") do |name, start, finish, id, payload| - Roast::Support::Monorail.track_command("run_complete", { - "workflow_path" => payload[:workflow_path], - "success" => payload[:success], - "execution_time" => payload[:execution_time] - }) -end - -# Track AI model usage and performance -ActiveSupport::Notifications.subscribe("roast.chat_completion.complete") do |name, start, finish, id, payload| - Roast::Support::Monorail.track_command("ai_usage", { - "model" => payload[:model], - "execution_time" => payload[:execution_time], - "response_size" => payload[:response_size], - "has_json" => payload[:parameters][:json] || false, - "has_loop" => payload[:parameters][:loop] || false - }) -end - -# Track tool execution and caching -ActiveSupport::Notifications.subscribe("roast.tool.complete") do |name, start, finish, id, payload| - Roast::Support::Monorail.track_command("tool_usage", { - "function_name" => payload[:function_name], - "execution_time" => payload[:execution_time], - "cache_enabled" => payload[:cache_enabled] - }) -end -``` - -See `examples/monorail_initializer.rb` for a complete example of Monorail integration. - -## Best Practices - -1. **Keep initializers focused**: Each initializer should handle a specific concern (logging, metrics, error reporting, etc.) - -2. **Handle errors gracefully**: Wrap your subscriber code in error handling to prevent crashes: - ```ruby - ActiveSupport::Notifications.subscribe("roast.workflow.start") do |name, start, finish, id, payload| - begin - # Your instrumentation code here - rescue => e - $stderr.puts "Instrumentation error: #{e.message}" - end - end - ``` - -3. **Avoid blocking operations**: Instrumentation should be fast and non-blocking. For heavy operations, consider using async processing. - -4. **Use pattern matching**: Subscribe to specific event patterns to reduce overhead: - ```ruby - # Subscribe only to workflow events - ActiveSupport::Notifications.subscribe(/roast\.workflow\./) do |name, start, finish, id, payload| - # Handle only workflow events - end - ``` - -5. **Consider performance impact**: While instrumentation is valuable, too many subscribers can impact performance. Be selective about what you instrument. - -## Testing Your Instrumentation - -You can test your instrumentation by creating a simple workflow and observing the events: - -```yaml -# test_instrumentation.yml -name: instrumentation_test -steps: - - test_step -``` - -Then run: -```bash -roast execute test_instrumentation.yml some_file.rb -``` - -Your instrumentation should capture the workflow start, step execution, and workflow completion events. - -## Available Tools - -Roast provides several built-in tools that you can use in your workflows: - -### WriteFile Tool - -Writes content to a file, creating the file if it doesn't exist or overwriting it if it does. - -```ruby -# Example usage in a prompt -write_file(path: "output.txt", content: "This is the file content") -``` - -### UpdateFiles Tool - -Applies a unified diff/patch to one or more files. Changes are applied atomically when possible. - -```ruby -# Example usage in a prompt -update_files(diff: <<~DIFF, base_path: "/path/to/project", create_files: true) - --- a/file1.txt - +++ b/file1.txt - @@ -1,3 +1,4 @@ - line1 - +new line - line2 - line3 - - --- a/file2.txt - +++ b/file2.txt - @@ -5,7 +5,7 @@ - line5 - line6 - -old line7 - +updated line7 - line8 -DIFF -``` - -This tool is especially useful for making targeted changes to multiple files at once, without having to replace entire file contents. \ No newline at end of file diff --git a/docs/ITERATION_SYNTAX.md b/docs/ITERATION_SYNTAX.md deleted file mode 100644 index 99546af5..00000000 --- a/docs/ITERATION_SYNTAX.md +++ /dev/null @@ -1,147 +0,0 @@ -# Using Iteration with Standardized Syntax - -## Overview - -Roast supports powerful iteration constructs with the `repeat` and `each` workflow steps. These features now support a standardized approach to evaluating expressions using the double-curly braces syntax (`{{...}}`). - -## Syntax Options for Iteration Inputs - -Both `until` conditions (in `repeat`) and collection expressions (in `each`) accept the following formats: - -### 1. Ruby Expressions with `{{...}}` Syntax - -For evaluating Ruby code in the workflow context: - -```yaml -# Repeat until a condition is met -- repeat: - steps: - - process_item - until: "{{output['counter'] >= 5}}" - max_iterations: 10 - -# Iterate over a collection -- each: "{{output['items'].filter { |item| item.active? }}}" - as: "current_item" - steps: - - process_item -``` - -### 2. Bash Commands with `$(...)` Syntax - -For executing shell commands and using their results: - -```yaml -# Repeat until a command succeeds -- repeat: - steps: - - check_service - until: "$(curl -s -o /dev/null -w '%{http_code}' http://service.local/ | grep -q 200)" - max_iterations: 20 - -# Iterate over files returned by a command -- each: "$(find . -name '*.rb' -type f)" - as: "current_file" - steps: - - process_file -``` - -### 3. Step Names (as strings) - -For using the result of another step: - -```yaml -# Repeat until a step returns a truthy value -- repeat: - steps: - - process_batch - until: "check_completion" - max_iterations: 100 - -# Iterate over items returned by a step -- each: "get_pending_items" - as: "pending_item" - steps: - - process_pending_item -``` - -### 4. Prompt Content - -For defining prompts directly in the workflow: - -```yaml -# Using a prompt to determine continuation -- repeat: - steps: - - process_content - until: - prompt: prompts/check_completion.md - model: claude-3-haiku - max_iterations: 10 - -# Using a prompt to generate a collection -- each: - prompt: prompts/generate_test_cases.md - model: claude-3-haiku - as: "test_case" - steps: - - run_test -``` - -## Type Coercion - -### Smart Defaults - -Roast applies intelligent defaults for boolean coercion based on the type of expression: - -- **Ruby expressions** (`{{expr}}`) → Regular boolean coercion (`!!value`) -- **Bash commands** (`$(cmd)`) → Exit code interpretation (0 = true, non-zero = false) -- **Inline prompts/step names** → LLM boolean interpretation (analyzes yes/no intent) - -### Manual Coercion - -You can override the smart defaults by specifying `coerce_to` directly in the step: - -```yaml -# Override prompt to use regular boolean instead of LLM boolean -- repeat: - until: "check_condition" - coerce_to: boolean - steps: - - process_item - -# Force a step result to be treated as iterable -- each: "get_items" - as: "item" - coerce_to: iterable - steps: - - process: "{{item}}" -``` - -Available coercion types: -- `boolean` - Standard Ruby truthiness (`!!` operator) -- `llm_boolean` - Natural language yes/no interpretation -- `iterable` - Convert to array (splits strings on newlines) - -## Migrating Existing Workflows - -If you're updating existing workflows: - -1. For Ruby expressions, wrap them in `{{...}}`: - ```yaml - # Old - until: "output['counter'] >= 5" - - # New - until: "{{output['counter'] >= 5}}" - ``` - -2. Bash commands, step names, and prompts can remain unchanged. - -## Best Practices - -- Use `{{...}}` for all Ruby expressions to make them explicit -- For complex conditions, consider creating a dedicated step that returns a boolean -- For collections, ensure they return iterable objects (arrays, hashes, etc.) -- Always set reasonable `max_iterations` limits on repeat loops -- Use meaningful variable names in `each` loops \ No newline at end of file diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md deleted file mode 100644 index 3cefb1b8..00000000 --- a/docs/VALIDATION.md +++ /dev/null @@ -1,178 +0,0 @@ -# Workflow Validation - -Roast provides comprehensive validation for workflow configurations to catch errors early and improve the development experience. - -## Using the Validation Command - -### Validate a Single Workflow - -```bash -# Validate a specific workflow by path -roast validate path/to/workflow.yml - -# Or by workflow name (assumes roast/ directory structure) -roast validate my_workflow -``` - -### Validate All Workflows - -```bash -# Validate all workflows in the roast/ directory -roast validate -``` - -### Strict Mode - -Use the `--strict` or `-s` flag to treat warnings as errors: - -```bash -roast validate --strict -``` - -## Validation Levels - -### 1. Schema Validation - -Ensures your workflow conforms to the JSON schema: -- Required fields (name, tools, steps) -- Correct data types -- Valid structure for complex steps (if/then, case/when, each, repeat) - -### 2. Dependency Checking - -#### Tool Dependencies -Validates that all declared tools are available: -- Checks for tool module existence -- Supports MCP tool configurations -- Provides helpful suggestions for typos - -#### Step References -Validates that steps referenced in conditions exist: -- Checks `if`, `unless`, and `case` conditions -- Distinguishes between step references and expressions - -#### Resource Dependencies -Validates file resources: -- Warns if target files don't exist (unless using glob patterns) -- Checks for missing prompt files - -### 3. Configuration Linting - -#### Naming Conventions -- Workflows should have descriptive names -- Step names should use snake_case - -#### Complexity Checks -- Warns about workflows with too many steps (>20) -- Detects excessive nesting depth (>5 levels) - - -#### Best Practices -- Warns about missing error handling -- Detects unused tool declarations - -## Error Messages - -The validator provides clear, actionable error messages: - -``` -Workflow validation failed with 2 error(s): - -• Missing required field: 'steps' (Add 'steps' to your workflow configuration) -• Tool 'Roast::Tools::BashCommand' is not available (Did you mean: Roast::Tools::Bash?) -``` - -## Example Workflow - -Here's an example of a well-validated workflow: - -```yaml -name: Data Processing Workflow -tools: - - Roast::Tools::Bash - - Roast::Tools::ReadFile - - Roast::Tools::WriteFile - -# Use inputs for sensitive data -inputs: - - api_token: "Enter your API token" - -# Enable error handling -exit_on_error: true - -steps: - - validate_input - - fetch_data - - process_data - - save_results -``` - -## Integration with CI/CD - -You can integrate workflow validation into your CI/CD pipeline: - -```yaml -# .github/workflows/validate.yml -name: Validate Workflows -on: [push, pull_request] - -jobs: - validate: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: ruby/setup-ruby@v1 - with: - ruby-version: '3.0' - bundler-cache: true - - name: Validate all workflows - run: bundle exec roast validate --strict -``` - -## Programmatic Usage - -You can also use the validator programmatically: - -```ruby -require 'roast' - -yaml_content = File.read('workflow.yml') -validator = Roast::Workflow::Validators::ValidationOrchestrator.new(yaml_content, 'workflow.yml') - -if validator.valid? - puts "Workflow is valid!" - - # Check for warnings - validator.warnings.each do |warning| - puts "Warning: #{warning[:message]}" - puts " → #{warning[:suggestion]}" - end -else - # Handle errors - validator.errors.each do |error| - puts "Error: #{error[:message]}" - puts " → #{error[:suggestion]}" - end -end -``` - -## Architecture - -The validation system follows SOLID principles with a modular design: - -- **ValidationOrchestrator**: Coordinates all validators and aggregates results -- **SchemaValidator**: Handles YAML parsing and JSON schema validation -- **DependencyValidator**: Validates tools, step references, and resources -- **LintingValidator**: Enforces best practices and code quality standards -- **StepCollector**: Provides efficient caching for step traversal - -This architecture makes it easy to extend validation with new rules or customize existing behavior. - -## Future Enhancements - -The validation system is designed to be extensible. Future enhancements may include: - -- Detection of circular dependencies -- Performance analysis and optimization suggestions -- Custom validation rules via plugins -- Integration with language servers for real-time validation \ No newline at end of file