feat(clover): Add ability to manually manage assets as part of clover pipeline #7870
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dependency Review✅ No vulnerabilities or OpenSSF Scorecard issues found.Scanned FilesNone |
… pipeline
This PR introduces a comprehensive framework for defining manually-managed assets in Clover's code generation pipeline. Previously, Clover only supported auto-generated CloudFormation resources from AWS schemas. This enhancement allows developers to define custom resources with complete control over their schemas, functions, and behavior.
## Motivation
Some AWS resources don't map cleanly to CloudFormation's CRUD model:
- **Query resources** that lookup existing infrastructure (e.g., AMI lookups)
- **Validation resources** that check prerequisites without creating resources
- **Custom integrations** that wrap multiple AWS APIs
- **Resources with special behaviors** that need custom qualification or attribute logic
Rather than forcing these into the auto-generation pipeline or maintaining separate codebases, this PR enables defining them inline with provider-specific configuration.
## Architecture
### Core Components
#### 1. **Generic Pipeline Integration** (`src/pipelines/generic/loadExtraAssets.ts`)
A new pipeline step that:
- Loads manually-managed asset schemas from provider configuration
- Transforms them into `ExpandedPkgSpec` using the same pipeline as auto-generated assets
- Applies custom functions (actions, qualifications, attributes, code generation, management)
- Merges them with auto-generated specs for unified processing
**Key Features:**
- **Custom function override**: When custom functions are defined, they completely replace provider defaults
- **Flexible bindings**: Attribute functions can specify custom argument bindings via `attributeBindings` callback
- **Property attachment**: `attachAttributeFunctions` callback allows programmatic attachment of attribute functions to specific properties
- **Metadata customization**: Override display names, colors, categories, and descriptions per resource
#### 2. **Provider Configuration Extension** (`src/pipelines/types.ts`)
Added `extraAssets` configuration to `ProviderConfig`:
```typescript
extraAssets?: {
// Function to load extra asset schemas
loadSchemas: () => Promise<SuperSchema[]> | SuperSchema[];
// Optional: Custom property classification for manually-managed assets
classifyProperties?: (schema: SuperSchema) => OnlyProperties;
// Optional: Map of resource type names to their custom configurations
customFuncs?: Record<string, {
// Metadata overrides
metadata?: {
displayName?: string;
category?: string;
color?: string;
description?: string;
};
// Custom function definitions
actions?: Record<string, FuncSpecInfo & { actionKind: ActionFuncSpecKind }>;
codeGeneration?: Record<string, FuncSpecInfo>;
management?: Record<string, FuncSpecInfo & { handlers: CfHandlerKind[] }>;
qualification?: Record<string, FuncSpecInfo>;
attribute?: Record<string, FuncSpecInfo>;
// Simplified attribute function configuration
// Just specify which property to attach to and which properties to use as inputs
attributeFunctions?: (variant: ExpandedSchemaVariantSpec) => Record<string, {
attachTo: string; // Property name to attach the function to
inputs: string[]; // Domain property names to pass as inputs
}>;
}>;
}
```
#### 3. **Function Factory Enhancement** (`src/pipelines/generic/funcFactories.ts`)
Added `createAttributeFuncs()` to support attribute function generation with flexible binding strategies:
- **Uniform bindings**: Same bindings for all functions
- **Per-function bindings**: Custom bindings per function name
- **Dynamic bindings**: Generator function for complex binding logic
#### 4. **Property Path Utilities** (`src/pipelines/generic/index.ts`)
New helper functions for constructing SI property paths:
```typescript
export const PROP_PATH_SEPARATOR = "\x0B"; // Vertical tab character
export function buildPropPath(parts: string[]): string {
return parts.join(PROP_PATH_SEPARATOR);
}
export function createPropFinder(
variant: ExpandedSchemaVariantSpec,
schemaName?: string
): (name: string) => PropSpec;
```
**Benefits:**
- Single source of truth for the separator character
- Type-safe path construction
- More readable than string template literals
- Easy to find properties with better error messages
### AWS Provider Implementation
#### 5. **AWS Extra Assets Registry** (`src/pipelines/aws/extraAssets.ts`)
Central registry for all AWS manually-managed assets:
```typescript
export const AWS_MANUALLY_MANAGED_ASSETS = {
"AWS::EC2::AMI": ec2Ami,
// Future assets can be added here
};
export function loadAwsExtraAssets(): SuperSchema[];
export function getAwsExtraAssetFuncs(): Record<string, any>;
```
#### 6. **AWS Provider Configuration** (`src/pipelines/aws/provider.ts`)
Integrated extra assets into AWS provider config:
```typescript
export const AWS_PROVIDER_CONFIG: ProviderConfig = {
// ... existing config
extraAssets: {
loadSchemas: loadAwsExtraAssets,
customFuncs: getAwsExtraAssetFuncs(),
},
};
```
#### 7. **AWS Pipeline Integration** (`src/pipelines/aws/pipeline.ts`)
Added `loadExtraAssets()` step in the pipeline:
- Runs **after** standard CloudFormation transformations
- Runs **before** provider-specific overrides
- Runs **before** asset func generation
This ensures manually-managed assets are included in all downstream processing.
## Example Implementation: AWS::EC2::AMI
The first manually-managed asset demonstrates the full capabilities of the framework.
### Directory Structure
```
src/pipelines/aws/manually-managed-assets/AWS::EC2::AMI/
├── schema.ts # Complete resource definition
├── attributes/
│ └── awsEc2AmiQueryImageId.ts # Queries AMI ID from AWS
└── qualifications/
└── qualificationAmiExists.ts # Validates AMI exists
```
### Schema Definition (`schema.ts`)
```typescript
export const schema: CfSchema = {
typeName: "AWS::EC2::AMI",
description: "Amazon Machine Image (AMI) query and validation resource",
properties: {
// Query parameters (domain properties)
ExecutableUsers: { type: "string", ... },
Owners: { type: "string", ... },
UseMostRecent: { type: "boolean", default: true },
Filters: { type: "array", ... },
ImageId: { type: "string", ... },
region: { type: "string", ... },
credential: { type: "string", ... },
},
primaryIdentifier: ["/properties/ImageId"],
};
export const config = {
metadata: {
displayName: "AMI Query",
category: "AWS::EC2",
color: "#FF9900",
},
qualification: {
"Validate AMI Query": {
id: "e8b9a8a41fd88e1a...",
path: "./qualifications/qualificationAmiExists.ts",
backendKind: "jsAttribute",
responseType: "qualification",
},
},
attribute: {
"Query AMI ID": {
id: "6e74c3c417eb8fd0...",
path: "./attributes/awsEc2AmiQueryImageId.ts",
backendKind: "jsAttribute",
responseType: "string",
},
},
// Attribute function configuration - simple and declarative!
attributeFunctions: (_variant: ExpandedSchemaVariantSpec) => {
return {
"Query AMI ID": {
attachTo: "ImageId", // Property to attach the function to
inputs: ["region", "UseMostRecent", "Owners", "Filters", "ExecutableUsers"],
},
// Easy to add more attribute functions!
// "Calculate Timeout": {
// attachTo: "TotalTimeout",
// inputs: ["Config.Timeout", "Config.MaxRetries"], // Supports nested paths!
// },
};
},
};
```
### Key Implementation Details
1. **Simplified Attribute Configuration**: The new `attributeFunctions` callback is dramatically simpler:
- **Before**: ~90 lines of repetitive boilerplate with `attributeBindings` and `attachAttributeFunctions`
- **After**: 7 lines of clear, declarative configuration
- Just specify which property to attach to and which properties to use as inputs
- The system automatically creates function bindings, attaches the function, and sets up property inputs
2. **No Duplication**: Previously, you had to list the same properties twice:
- Once in `attributeBindings` (for function arguments)
- Again in `attachAttributeFunctions` (for property inputs)
- Now you list them once in `attributeFunctions.inputs`
3. **Automatic Property Resolution**: The system automatically:
- Looks up property definitions to get types and uniqueIds
- Creates `FuncArgumentSpec` bindings with correct types
- Builds property paths with the correct separator (`\x0B`)
- Attaches functions to properties with the correct function ID
- Sets up property inputs with all required metadata
4. **Nested Property Support**: Use dot notation to reference nested properties:
- Simple properties: `"region"`, `"ImageId"`
- Nested objects: `"Config.MaxRetries"`, `"Settings.Timeout.Seconds"`
- Array elements: `"Tags.Key"`, `"Filters.Name"`
- The system automatically traverses object hierarchies and arrays
5. **Multiple Functions**: Add as many attribute functions as needed - just add entries to the returned object
6. **Qualification Inputs**: Manually-managed assets get `["domain", "secrets"]` as qualification inputs (vs. default `["domain", "code"]`), allowing access to AWS credentials
## Secret Handling Enhancements
This PR also introduces significant improvements to how secrets are handled in manually-managed assets:
### 1. **Explicit Secret Kind Declaration** (`src/pipelines/aws/schema.ts`)
Added `secretKinds` field to `CfSchema` interface:
```typescript
export interface CfSchema extends SuperSchema {
// ... existing fields
/**
* Explicit mapping of property names to their secret kinds.
* Properties listed here will be treated as secrets.
*
* Example:
* secretKinds: {
* credential: "AWS Credential",
* apiKey: "API Key",
* }
*/
secretKinds?: Record<string, string>;
}
```
**Why this approach?**
- **Explicit over implicit**: No guessing - you declare exactly what's a secret
- **Type-safe**: Properties not in secretKinds stay in domain
- **Flexible**: Different properties can have different secret kinds
- **No breaking changes**: writeOnlyProperties can still be used for non-secret write-only props
### 2. **Automatic Secret Configuration** (`src/pipelines/generic/loadExtraAssets.ts`)
When properties are listed in `secretKinds`:
- Automatically moved to the secrets section
- `widgetKind` set to "Secret"
- `widgetOptions` configured with the specified secret kind
- No manual `configureProperties` code needed
```typescript
// In schema.ts - just declare it
secretKinds: {
credential: "AWS Credential",
}
// The pipeline automatically:
// - Moves credential to secrets section
// - Sets widgetKind: "Secret"
// - Sets widgetOptions: [{ label: "secretKind", value: "AWS Credential" }]
```
### 3. **Secret Kind Extraction in Code Generation** (`src/pipelines/generic/generateAssetFuncs.ts`)
Fixed `generateSecretPropBuilderString` to extract the actual secret kind from widgetOptions:
```typescript
function generateSecretPropBuilderString(prop: ExpandedPropSpec, indent_level: number): string {
// Extract the secretKind from the widget options
const secretKindOption = prop.data?.widgetOptions?.find(
(opt) => opt.label === "secretKind"
);
const secretKind = secretKindOption?.value ?? prop.name;
return (
`new SecretPropBuilder()\n` +
`${indent(indent_level)}.setName("${prop.name}")\n` +
`${indent(indent_level)}.setSecretKind("${secretKind}")\n` +
`${indent(indent_level)}.build()`
);
}
```
**Before**: Always used `prop.name` as secretKind (incorrect)
**After**: Extracts from widgetOptions (correct)
## Schema Customization Features
### 1. **Default Values** (`src/spec/props.ts`)
Schema `default` values are now properly extracted and applied:
```typescript
// In schema
UseMostRecent: {
type: "boolean",
default: true, // ← This is now extracted
}
// Generated code
const UseMostRecentProp = new PropBuilder()
.setDefaultValue(true) // ← Automatically included
.build();
```
### 2. **Custom Documentation Links** (`src/spec/props.ts`)
Support for custom `docLink` field in schema properties:
```typescript
// In schema
ExecutableUsers: {
type: "string",
description: "...",
docLink: "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/...",
} as any,
// Generated code uses the custom link instead of provider default
```
**Why needed?** The AWS provider defaults to CloudFormation URLs, but manually-managed assets may reference different APIs (EC2, IAM, etc.)
### 3. **Custom Array Item Names** (`src/spec/props.ts`)
Support for custom `itemName` field to prevent breaking changes:
```typescript
// In schema
Filters: {
type: "array",
itemName: "Filter", // ← Override default "FiltersItem"
items: { ... }
} as any,
// Generated code
.setEntry(
new PropBuilder()
.setName("Filter") // ← Uses custom name
// ...
)
```
**Why critical?** Changing array item names breaks existing user configurations. This allows preserving backward compatibility.
## Migration Path
Existing auto-generated resources are **completely unaffected**:
- No changes to CloudFormation resource generation
- Default functions remain the same
- Pipeline order ensures compatibility
To add a new manually-managed asset:
1. Create directory: `src/pipelines/{provider}/manually-managed-assets/{ResourceType}/`
2. Define `schema.ts` with schema and config (including `secretKinds` if needed)
3. Implement function files
4. Register in `extraAssets.ts`
5. No changes to pipeline or provider config needed!
## Breaking Change Prevention
The `itemName` feature prevents breaking changes for existing users:
- Array item names (like "Filter" vs "FiltersItem") are now controllable
- Existing configurations using "Filter" will continue to work
- Without this, changing from "Filter" to "FiltersItem" would break all user data
stack72
force-pushed
the
manually-managed-assets
branch
from
67a9c98 to
d91cfb4
Compare
|
Working with Module Index at: https://module-index.systeminit.com AWS - 1 new, 122 changed asset(s)Hetzner - 0 new, 11 changed asset(s)Microsoft - 0 new, 325 changed asset(s) |
Contributor
Author
|
/diff Hetzner::Cloud::Certificates |
|
Working with Module Index at: https://module-index.systeminit.com Diffed Hetzner::Cloud::Certificates with the module index:Replaced value within contents of prop /root/domain/type at data/defaultValue: |
Contributor
|
A wonderful PR! |
aaron-dernley
approved these changes
Nov 28, 2025
sprutton1
reviewed
Dec 9, 2025
Comment on lines
+61
to
+64
| // Load extra assets BEFORE generating asset funcs so they're included | ||
| // Extra assets skip AWS-specific transformations since they're manually defined | ||
| specs = await loadExtraAssets(specs, AWS_PROVIDER_CONFIG); | ||
|
|
Contributor
There was a problem hiding this comment.
I'm a bit torn on this. If you did this right after parse schema, you would get a lot more for free (credential, region, intrinsics, suggestions), but then you would have to override the funcs instead of blatting them out here.
sprutton1
reviewed
Dec 9, 2025
Comment on lines
+70
to
+72
|
|
||
| // Generate asset funcs (schemaVariantDefinition) for ALL assets | ||
| // This must happen after loadExtraAssets so extra assets are included |
Contributor
There was a problem hiding this comment.
I am not sure that a lot of these comments add much value. Claude tends to do this after you chastise it for being silly.
sprutton1
reviewed
Dec 9, 2025
| if ("createOnlyProperties" in schema && "readOnlyProperties" in schema) { | ||
| const cfSchema = schema as CfSchema; | ||
| return { | ||
| createOnly: normalizeOnlyProperties(cfSchema.createOnlyProperties), |
Contributor
There was a problem hiding this comment.
I am confused why this is here. I don't see how it's used. Maybe I'm missing something?
sprutton1
reviewed
Dec 9, 2025
| * SI's property path separator - vertical tab character. | ||
| * Used to construct hierarchical property paths like "root\x0Bdomain\x0BpropertyName" | ||
| */ | ||
| export const PROP_PATH_SEPARATOR = "\x0B"; |
Contributor
There was a problem hiding this comment.
This already exists in spec/sockets.ts. We can move it out of there.
sprutton1
reviewed
Dec 9, 2025
Comment on lines
+67
to
+69
| export function buildPropPath(parts: string[]): string { | ||
| return parts.join(PROP_PATH_SEPARATOR); | ||
| } |
sprutton1
reviewed
Dec 9, 2025
| * const filterName = findProp("Filters.Name"); // Array element property | ||
| * ``` | ||
| */ | ||
| export function createPropFinder( |
Contributor
There was a problem hiding this comment.
This is interesting, but I wonder if findObjectProp() does almost the same thing?
sprutton1
reviewed
Dec 9, 2025
| * @param providerConfig - The provider configuration | ||
| * @returns Combined array of auto-generated and manually-managed asset specs | ||
| */ | ||
| export async function loadExtraAssets( |
Contributor
There was a problem hiding this comment.
a lot of what's here almost feels like it should run as a separate pipeline. There's a lot of code that already exists in other parts of clover, especially around the func generation. Maybe we can reuse more of that?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a comprehensive framework for defining manually-managed assets in Clover's code generation pipeline. Previously, Clover only supported auto-generated CloudFormation resources from AWS schemas. This enhancement allows developers to define custom resources with complete control over their schemas, functions, and behavior.
Motivation
Some AWS resources don't map cleanly to CloudFormation's CRUD model:
Rather than forcing these into the auto-generation pipeline or maintaining separate codebases, this PR enables defining them inline with provider-specific configuration.
Architecture
Core Components
1. Generic Pipeline Integration (
src/pipelines/generic/loadExtraAssets.ts)A new pipeline step that:
ExpandedPkgSpecusing the same pipeline as auto-generated assetsKey Features:
attributeBindingscallbackattachAttributeFunctionscallback allows programmatic attachment of attribute functions to specific properties2. Provider Configuration Extension (
src/pipelines/types.ts)Added
extraAssetsconfiguration toProviderConfig:3. Function Factory Enhancement (
src/pipelines/generic/funcFactories.ts)Added
createAttributeFuncs()to support attribute function generation with flexible binding strategies:4. Property Path Utilities (
src/pipelines/generic/index.ts)New helper functions for constructing SI property paths:
Benefits:
AWS Provider Implementation
5. AWS Extra Assets Registry (
src/pipelines/aws/extraAssets.ts)Central registry for all AWS manually-managed assets:
6. AWS Provider Configuration (
src/pipelines/aws/provider.ts)Integrated extra assets into AWS provider config:
7. AWS Pipeline Integration (
src/pipelines/aws/pipeline.ts)Added
loadExtraAssets()step in the pipeline:This ensures manually-managed assets are included in all downstream processing.
Example Implementation: AWS::EC2::AMI
The first manually-managed asset demonstrates the full capabilities of the framework.
Directory Structure
Schema Definition (
schema.ts)Key Implementation Details
Simplified Attribute Configuration: The new
attributeFunctionscallback is dramatically simpler:attributeBindingsandattachAttributeFunctionsNo Duplication: Previously, you had to list the same properties twice:
attributeBindings(for function arguments)attachAttributeFunctions(for property inputs)attributeFunctions.inputsAutomatic Property Resolution: The system automatically:
FuncArgumentSpecbindings with correct types\x0B)Nested Property Support: Use dot notation to reference nested properties:
"region","ImageId""Config.MaxRetries","Settings.Timeout.Seconds""Tags.Key","Filters.Name"Multiple Functions: Add as many attribute functions as needed - just add entries to the returned object
Qualification Inputs: Manually-managed assets get
["domain", "secrets"]as qualification inputs (vs. default["domain", "code"]), allowing access to AWS credentialsSecret Handling Enhancements
This PR also introduces significant improvements to how secrets are handled in manually-managed assets:
1. Explicit Secret Kind Declaration (
src/pipelines/aws/schema.ts)Added
secretKindsfield toCfSchemainterface:Why this approach?
2. Automatic Secret Configuration (
src/pipelines/generic/loadExtraAssets.ts)When properties are listed in
secretKinds:widgetKindset to "Secret"widgetOptionsconfigured with the specified secret kindconfigurePropertiescode needed3. Secret Kind Extraction in Code Generation (
src/pipelines/generic/generateAssetFuncs.ts)Fixed
generateSecretPropBuilderStringto extract the actual secret kind from widgetOptions:Before: Always used
prop.nameas secretKind (incorrect)After: Extracts from widgetOptions (correct)
Schema Customization Features
1. Default Values (
src/spec/props.ts)Schema
defaultvalues are now properly extracted and applied:2. Custom Documentation Links (
src/spec/props.ts)Support for custom
docLinkfield in schema properties:Why needed? The AWS provider defaults to CloudFormation URLs, but manually-managed assets may reference different APIs (EC2, IAM, etc.)
3. Custom Array Item Names (
src/spec/props.ts)Support for custom
itemNamefield to prevent breaking changes:Why critical? Changing array item names breaks existing user configurations. This allows preserving backward compatibility.
Migration Path
Existing auto-generated resources are completely unaffected:
To add a new manually-managed asset:
src/pipelines/{provider}/manually-managed-assets/{ResourceType}/schema.tswith schema and config (includingsecretKindsif needed)extraAssets.tsBreaking Change Prevention
The
itemNamefeature prevents breaking changes for existing users: