feat: add colorized option for highstate output

[mdschmitt]

What does this PR do?

Adds optional colorization for diff chunks in the highstate output module.

disclaimer: I used AI to assist me in creating and testing this code, but it works appropriately as far as I can tell. The original version I had wasn't nearly as clean and smooth as this ended up.

I had some trouble getting all the pre-commit hooks to play nice on my Fedora laptop but I'm fairly sure that a workflow run will succeed. I'll obviously make changes if that's not the case.

What issues does this PR fix or reference?

I didn't submit an issue for this; I did however mention it a while back in Discord.
This is me attempting to upstream the polished final result.

Previous Behavior

All diff output in highstate output module was green regardless of additions/deletions

New Behavior

If specified, the full_color, changes_color and such output modes will show a colorized diff in the state return output (such as for file.managed). The results are colorized like so:

• Added lines are green
• Removed lines are red
• Hunk headers are cyan
• Surrounding context lines are standard white/gray.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

No, but signed with ssh key (so it looks like GitHub itself re-signed it with gpg key ID B5690EEEBB952194)

[twangboy]

Thanks for putting this together. Having properly colorized unified diffs in the highstate output is a massive quality-of-life improvement for troubleshooting long state files.

However, looking closely at the implementation in salt/output/highstate.py, there are a few architectural gaps and potential regressions that we should address before this gets merged:

1. Re-inventing the Wheel on Color Customization
   Inside both _render_diff and _render_changes_dict, the code completely re-initializes the color engine:

colors = salt.utils.color.get_colors(__opts__.get("color"), __opts__.get("color_theme"))

Because the highstate outputter runs these utility functions iteratively across every single state return block, pulling the entire terminal ANSI mapping from scratch adds unnecessary performance overhead (especially on large runs with hundreds of states or multiple minions).

The Fix: The main entry point _format_host() already instantiates the colors dictionary once at startup. We should simply pass that existing colors dictionary down into these utility helper functions as an argument.

2. Bypassing the Native nested Outputter
   The introduction of a custom recursive dictionary layout engine (_render_changes_dict) bypasses Salt's native loader:

# Does not go through the Salt loader, so nested_indent is guaranteed to apply correctly 
regardless of Salt version.

Bypassing the standard nested outputter breaks standard user configuration setups (like custom nested_indent preferences, specific object serialization, and custom output templates). We should leverage Salt's pluggable architecture rather than duplicating a hardcoded string-builder that isolates the colorizer from future core output modifications.

3. Mutating Mutable Objects Mid-Stream
   Using changes.pop("diff") and then putting it back alters the live data structure during output formatting. If downstream asynchronous hooks or master-side returners still hold a reference to that changes object down the line, temporary keys mutations can cause race conditions or obscure bugs.

Suggested Refactor Strategy
Instead of attempting to manually format the entire dictionary, a cleaner and safer approach is to use a surgical regex replacement wrapper. We can let Salt's native nested outputter format the dictionary structure exactly as it always does, and then selectively find and colorize just the text within the diff: block.

Here is how you can refactor it cleanly:

1. Keep _render_diff focused (passing colors explicitly):

def _render_diff(diff_str, indent, colors):
    """
    Render a unified diff string with per-line ANSI colorization.
    """
    prefix = " " * indent
    if __opts__.get("color") is False:
        return "\n".join(prefix + line for line in diff_str.splitlines())

    GREEN = str(colors["GREEN"])
    ENDC = str(colors["ENDC"])
    RED = str(colors["RED"])
    CYAN = str(colors["CYAN"])
    WHITE = str(colors["LIGHT_GRAY"])
    LIGHT_RED = str(colors["LIGHT_RED"])
    
    result = []
    for line in diff_str.splitlines():
        if line.startswith("---"):
            color = LIGHT_RED
        elif line.startswith("+++"):
            color = GREEN
        elif line.startswith("@@"):
            color = CYAN
        elif line.startswith("+"):
            color = GREEN
        elif line.startswith("-"):
            color = RED
        else:
            color = WHITE
        result.append(f"{prefix}{color}{line}{ENDC}")
    return "\n".join(result)

3. Rewrite _nested_changes_colorized as a Wrapper:

By running the standard nested outputter first, we capture the baseline format. We then parse out the exact block that belongs to the string-based diff using a regex pattern that respects whatever indentation depth was generated:

def _nested_changes_colorized(changes, colors):
    """
    Print the changes data with diff colorization by intercepting the standard
    nested outputter payload.
    """
    # 1. Get the baseline structure from the actual nested outputter
    nested_output = salt.output.out_format(changes, "nested", __opts__, nested_indent=14)
    
    # If there's no diff string to format, return it standard
    if not isinstance(changes, dict) or not isinstance(changes.get("diff"), str):
        return "\n" + nested_output

    # 2. Extract the raw diff block to figure out its runtime indentation dynamically
    # This matches 'diff:' and captures everything up until the next unindented sibling key
    diff_pattern = re.compile(r"^(\s*)diff:\n([\s\S]*?)(?=(?:^\s*[\w\d_.-]+:|\Z))", re.MULTILINE)
    match = diff_pattern.search(nested_output)
    
    if match:
        key_indent_str, raw_diff_content = match.groups()
        key_indent = len(key_indent_str)
        val_indent = key_indent + 4
        
        # Colorize just the diff payload line-by-line
        colorized_diff = _render_diff(changes["diff"], val_indent, colors)
        
        # Reconstruct the block with the colorized diff string
        CYAN = str(colors["CYAN"])
        ENDC = str(colors["ENDC"])
        colorized_block = f"{key_indent_str}{CYAN}diff:{ENDC}\n{colorized_diff}\n"
        
        # Swap the plain text block out for the colorized block safely
        nested_output = diff_pattern.sub(colorized_block, nested_output).rstrip()

    return "\n" + nested_output

4. Update _format_changes Calls:
   Simply pass the existing colors mapping down when calling the colorized wrapper:

if colorize:
    return True, _nested_changes_colorized(changes, colors)

This keeps the engine highly performant, preserves standard loader behaviors for the structural output configuration, and eliminates mutable dictionary state manipulations. Let me know what you think!

[mdschmitt]

@twangboy I changed a few things around and now it works even with file.recurse diff changes. I kinda don't trust myself with regex-fu so I went with a couple little helpers instead (except the one that helps parse out whitespace for the sub-key in file.recurse so it renders at the right depth). If you don't like it as-is, that's cool and I'll try again using the regex route you suggested.

Can you review and let me know what you think of this version?

[mdschmitt]

Shoot, I did another run-through with a few more tweaks, can you take one more look please? My apologies for the force-push, I squashed some disparate commits into the most recent one and hadn't realized you already approved.