[BUG] ctx.obj not passed to autocompletion function

[DonalChilde]

Describe the bug

I have an expensive object stored in the context obj for use in my sub commands.
I expected to be able to pull that object out of the ctx.obj in my autocompletion function and use it to generate the completion list.
There is a ctx object, but it has no ctx.obj. However, my callbacks that use ctx.obj work as expected.

To Reproduce

Steps to reproduce the behavior with a minimum self-contained file.

Replace each part with your own scenario:

• Create a file main.py with:

from typing import Dict
import typer

app = typer.Typer()
app2 = typer.Typer()
app.add_typer(app2, name="schlock")

RULES = {
    "1": "Pillage, then burn.",
    "2": "A Sergeant in motion outranks a Lieutenant who doesn't know what's going on.",
    "3": "An ordnance technician at a dead run outranks everybody.",
    "4": "Close air support covereth a multitude of sins.",
}


@app.callback()
def load_ctx(ctx: typer.Context):
    ctx.obj = {}
    ctx.obj["rules"] = RULES


@app.command()
def hello(ctx: typer.Context, name: str):
    typer.echo(f"Hi {name}")


def autocomplete_rules(ctx: typer.Context):
    rules: Dict = ctx.obj["rules"]
    comps = list(rules)
    return comps


def callback_check(ctx: typer.Context, value: str):
    rules: Dict = ctx.obj["rules"]
    comps = list(rules)
    if value not in comps:
        raise typer.BadParameter(f"Only 1-4 are allowed. tried: {value}")
    return value


@app2.command()
def says(
    ctx: typer.Context,
    index: str = typer.Argument(
        ...,
        help="Choose 1-4.",
        autocompletion=autocomplete_rules,
        callback=callback_check,
    ),
):
    """Choose a saying."""
    typer.echo(ctx.obj["rules"][index])


if __name__ == "__main__":
    app()

• Call it with:

typer main.py run schlock says [tab][tab]

• It outputs:

File "/home/chad/projects/eve/eve_esi/.venv/lib/python3.9/site-packages/typer/main.py", line 850, in wrapper
    return callback(**use_params)  # type: ignore
  File "src/eve_esi_jobs/typer_cli/typer_bug.py", line 28, in autocomplete_rules
    rules: Dict = ctx.obj["rules"]
TypeError: 'NoneType' object is not subscriptable

• But I expected it to output:

1 2 3 4

Expected behavior

I expected be be able to access the context in autocompletion function.
If you try a value of 5, you can see that the context is available in the callback function.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

• OS: pop_os 20.10
• Typer Version 0.3.2

python -c "import typer; print(typer.__version__)"

• Python version, get it with:

python 3.9.0

Additional context

Add any other context about the problem here.

[flapili]

you can do

ctx = click.get_current_context()

it's not ideal but it works

[DonalChilde]

That’s fantastic, thanks!

Edit:
Unfortunately, this fails with the the error -

File "/home/chad/projects/eve/eve_esi/src/eve_esi_jobs/typer_cli/cli_helpers.py", line 189, in completion_op_id
ctx_hack = get_current_context()
File "/home/chad/projects/eve/eve_esi/.venv/lib/python3.9/site-packages/click/globals.py", line 25, in get_current_context
raise RuntimeError("There is no active click context.")
RuntimeError: There is no active click context.

Trying to wrap the autocomplete function with a simple @click.command() also does not work. that gives a different error from typer -

File "/home/chad/projects/eve/eve_esi/.venv/lib/python3.9/site-packages/typer/utils.py", line 9, in get_params_from_function
type_hints = get_type_hints(func)
File "/usr/lib/python3.9/typing.py", line 1377, in get_type_hints
raise TypeError('{!r} is not a module, class, method, '
TypeError: is not a module, class, method, or function.

But it was certainly worth a try.

[sathoune]

It did not work because callback is not invoked during autocompletion.

def autocomplete_rules(ctx: typer.Context):
    load_ctx(ctx)
    rules: Dict = ctx.obj["rules"]
    comps = list(rules)
    return comps

with this, I am not getting an error but numbers as expected.

[STKFLT]

I have created a typer script that uses global options implemented via the callback feature. (see https://typer.tiangolo.com/tutorial/commands/callback/ + #42)

I tried to implement an autocomplete functionality for a sub-command but the callback function is never triggered and thus the autocomplete function crashes when accessing ctx.obj['my_expected_data']

The solution from @captaincapitalism will not work in this case because the global parameters will not be parsed or populated unless typer invokes the callback within the context

here is some code that replicates the problem:

#!/usr/bin/env python3
import typer


app = typer.Typer()


@app.callback()
def common(ctx: typer.Context, extra_color: str = typer.Option("yellow", help="some global param")):
    ctx.obj = {"extra_color": extra_color}


def color_autocompleter(ctx: typer.Context, incomplete: str):
    for color in ['red', 'green', 'blue', ctx.obj['extra_color']]:
        if color.startswith(incomplete):
            yield color


@app.command('my_command')
def my_command(ctx: typer.Context, color: str = typer.Option(default=None, autocompletion=color_autocompleter)):
    typer.echo(f"{color=}")


if __name__ == "__main__":
    app()

trigger the issue with:

./typer_test.py --install-completion
./typer_test.py --extra-color orange my_command --color <TAB><TAB>

I traced the problem into click and found a solution, but I don't have the bandwidth right now to figure out it if I'm actually doing things right according to click internals and submit a PR, so use this terrible monkey-patch at your own risk:

# monkey-patch click to call callbacks when resolving context for autocompletion
# allows global parameters (as implemented in typer) to be parsed and modifications to "ctx.obj" to be accessible within typer autocompletion functions 
# this probably sucks and is probably the wrong way to do this. This has only been tested with bash and has not been tested AT ALL with chain commands, hopefully it all works.
# Copied directly from click source, license here: https://github.com/pallets/click/blob/main/LICENSE.rst
"""
Copyright 2014 Pallets

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

    Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
    Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
    Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""

from click import shell_completion
from click.shell_completion import t
from click import BaseCommand, Context, MultiCommand


def _resolve_context(
    cli: BaseCommand, ctx_args: t.Dict[str, t.Any], prog_name: str, args: t.List[str]
) -> Context:
    """Produce the context hierarchy starting with the command and
    traversing the complete arguments. This only follows the commands,
    it doesn't trigger input prompts or ~~callbacks~~. (it does trigger callbacks now)

    :param cli: Command being called.
    :param prog_name: Name of the executable in the shell.
    :param args: List of complete args before the incomplete value.
    """
    ctx_args["resilient_parsing"] = True
    ctx = cli.make_context(prog_name, args.copy(), **ctx_args)
    args = ctx.protected_args + ctx.args

    while args:
        command = ctx.command

        if isinstance(command, MultiCommand):
            if not command.chain:
                name, cmd, args = command.resolve_command(ctx, args)

                if cmd is None:
                    return ctx

                if command.callback is not None:
                    # we are a group command, invoke so global options are parsed
                    ctx.invoke(command.callback, **ctx.params)
                ctx = cmd.make_context(name, args, parent=ctx, resilient_parsing=True)
                args = ctx.protected_args + ctx.args
            else:
                while args:
                    name, cmd, args = command.resolve_command(ctx, args)

                    if cmd is None:
                        return ctx

                    if command.callback is not None:
                        # we are a group command, invoke so global options are parsed
                        ctx.invoke(command.callback, **ctx.params)
                    sub_ctx = cmd.make_context(
                        name,
                        args,
                        parent=ctx,
                        allow_extra_args=True,
                        allow_interspersed_args=False,
                        resilient_parsing=True,
                    )
                    args = sub_ctx.args

                ctx = sub_ctx
                args = [*sub_ctx.protected_args, *sub_ctx.args]
        else:
            break

    return ctx


shell_completion._resolve_context = _resolve_context

[alexreg]

Not convinced this is a bug. Does it indicate anywhere that the app's callback function should run even on shell completion?

[STKFLT]

@alexreg I don't necessarily disagree. I'm not sure how callback functions on apps with subcommands are used by all of Typer's users and its probable that having it trigger on autocomplete would break some code bases. In fact in the click "resolve_context()" docstring its clear that it isn't supposed to call the callback.

For my use case it feels like a bug since the recommended/only way to introduce global flags in Typer is through the use of an app callback. Click uses a similar method to add global flags (see "The Root Command" section: https://click.palletsprojects.com/en/8.1.x/complex/#building-a-git) and would theoretically have the same issue if you tried to access something defined in the context within an autocomplete function.

Obviously my example was contrived but my actual use case is passing the hostname of an API server that my CLI app interacts with. The autocomplete function needs to know that hostname in order to collect the possible completion options from the API.

The simplest solution in my view would be adding a second callback option for handling app-level parameters that would be triggered in both shell completion and regular execution. But this seems like a pretty niche feature and one that would have to be implemented in click before it gets integrated into Typer.

I mainly just commented here so anyone else who runs into this doesn't have to stumble down the same rabbit-hole.

[alexreg]

@STKFLT Yeah, that's a fair point. So, while this may not be a bug, it is nevertheless perhaps an oversight, for the reasons you give.

The issue is one of design, and lies with Click (not Typer) at the end of the day. I know at least one other CLI library — for another language — that does shell completion basically by performing command-line processing as usual, except that it puts a flag on the context object, so an early exit can be made when performing shell completion. That has the advantage of solving your use case (and even more general ones), but of course doesn't separate concerns quite so well.

[zmarffy]

Obviously my example was contrived but my actual use case is passing the hostname of an API server that my CLI app interacts with. The autocomplete function needs to know that hostname in order to collect the possible completion options from the API.

This is not niche. I am working on something myself that has this same exact use case. Maybe we just want to not allow that to be passed as an argument but instead only read it from an environment variable.