Error while parsing environment variables using generic Secret with type parameters

[DashaVed]

Description

I inherit from the generic Secret to override the behavior of the _display method. I get something like SecretMultiHostUrl[MongoDsn]. Example:

from pydantic import Secret

MultiHostUrlType = TypeVar("MultiHostUrlType", bound=MultiHostUrl)

class SecretMultiHostUrl(Secret[MultiHostUrlType]):
    def __init__(self, url: MultiHostUrlType) -> None:
        super().__init__(url)
        self._display_value = ...

    def _display(self) -> str:
        return str(self._display_value)
        

class SettingsMongo(BaseSettings):  
    mongo_uri: SecretMultiHostUrl[MongoDsn]

I get mongo_uri from the environment variables, but eventually the app crashes with an error:

raise SettingsError(
                    f'error parsing value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
E               pydantic_settings.sources.SettingsError: error parsing value for field "mongo_uri" from source "EnvSettingsSource"

This error occurs because the field is considered complex. In the method _annotation_is_complex(annotation: type[Any] | None, metadata: list[Any])

annotation = __main__.SecretMultiHostUrl[typing.Annotated[pydantic_core._pydantic_core.MultiHostUrl, UrlConstraints(max_length=None, allowed_schemes=['mongodb', 'mongodb+srv'], host_required=None, default_host=None, default_port=27017, default_path=None)]]

origin = <class '__main__.SecretMultiHostUrl'>. At the time of checking for the origin attribute __get_pydantic_core_schema__ I get True. Therefore, the field is considered complex, and at the moment of decoding this field, the code crashes with the error JSONDecodeError.

I would like to know if this is a bug or an expected behavior. Maybe there are standard ways to solve this problem?

I know that in pydantic-settings 2.7.0 there is a NoDecode that solves this problem. I also know that I can provide my own source class in Settings model. But these methods are not suitable in my case.

Traceback

self = EnvSettingsSource(env_nested_delimiter=None, env_prefix_len=0)

    def __call__(self) -> dict[str, Any]:
        data: dict[str, Any] = {}
    
        for field_name, field in self.settings_cls.model_fields.items():
            try:
                field_value, field_key, value_is_complex = self.get_field_value(field, field_name)
            except Exception as e:
                raise SettingsError(
                    f'error getting value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
    
            try:
>               field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

..\venv\Lib\site-packages\pydantic_settings\sources.py:323: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\venv\Lib\site-packages\pydantic_settings\sources.py:513: in prepare_field_value
    raise e
..\venv\Lib\site-packages\pydantic_settings\sources.py:510: in prepare_field_value
    value = self.decode_complex_value(field_name, field, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
..\venv\Lib\site-packages\pydantic_settings\sources.py:147: in decode_complex_value
    return json.loads(value)
           ^^^^^^^^^^^^^^^^^
..\..\..\.pyenv\pyenv-win\versions\3.12.0\Lib\json\__init__.py:346: in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
..\..\..\.pyenv\pyenv-win\versions\3.12.0\Lib\json\decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x0000026DB1D7AE40>
s = 'mongodb://ycrawford:password@email-66.hall.info:27030/situation?replicaSet=myRepl'
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

..\..\..\.pyenv\pyenv-win\versions\3.12.0\Lib\json\decoder.py:355: JSONDecodeError

The above exception was the direct cause of the following exception:

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x0000026DB5BCDFA0>
settings_factory_mongo = <function settings_factory_mongo.<locals>._factory at 0x0000026DB5BDF4C0>
password = 'password'
mongo_uri_with_auth = 'mongodb://ycrawford:password@email-66.hall.info:27030/situation?replicaSet=myRepl'

    def test_secret_mongo_dsn(monkeypatch, settings_factory_mongo, password, mongo_uri_with_auth):
        monkeypatch.setenv('MONGO_URI', mongo_uri_with_auth)
>       settings = settings_factory_mongo()
                   ^^^^^^^^^^^^^^^^^^^^^^^^

test_mongo_uri.py:3: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
conftest.py:39: in _factory
    return SettingsMongo()
           ^^^^^^^^^^^^^^^
..\venv\Lib\site-packages\pydantic_settings\main.py:85: in __init__
    **__pydantic_self__._settings_build_values(
..\venv\Lib\site-packages\pydantic_settings\main.py:187: in _settings_build_values
    return deep_update(*reversed([source() for source in sources]))
                                  ^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = EnvSettingsSource(env_nested_delimiter=None, env_prefix_len=0)

    def __call__(self) -> dict[str, Any]:
        data: dict[str, Any] = {}
    
        for field_name, field in self.settings_cls.model_fields.items():
            try:
                field_value, field_key, value_is_complex = self.get_field_value(field, field_name)
            except Exception as e:
                raise SettingsError(
                    f'error getting value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
    
            try:
                field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
            except ValueError as e:
>               raise SettingsError(
                    f'error parsing value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
E               pydantic_settings.sources.SettingsError: error parsing value for field "mongo_uri" from source "EnvSettingsSource"

..\venv\Lib\site-packages\pydantic_settings\sources.py:325: SettingsError

Python, Pydantic & OS Version

pydantic version: 2.7.0
        pydantic-core version: 2.18.1
          pydantic-core build: profile=release pgo=true
                 install path: ..\venv\Lib\site-packages\pydantic
               python version: 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]
                     platform: Windows-11-10.0.26200-SP0
             related packages: mypy-1.18.2 pydantic-settings-2.2.1 typing_extensions-4.15.0
                       commit: unknown

[hramezani]

Thanks @DashaVed for reporting this issue.

All your explanations are true. pydantic-settings considers the field as a complex field and tries to json decode it.
We are always improving complex field calculations but sometimes it is hard to cover all the cases. That's why we introduced the NoDecode.

BTW, you can improve the implementation if you want, but I am afraid it makes it more complex than the thing that we have right now.

[DashaVed]

Thank you @hramezani for the explanation and for the work you put into improving complex field handling.

I’ll keep your suggestion about improving the implementation in mind and will take a look if I see a practical way to do it without adding too much complexity. Thanks!