can't write to onelake using the latest nightly build

[djouallah]

Apache Iceberg version

None

Please describe the bug 🐞

using this code

!pip install --extra-index-url https://test.pypi.org/simple/ pyiceberg==0.10.0.dev20250813003016

from pyiceberg.catalog.sql import SqlCatalog
import duckdb
def connect_catalog():
      catalog = SqlCatalog(
      "default",
      **{
          "uri": "sqlite:///:memory:",
          "adls.account-name"         : 'onelake' ,
          "adls.client-id"            : client_id,
          "adls.tenant-id"            : tenant_id,
          "adls.client-secret"        : client_secret,
          "legacy-current-snapshot-id": True ,
          "warehouse"                 : "abfss://iceberg@onelake.dfs.fabric.microsoft.com/external.Lakehouse/Files"
      },
                        )
      return catalog
catalog = connect_catalog()
db = "aemo"
catalog.create_namespace_if_not_exists(db)
tbl = db+"."+"calendar"
if not catalog.table_exists(tbl):
  df=duckdb.sql(""" SELECT cast(unnest(generate_series(cast ('2018-04-01' as date), cast('2024-12-31' as date), interval 1 day)) as date) as date,
            EXTRACT(year from date) as year,
            EXTRACT(month from date) as month
            """).arrow()
  catalog.create_table(tbl,schema=df.schema)
  catalog.load_table(tbl).overwrite(df)
  print('calendar created')
else:
    print("table exist already")

I am getting this error

ClientSecretCredential.get_token_info failed: The current credential is not configured to acquire tokens for tenant xxxxxxxxxxxxxxxxxxxxxxxxxx. To enable acquiring tokens for this tenant add it to the additionally_allowed_tenants when creating the credential, or add "*" to additionally_allowed_tenants to allow acquiring tokens for any tenant.

btw, this works fine and I got a token

from azure.identity import ClientSecretCredential

# Create credential with specific values
credential = ClientSecretCredential(
    tenant_id=tenant_id,
    client_id=client_id,
    client_secret=client_secret
)

# Get token for Azure Storage
token = credential.get_token("https://storage.azure.com/.default").token

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Fokko]

@djouallah Thanks for raising this. We're very close on the 0.10 release and I would love to include this fix. Could you share a full stacktrace to help me identify where it blows up? For example, it is unclear to me which FileIO you're using. Thanks!

[djouallah]

---------------------------------------------------------------------------
ClientAuthenticationError                 Traceback (most recent call last)
Cell In[3], line 26
     21 if not catalog.table_exists(tbl):
     22   df=duckdb.sql(""" SELECT cast(unnest(generate_series(cast ('2018-04-01' as date), cast('2024-12-31' as date), interval 1 day)) as date) as date,
     23             EXTRACT(year from date) as year,
     24             EXTRACT(month from date) as month
     25             """).arrow()
---> 26   catalog.create_table(tbl,schema=df.schema)
     27   catalog.load_table(tbl).overwrite(df)
     28   print('calendar created')

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pyiceberg\catalog\sql.py:218, in SqlCatalog.create_table(self, identifier, schema, location, partition_spec, sort_order, properties)
    214 metadata = new_table_metadata(
    215     location=location, schema=schema, partition_spec=partition_spec, sort_order=sort_order, properties=properties
    216 )
    217 io = load_file_io(properties=self.properties, location=metadata_location)
--> 218 self._write_metadata(metadata, io, metadata_location)
    220 with Session(self.engine) as session:
    221     try:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pyiceberg\catalog\__init__.py:966, in MetastoreCatalog._write_metadata(metadata, io, metadata_path)
    964 @staticmethod
    965 def _write_metadata(metadata: TableMetadata, io: FileIO, metadata_path: str) -> None:
--> 966     ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pyiceberg\serializers.py:130, in ToOutputFile.table_metadata(metadata, output_file, overwrite)
    122 @staticmethod
    123 def table_metadata(metadata: TableMetadata, output_file: OutputFile, overwrite: bool = False) -> None:
    124     """Write a TableMetadata instance to an output file.
    125 
    126     Args:
    127         output_file (OutputFile): A custom implementation of the iceberg.io.file.OutputFile abstract base class.
    128         overwrite (bool): Where to overwrite the file if it already exists. Defaults to `False`.
    129     """
--> 130     with output_file.create(overwrite=overwrite) as output_stream:
    131         # We need to serialize None values, in order to dump `None` current-snapshot-id as `-1`
    132         exclude_none = False if Config().get_bool("legacy-current-snapshot-id") else True
    134         json_bytes = metadata.model_dump_json(exclude_none=exclude_none).encode(UTF8)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pyiceberg\io\fsspec.py:328, in FsspecOutputFile.create(self, overwrite)
    310 def create(self, overwrite: bool = False) -> OutputStream:
    311     """Create an output stream for reading the contents of the file.
    312 
    313     Args:
   (...)    326         behavior will truncate the contents of the existing file when opening the output stream.
    327     """
--> 328     if not overwrite and self.exists():
    329         raise FileExistsError(f"Cannot create file, file already exists: {self.location}")
    330     return self._fs.open(self.location, "wb")

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pyiceberg\io\fsspec.py:308, in FsspecOutputFile.exists(self)
    306 def exists(self) -> bool:
    307     """Check whether the location exists."""
--> 308     return self._fs.lexists(self.location)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fsspec\spec.py:669, in AbstractFileSystem.lexists(self, path, **kwargs)
    666 def lexists(self, path, **kwargs):
    667     """If there is a file at the given path (including
    668     broken links)"""
--> 669     return self.exists(path)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\adlfs\spec.py:1356, in AzureBlobFileSystem.exists(self, path)
   1355 def exists(self, path):
-> 1356     return sync(self.loop, self._exists, path)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fsspec\asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fsspec\asyn.py:56, in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\adlfs\spec.py:1379, in AzureBlobFileSystem._exists(self, path)
   1377 async with self.service_client.get_blob_client(container_name, path) as bc:
   1378     try:
-> 1379         if await bc.exists(version_id=version_id):
   1380             return True
   1381     except HttpResponseError:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\tracing\decorator_async.py:119, in distributed_trace_async.<locals>.decorator.<locals>.wrapper_use_tracer(*args, **kwargs)
    117 # If tracing is disabled globally and user didn't explicitly enable it, don't trace.
    118 if user_enabled is False or (not tracing_enabled and user_enabled is None):
--> 119     return await func(*args, **kwargs)
    121 # Merge span is parameter is set, but only if no explicit parent are passed
    122 if merge_span and not passed_in_parent:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\aio\_blob_client_async.py:1050, in BlobClient.exists(self, **kwargs)
   1048 except HttpResponseError as error:
   1049     try:
-> 1050         process_storage_error(error)
   1051     except ResourceNotFoundError:
   1052         return False

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\_shared\response_handlers.py:94, in process_storage_error(storage_error)
     89     storage_error.message = (
     90         storage_error.message
     91         + ". This is likely due to an invalid shared key. Please check your shared key and try again."
     92     )
     93 if not storage_error.response or storage_error.response.status_code in [200, 204]:
---> 94     raise storage_error
     95 # If it is one of those three then it has been serialized prior by the generated layer.
     96 if isinstance(
     97     storage_error,
     98     (PartialBatchErrorException, ClientAuthenticationError, ResourceNotFoundError, ResourceExistsError),
     99 ):

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\aio\_blob_client_async.py:1040, in BlobClient.exists(self, **kwargs)
   1038 version_id = get_version_id(self.version_id, kwargs)
   1039 try:
-> 1040     await self._client.blob.get_properties(
   1041         snapshot=self.snapshot,
   1042         version_id=version_id,
   1043         **kwargs)
   1044     return True
   1045 # Encrypted with CPK

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\tracing\decorator_async.py:119, in distributed_trace_async.<locals>.decorator.<locals>.wrapper_use_tracer(*args, **kwargs)
    117 # If tracing is disabled globally and user didn't explicitly enable it, don't trace.
    118 if user_enabled is False or (not tracing_enabled and user_enabled is None):
--> 119     return await func(*args, **kwargs)
    121 # Merge span is parameter is set, but only if no explicit parent are passed
    122 if merge_span and not passed_in_parent:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\_generated\aio\operations\_blob_operations.py:506, in BlobOperations.get_properties(self, snapshot, version_id, timeout, request_id_parameter, lease_access_conditions, cpk_info, modified_access_conditions, **kwargs)
    503 _request.url = self._client.format_url(_request.url)
    505 _stream = False
--> 506 pipeline_response: PipelineResponse = await self._client._pipeline.run(  # pylint: disable=protected-access
    507     _request, stream=_stream, **kwargs
    508 )
    510 response = pipeline_response.http_response
    512 if response.status_code not in [200]:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:229, in AsyncPipeline.run(self, request, **kwargs)
    227 pipeline_request = PipelineRequest(request, context)
    228 first_node = self._impl_policies[0] if self._impl_policies else _AsyncTransportRunner(self._transport)
--> 229 return await first_node.send(pipeline_request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

    [... skipping similar frames: _SansIOAsyncHTTPPolicyRunner.send at line 77 (2 times)]

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\policies\_redirect_async.py:76, in AsyncRedirectPolicy.send(self, request)
     74 original_domain = get_domain(request.http_request.url) if redirect_settings["allow"] else None
     75 while redirects_remaining:
---> 76     response = await self.next.send(request)
     77     redirect_location = self.get_redirect_location(response)
     78     if redirect_location and redirect_settings["allow"]:

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\_shared\policies_async.py:145, in AsyncStorageRetryPolicy.send(self, request)
    143             await self.sleep(retry_settings, request.context.transport)
    144             continue
--> 145         raise err
    146 if retry_settings["history"]:
    147     response.context["history"] = retry_settings["history"]

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\_shared\policies_async.py:125, in AsyncStorageRetryPolicy.send(self, request)
    123 while retries_remaining:
    124     try:
--> 125         response = await self.next.send(request)
    126         if is_retry(response, retry_settings["mode"]) or await is_checksum_retry(response):
    127             retries_remaining = self.increment(
    128                 retry_settings, request=request.http_request, response=response.http_response
    129             )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_base_async.py:77, in _SansIOAsyncHTTPPolicyRunner.send(self, request)
     75 response: PipelineResponse[HTTPRequestType, AsyncHTTPResponseType]
     76 try:
---> 77     response = await self.next.send(request)
     78 except Exception:
     79     await _await_result(self._policy.on_exception, request)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\policies\_authentication_async.py:113, in AsyncBearerTokenCredentialPolicy.send(self, request)
    111 self._token = None  # any cached token is invalid
    112 if "WWW-Authenticate" in response.http_response.headers:
--> 113     request_authorized = await self.on_challenge(request, response)
    114     if request_authorized:
    115         # if we receive a challenge response, we retrieve a new token
    116         # which matches the new target. In this case, we don't want to remove
    117         # token from the request so clear the 'insecure_domain_change' tag
    118         request.context.options.pop("insecure_domain_change", False)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\storage\blob\_shared\policies_async.py:283, in AsyncStorageBearerTokenCredentialPolicy.on_challenge(self, request, response)
    280     return False
    282 scope = challenge.resource_id + DEFAULT_OAUTH_SCOPE
--> 283 await self.authorize_request(request, scope, tenant_id=challenge.tenant_id)
    285 return True

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\policies\_authentication_async.py:87, in AsyncBearerTokenCredentialPolicy.authorize_request(self, request, *scopes, **kwargs)
     77 """Acquire a token from the credential and authorize the request with it.
     78 
     79 Keyword arguments are passed to the credential's get_token method. The token will be cached and used to
   (...)     83 :param str scopes: required scopes of authentication
     84 """
     86 async with self._lock:
---> 87     await self._request_token(*scopes, **kwargs)
     88 bearer_token = cast(Union[AccessToken, AccessTokenInfo], self._token).token
     89 request.http_request.headers["Authorization"] = "Bearer " + bearer_token

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\policies\_authentication_async.py:216, in AsyncBearerTokenCredentialPolicy._request_token(self, *scopes, **kwargs)
    209 async def _request_token(self, *scopes: str, **kwargs: Any) -> None:
    210     """Request a new token from the credential.
    211 
    212     This will call the credential's appropriate method to get a token and store it in the policy.
    213 
    214     :param str scopes: The type of access needed.
    215     """
--> 216     self._token = await self._get_token(*scopes, **kwargs)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\policies\_authentication_async.py:198, in AsyncBearerTokenCredentialPolicy._get_token(self, *scopes, **kwargs)
    195         if key in TokenRequestOptions.__annotations__:  # pylint: disable=no-member
    196             options[key] = kwargs.pop(key)  # type: ignore[literal-required]
--> 198     return await await_result(
    199         cast(AsyncSupportsTokenInfo, self._credential).get_token_info,
    200         *scopes,
    201         options=options,
    202     )
    203 return await await_result(
    204     cast(AsyncTokenCredential, self._credential).get_token,
    205     *scopes,
    206     **kwargs,
    207 )

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\core\pipeline\_tools_async.py:56, in await_result(func, *args, **kwargs)
     54 result = func(*args, **kwargs)
     55 if isinstance(result, Awaitable):
---> 56     return await result
     57 return result

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\identity\aio\_internal\get_token_mixin.py:114, in GetTokenMixin.get_token_info(self, options, *scopes)
     96 async def get_token_info(self, *scopes: str, options: Optional[TokenRequestOptions] = None) -> AccessTokenInfo:
     97     """Request an access token for `scopes`.
     98 
     99     This is an alternative to `get_token` to enable certain scenarios that require additional properties
   (...)    112         attribute gives a reason.
    113     """
--> 114     return await self._get_token_base(*scopes, options=options, base_method_name="get_token_info")

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\identity\aio\_internal\get_token_mixin.py:132, in GetTokenMixin._get_token_base(self, options, base_method_name, *scopes, **kwargs)
    129 enable_cae = options.get("enable_cae", False)
    131 try:
--> 132     token = await self._acquire_token_silently(
    133         *scopes, claims=claims, tenant_id=tenant_id, enable_cae=enable_cae, **kwargs
    134     )
    135     if not token:
    136         self._last_request_time = int(time.time())

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\identity\aio\_credentials\client_secret.py:64, in ClientSecretCredential._acquire_token_silently(self, *scopes, **kwargs)
     63 async def _acquire_token_silently(self, *scopes: str, **kwargs: Any) -> Optional[AccessTokenInfo]:
---> 64     return self._client.get_cached_access_token(scopes, **kwargs)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\identity\_internal\aad_client_base.py:86, in AadClientBase.get_cached_access_token(self, scopes, **kwargs)
     85 def get_cached_access_token(self, scopes: Iterable[str], **kwargs: Any) -> Optional[AccessTokenInfo]:
---> 86     tenant = resolve_tenant(
     87         self._tenant_id, additionally_allowed_tenants=self._additionally_allowed_tenants, **kwargs
     88     )
     90     cache = self._get_cache(**kwargs)
     91     for token in cache.search(
     92         TokenCache.CredentialType.ACCESS_TOKEN,
     93         target=list(scopes),
     94         query={"client_id": self._client_id, "realm": tenant},
     95     ):

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\azure\identity\_internal\utils.py:130, in resolve_tenant(default_tenant, tenant_id, additionally_allowed_tenants, **_)
    122     _LOGGER.info(
    123         "A token was requested for a different tenant than was configured on the credential, "
    124         "and the requested tenant ID was used to authenticate. Configured tenant ID: %s, "
   (...)    127         tenant_id,
    128     )
    129     return tenant_id
--> 130 raise ClientAuthenticationError(
    131     message="The current credential is not configured to acquire tokens for tenant {}. "
    132     "To enable acquiring tokens for this tenant add it to the additionally_allowed_tenants "
    133     'when creating the credential, or add "*" to additionally_allowed_tenants to allow '
    134     "acquiring tokens for any tenant.".format(tenant_id)
    135 )

ClientAuthenticationError: The current credential is not configured to acquire tokens for tenant 16b3c013-d300-468d-ac64-7eda0820b6d3. To enable acquiring tokens for this tenant add it to the additionally_allowed_tenants when creating the credential, or add "*" to additionally_allowed_tenants to allow acquiring tokens for any tenant.

[kevinjqliu]

Looks like this fails in AzureBlobFileSystem specifically in the way credential is handled.

i was able to get it working with this:

account_name = "onelake"
account_host = f"{account_name}.blob.fabric.microsoft.com"
credential = DefaultAzureCredential()
{
    "adls.account-name": account_name,
    "adls.account-host": account_host,
    "adls.credential": credential,
}

Use adls.credential to pass credential directly to the AzureBlobFileSystem constructor, this was added in #2299

To work with onelake, we also need to set adls.account-host which was added in #2016. The account-host parameter was only added to AzureBlobFileSystem in this commit which is only available for version >=2024.7.0

You can check which adlfs version youre using by running

import adlfs
print(adlfs.__version__)

I looks like pyiceberg already install adlfs 2024.12.0 by default so the issue is likely just with setting the account_host and credential

could you set those and confirm?

[djouallah]

i can confirm, this works and onelake recognized it as a valid iceberg table :), thank you all !!!