Skip to content

when using S3fs with localstack error is thrown about keys #51

@covatic-john

Description

@covatic-john

using enable_local_endpoints()

the following error is thrown when uploading to s3 with pandas and S3FS

024-03-06 11:28:48.862 | ERROR    | tests.integration.test_pipelines:test_analytics:124 - An error has been caught in function 'test_analytics', process 'MainProcess' (9286), thread 'MainThread' (7997807680):
Traceback (most recent call last):

  File "/Users/johnharrison/git/database_flattening/venv/lib/python3.10/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
                 │     │       └ {'Key': 'main/monitoring/analytics/mongo_query_date=2024-03-05 00:00:00/2024-03-05 00:00:00.csv', 'Bucket': 'ds-spectrum-tabl...
                 │     └ ()
                 └ <bound method ClientCreator._create_api_method.<locals>._api_call of <aiobotocore.client.S3 object at 0x14fe17280>>
  File "/Users/johnharrison/git/database_flattening/venv/lib/python3.10/site-packages/aiobotocore/client.py", line 408, in _make_api_call
    raise error_class(parsed_response, operation_name)
          │           │                └ 'PutObject'
          │           └ {'Error': {'Code': 'InvalidAccessKeyId', 'Message': 'The AWS Access Key Id you provided does not exist in our records.', 'AWS...
          └ <class 'botocore.exceptions.ClientError'>

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.


The above exception was the direct cause of the following exception:

sample code to test this

isna__sum.to_csv( f's3://tables/main/monitoring/analytics/{start_date}.csv')

error seems to be coming from aiobotocore

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions