Accessing a table stored in AWS S3Tables

[fstephany]

I'm trying to access a table stored in a S3Table bucket.

Here's how I try:

let cfg = aws_config::from_env()
    .profile_name("<aws-profile>")
    .load()
    .await;
let s3_tables_client = Client::new(&cfg);
let catalog = S3TablesCatalogBuilder::default()
    .with_client(s3_tables_client)
    .load(
        "s3tables_catalog",
        HashMap::from([
            (
                S3TABLES_CATALOG_PROP_TABLE_BUCKET_ARN.to_string(),
                "arn:aws:s3tables:eu-central-1:<account-id>:bucket/<bucket-identifier>".to_string(),
            ),
            ("region_name".to_string(), "eu-central-1".to_string()),
            ("profile_name".to_string(), "<aws-profile>".to_string()),
        ]),
    )
    .await?;

println!("Loading Table");

let ns_id = NamespaceIdent::new("<namespace>".to_owned());
println!(
    "<namespace> namespace exists: {}",
    catalog.namespace_exists(&ns_id).await?
);

let lf_identifier = TableIdent::new(ns_id.clone(), "<table_id>".to_owned());

match catalog.load_table(&lf_identifier).await {
    Ok(table) => println!("Table: {table:?}"),
    Err(err) => println!("Could not get table: {err:?}"),
};

When running this. I get the correct output for the list of tables in the namespace but execution when calling load_table with the following:

Could not get table: Unexpected => Failure in doing io operation
Source: ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.

If I run the script with the env var AWS_PROFILE set to the same profile than the one specified when creating the client, I get the following error:

Could not get table: Unexpected => Failure in doing io operation
Source: Unexpected (persistent) at read, context: { called: reqsign::LoadCredential, service: s3, path: metadata/.json, range: 0- } => loading credential to sign http request, source: error sending request for url (http://169.254.169.254/latest/api/token): error sending request for url (http://169.254.169.254/latest/api/token): operation timed out

The profile uses AWS SSO and a session is active. It looks like the AWS authentication from the AWS SDK is not passed upon the internals of iceberg-rust?

Any pointer on how to get this working?

[blackmwk]

For the first error, I think you didn't add config the storage, see

iceberg-rust/crates/iceberg/src/io/storage/config/s3.rs

Line 31 in 7c2d5d1

pub const S3_ENDPOINT: &str = "s3.endpoint";

I'm not sure about the second part, but it seems you have a network issue?

[fstephany]

Thanks for the pointer!

I've tried to add the endpoint when creating the catalog:

let catalog = S3TablesCatalogBuilder::default()
    .with_client(s3_tables_client)
    .load(
        "s3tables_catalog",
        HashMap::from([
            (
                S3TABLES_CATALOG_PROP_TABLE_BUCKET_ARN.to_string(),
                "arn:aws:s3tables:eu-central-1:<account_id>:bucket/<bucket-name>".to_string(),
            ),
            ("region_name".to_string(), "eu-central-1".to_string()),
            ("profile_name".to_string(), "<aws-profile>".to_string()),
            ("s3tables.region".to_string(), "eu-central-1".to_string()),
            ("s3.region".to_string(), "eu-central-1".to_string()),
            (
                "s3.endpoint".to_string(),
                "s3tables.eu-central-1.api.aws".to_string(),
            ),
        ]),
    )
    .await?;

The error is now the same as if I passed the the profile from an Environment variable:

Could not get table: Unexpected => Failure in doing io operation

Source: Unexpected (persistent) at read, context: { called: reqsign::LoadCredential, service: s3, path: metadata/<some-kind-of-id>.metadata.json, range: 0- } => loading credential to sign http request, source: error sending request for url (http://169.254.169.254/latest/api/token): error sending request for url (http://169.254.169.254/latest/api/token): operation timed out

What I don't understand is the relation between the aws_sdk_s3tables::Client used in the S3TablesCatalogBuilder and the S3 client that is used later. It looks like it does not share the aws_config::SdkConfig and rebuilds an S3 client from scratch?

Is there a working example of iceberg-rust using the AWS S3Tables native catalog? It's all very confusing for me at the moment 😅