Skip to content

Commit bac4fbe

Browse files
poikilothermpoikilotherm
committed
docs(installation): expand and reorganize S3 CORS documentation #12161
- Clarify CORS requirements for direct uploads and downloads - Add cross-references to Dataverse CORS settings - Provide step-by-step instructions and examples for AWS CLI and Minio - Refactor guidance on ensuring origin compatibility between Dataverse and S3
1 parent ba99839 commit bac4fbe

3 files changed

Lines changed: 73 additions & 23 deletions

File tree

‎doc/sphinx-guides/source/_static/installation/cors/cors.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"CORSRules": [
3+
{
4+
"AllowedOrigins": ["*"],
5+
"AllowedHeaders": ["*"],
6+
"AllowedMethods": ["PUT", "GET"],
7+
"ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Encoding", "Content-Range"]
8+
}
9+
]
10+
}

‎doc/sphinx-guides/source/_static/installation/cors/cors.xml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
3+
<CORSRule>
4+
<AllowedOrigin>*</AllowedOrigin>
5+
<AllowedHeader>*</AllowedHeader>
6+
<AllowedMethod>PUT</AllowedMethod>
7+
<AllowedMethod>GET</AllowedMethod>
8+
<ExposeHeader>ETag</ExposeHeader>
9+
<ExposeHeader>Accept-Ranges</ExposeHeader>
10+
<ExposeHeader>Content-Encoding</ExposeHeader>
11+
<ExposeHeader>Content-Range</ExposeHeader>
12+
</CORSRule>
13+
</CORSConfiguration>

‎doc/sphinx-guides/source/installation/big-data-support.rst‎

Lines changed: 50 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -49,47 +49,74 @@ The following features are disabled when S3 direct upload is enabled.
4949
- Creation of NcML auxiliary files (See :ref:`netcdf-and-hdf5`.)
5050
- Extraction of a geospatial bounding box from NetCDF and HDF5 files (see :ref:`netcdf-and-hdf5`) unless :ref:`dataverse.netcdf.geo-extract-s3-direct-upload` is set to true.
5151

52+
5253
.. _cors-s3-bucket:
5354

5455
Allow CORS for S3 Buckets
5556
~~~~~~~~~~~~~~~~~~~~~~~~~
5657

57-
**IMPORTANT:** One additional step that is required to enable direct uploads via a Dataverse installation and for direct download to work with previewers and direct upload to work with dvwebloader (:ref:`folder-upload`) is to allow cross site (CORS) requests on your S3 store.
58-
The example below shows how to enable CORS rules (to support upload and download) on a bucket using the AWS CLI command line tool. Note that you may want to limit the AllowedOrigins and/or AllowedHeaders further. https://github.com/gdcc/dataverse-previewers/wiki/Using-Previewers-with-download-redirects-from-S3 has some additional information about doing this.
58+
**IMPORTANT:** This additional step of allowing cross-site request to your S3 buckets is required to enable direct uploads via a Dataverse installation, direct download to work with previewers, or direct upload to work with *dvwebloader* (:ref:`folder-upload`).
59+
60+
To successfully enable direct uploads (e.g. :ref:`folder-upload`) or direct downloads (e. g. consumed by previewers), you must both:
61+
* Enable CORS in Dataverse (see :ref:`dataverse.cors`).
62+
* Configure a matching/compatible CORS policy on each S3 bucket (and any CDN/proxy in front of it) that will be used.
5963

60-
Dataverse itself will only emit the necessary ``Access-Control-*`` headers to browsers when CORS has been explicitly enabled via the JVM/MicroProfile setting :ref:`dataverse.cors.origin <dataverse.cors.origin>`. You must both:
64+
**NOTE:** Make sure the bucket's CORS configuration ``AllowedOrigins`` is at least as permissive as the origins you configure in :ref:`dataverse.cors.origin`.
65+
If the bucket allows the wildcard ``*`` but the Dataverse application only allows a subset, the browser will still enforce the more restrictive application response!
6166

62-
* Configure an appropriate ``dataverse.cors.origin`` value (single origin, comma-separated list, or ``*``) on the Dataverse application server; and
63-
* Configure a matching/compatible CORS policy on each S3 bucket (and any CDN/proxy in front of it) that will be used for direct upload or for redirect (download-redirect) operations consumed by previewers.
67+
Detailed information for the most common S3 admin tools around CORS:
6468

65-
If you specify multiple origins in ``dataverse.cors.origin`` Dataverse will echo back the requesting origin (when it matches) and will include ``Vary: Origin`` so that shared caches do not serve one origin's response to another. If you configure ``*`` Dataverse will respond with ``Access-Control-Allow-Origin: *`` (note that browsers will not allow credentialed requests with a wildcard).
69+
- `AWS <https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html>`_
70+
- `Minio mc <https://docs.min.io/enterprise/aistor-object-store/reference/cli/mc-cors>`_
71+
- `s3cmd <https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/215253125/Object+Store+S3+CORS+Policies>`_
6672

67-
Make sure the bucket CORS configuration ``AllowedOrigins`` is at least as permissive as the origins you configure in ``dataverse.cors.origin``. If the bucket allows ``*`` but the Dataverse application only allows a subset, the browser will still enforce the more restrictive application response.
73+
Get Current CORS Policy on Bucket
74+
+++++++++++++++++++++++++++++++++
6875

6976
If you'd like to check the CORS configuration on your bucket before making changes:
7077

71-
``aws s3api get-bucket-cors --bucket <BUCKET_NAME>``
78+
.. tabs::
79+
.. group-tab:: AWS CLI
80+
:code:`aws s3api get-bucket-cors --bucket <BUCKET_NAME>`
81+
82+
.. group-tab:: Minio Client (mc)
83+
:code:`mc cors get <STORE_NAME>/<BUCKET_NAME>`
84+
85+
Set CORS Policy on Bucket
86+
+++++++++++++++++++++++++
87+
88+
The examples below shows how to enable CORS rules (to support upload and download) on a bucket.
89+
90+
**Note:** You may want to limit the ``AllowedOrigins`` and/or ``AllowedHeaders`` further.
91+
`GDCC/dataverse-previewers <https://github.com/gdcc/dataverse-previewers/wiki/Using-Previewers-with-download-redirects-from-S3>`_ has some additional information about doing this.
92+
93+
Both JSON and XML format are explained in detail in `AWS Docs <https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html#cors-example-1>`_.
94+
95+
.. tabs::
96+
.. group-tab:: AWS CLI
97+
Create a file :download:`cors.json </_static/installation/cors/cors.json>` as follows:
98+
99+
.. literalinclude:: /_static/installation/cors/cors.json
100+
:name: aws-cors
101+
:language: json
102+
103+
Proceed with making the changes:
104+
105+
:code:`aws s3api put-bucket-cors --bucket <BUCKET_NAME> --cors-configuration file://cors.json`
72106

73-
To proceed with making changes:
107+
Alternatively, you can enable CORS using the AWS S3 web interface, using json-encoded rules as in the example above.
74108

75-
``aws s3api put-bucket-cors --bucket <BUCKET_NAME> --cors-configuration file://cors.json``
109+
.. group-tab:: Minio Client (mc)
110+
Create a file :download:`cors.xml </_static/installation/cors/cors.xml>` as follows:
76111

77-
with the contents of the file cors.json as follows:
112+
.. literalinclude:: /_static/installation/cors/cors.xml
113+
:name: xml-cors
114+
:language: xml
78115

79-
.. code-block:: json
116+
Proceed with making the changes:
80117

81-
{
82-
"CORSRules": [
83-
{
84-
"AllowedOrigins": ["*"],
85-
"AllowedHeaders": ["*"],
86-
"AllowedMethods": ["PUT", "GET"],
87-
"ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Encoding", "Content-Range"]
88-
}
89-
]
90-
}
118+
:code:`mc cors set <STORE_NAME>/<BUCKET_NAME> ./cors.xml`
91119

92-
Alternatively, you can enable CORS using the AWS S3 web interface, using json-encoded rules as in the example above.
93120

94121
.. _s3-tags-and-direct-upload:
95122

0 commit comments

Comments
 (0)