Skip to content

Commit 7e07bc5

Browse files
robm26robm26
committed
LSQL content additions
1 parent d41d760 commit 7e07bc5

34 files changed

+533
-138
lines changed

‎content/relational-migration/application refactoring/index.en.md‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,28 @@ title : "Application Refactoring"
33
weight : 40
44
---
55

6-
# Lorem
7-
Ipsum
6+
## Updating the Client Application for DynamoDB
7+
After you have chosen your DynamoDB table schema, and migrated any historical data over,
8+
you can consider what code changes are required so a new version of your app can call the DynamoDB
9+
read and write APIs.
10+
11+
The web app we have been using includes forms and buttons to perform standard CRUD (Create, Read, Update, Delete) operations.
12+
13+
The web app makes HTTP calls to the published API using standard GET and POST methods against certain API paths.
14+
15+
1. In Cloud9, open the left nav and locate the file **app.py**
16+
2. Double click to open and review this file
17+
18+
In the bottom half of the file you will see several small handler functions that
19+
pass core read and write requests on to the **db** object's functions.
20+
21+
22+
Notice the file contains a conditional import for the **db** object.
23+
24+
```python
25+
if migration_stage == 'relational':
26+
from chalicelib import mysql_calls as db
27+
else:
28+
from chalicelib import dynamodb_calls as db
29+
```
30+

‎content/relational-migration/application refactoring/index2.en.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
title : "DynamoDB-ready middle tier"
3+
weight : 41
4+
---
5+
6+
## Deploy a new DynamoDB-ready API
7+
8+
If you recall, we had run the command ```chalice deploy --stage relational``` previously
9+
to create the MySQL-ready middle tier.
10+
11+
We can repeat this to create a new API Gateway and Lambda stack, this time using the DynamoDB stage.
12+
13+
1. Within the Cloud9 terminal window, run:
14+
```bash
15+
chalice deploy --stage dynamodb
16+
```
17+
2. When this completes, find the new Rest API URL and copy it.
18+
3. You can paste this into a new browser tab to test it. You should see a status message indicating
19+
the DynamoDB version of the API is working.
20+
21+
We now need a separate browser to test out the full web app experience, since
22+
the original browser has a cookie set to the relational Rest API.
23+
24+
4. If you have multiple browsers on your laptop, such as Edge, Firefox, or Safari,
25+
open a different browser and navigate to the web app:
26+
27+
[https://dynamodbworkshop.s3.amazonaws.com/migration/index.html](https://dynamodbworkshop.s3.amazonaws.com/migration/index.html).
28+
29+
(You can also open the same browser in Incognito Mode for this step.)
30+
31+
5. Click the Target API button and paste in the new Rest API URL.
32+
6. Notice the title of the page has updated to **DynamoDB App** in a blue color.

‎content/relational-migration/application refactoring/index3.en.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
title : "Testing and reviewing DynamoDB code"
3+
weight : 42
4+
---
5+
6+
## Test drive your DynamoDB application
7+
8+
1. Click Tables to see a list of available tables in the account. You should see the
9+
Customers table, vCustOrders table, and a few other tables used by separate workshops.
10+
11+
3. Click on the Customers table, click the SCAN button to see the table's data.
12+
4. Test the CRUD operations such as get-item, and the update and delete buttons in the data grid,
13+
to make sure they work against the DynamoDB table.
14+
4. Click on the Querying tab to display the form with GSI indexes listed.
15+
5. On the idx_region GSI, enter 'North' and press GO.
16+
17+
![DynamoDB GSI Form](/static/images/relational-migration/ddb_gsi.png)
18+
19+
## Updating DynamoDB functions
20+
21+
Let's make a small code change to demonstrate the process to customize the DynamoDB functions.
22+
23+
6. In Cloud9, left nav, locate the chalicelib folder and open it.
24+
7. Locate and open the file dynamodb_calls.py
25+
8. Look on about line 148 for the line ```get_request['ConsistentRead'] = False```
26+
9. Update this from False to True and click File/Save to save your work.
27+
10. In the terminal prompt, redeploy: ```chalice deploy --stage dynamodb```
28+
11. Return to the web app, click on the Customers table, and enter cust_id value "0001" and click the GET ITEM button.
29+
12. Verify a record was retrieved for you. This record was found using a strongly consistent read.
30+
13. Feel free to extend the DynamoDB code to add new functions or modify existing ones.
31+

‎content/relational-migration/data migration/index.en.md‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,18 @@ title : "Data Migration"
33
weight : 30
44
---
55

6-
# Lorem
7-
Ipsum
6+
## Transform, Extract, Convert, Stage Import
7+
8+
Recall our strategy for migrating table data into DynamoDB via S3 was
9+
summarized in the :link[Workshop Introduction]{href="../introduction/index5" target=_blank}.
10+
11+
For each table or view that we want to migrate, we need a routine that will ```SELECT *``` from it,
12+
and convert the result dataset into DynamoDB JSON before writing it to an S3 bucket.
13+
14+
![Migration Flow](/static/images/relational-migration/migrate_flow.png)
15+
16+
For migrations of very large tables we may choose to use purpose-built data tools like
17+
AWS Glue, Amazon EMR, or Amazon DMS. These tools can help you define and coordinate multiple
18+
parallel jobs that perform the work to extract, transform, and stage data into S3.
19+
20+
In this workshop we can use a Python script to demonstrate this ETL process.

‎content/relational-migration/data migration/index2.en.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
title : "ETL Scripts"
3+
weight : 31
4+
---
5+
6+
7+
## mysql_s3.py
8+
9+
A script called mysql_s3.py is provided that performs all the work to convert and load a query result
10+
set into S3. We can run this script in preview mode by using the "stdout" parameter.
11+
12+
1. Run:
13+
```bash
14+
python3 mysql_s3.py Customers stdout
15+
```
16+
You should see results in DynamoDB JSON format:
17+
18+
![mysql_s3.py output](/static/images/relational-migration/mysql_s3_output.png)
19+
20+
2. Next, run it for our view:
21+
```bash
22+
python3 mysql_s3.py vCustOrders stdout
23+
```
24+
You should see similar output from the view results.
25+
26+
The script can write these to S3 for us. We just need to omit the "stdout" command line parameter.
27+
28+
3. Now, run the script without preview mode:
29+
```bash
30+
python3 mysql_s3.py Customers
31+
```
32+
You should see confirmation that objects have been written to S3:
33+
34+
![mysql_s3.py output](/static/images/relational-migration/mysql_s3_write_output.png)
35+
36+

‎content/relational-migration/data migration/index3.en.md‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
title : "Full Migration"
3+
weight : 32
4+
---
5+
6+
## DynamoDB Import from S3
7+
8+
The Import from S3 feature is a convenient way to have data loaded into a new DynamoDB table.
9+
Learn more about this feature [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.HowItWorks.html).
10+
11+
Import creates a brand new table, and is not able to load data into an existing table.
12+
Therefore, it is most useful during the one-time initial load of data during a migration.
13+
14+
## migrate.sh
15+
16+
A script is provided that performs multiple steps to coordinate a migration:
17+
* Runs **mysql_desc_ddb.py** and stores the result in a table definition JSON file
18+
* Runs **mysql_s3.py** to extract, transform, and load data into an S3 bucket
19+
* Uses the **aws dynamodb import-table** CLI command to request a new table, by providing the bucket name and table definition JSON file
20+
21+
1. Run:
22+
```bash
23+
./migrate.sh Customers
24+
```
25+
The script should produce output as shown here:
26+
27+
![Migrate Output](/static/images/relational-migration/migrate_output.png)
28+
29+
Notice the ARN returned. This is the ARN of the Import job, not the new DynamoDB table.
30+
31+
The import will take a few minutes to complete.
32+
33+
2. Optional: You can check the status of an import job using this command, by setting the Import ARN on line two.
34+
35+
```bash
36+
aws dynamodb describe-import \
37+
--import-arn '<paste ARN here>' \
38+
--output json --query '{"Status ":ImportTableDescription.ImportStatus, "FailureCode ":ImportTableDescription.FailureCode, "FailureMessage ":ImportTableDescription.FailureMessage }'
39+
```
40+
41+
We can also check the import status within the AWS Console.
42+
43+
3. Click into the separate browser tab titled "AWS Cloud9" to open the AWS Console.
44+
4. In the search box, type DynamoDB to visit the DyanmoDB console.
45+
5. From the left nav, click Imports from S3.
46+
6. Notice your import is listed along with the current status.
47+
7. Once the import has completed, you can click it to see a summary including item count and the size of the import.
48+
8. On the left nav, click to Tables.
49+
9. In the list of tables, click on the Customers table.
50+
10. On the top right, click on Explore Table Items.
51+
11. Scroll down until you see a grid with your imported data.
52+
53+
Congratulations! You have completed a relational-to-DynamoDB migration.

‎content/relational-migration/data migration/index4.en.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
title : "VIEW migration"
3+
weight : 33
4+
---
5+
6+
## Migrating from a VIEW
7+
8+
In the previous step, you simply ran ```./migrate.sh Customers``` to perform a migration of this table
9+
and data to DynamoDB.
10+
11+
You can repeat this process to migrate the custom view vCustOrders.
12+
13+
1. Run:
14+
```bash
15+
./migrate.sh vCustOrders
16+
```
17+
18+
The script assumes you want a two-part primary key of Partition Key and Sort Key, found in the two leading columns.
19+
20+
In case you want a Partition Key only table, you could specify this like so.
21+
```bash
22+
./migrate.sh vCustOrders 1
23+
```
24+
However, this is not advisable since this particular dataset is not unique by just the first column.
25+
26+
![View output](/static/images/relational-migration/view_result.png)
27+
28+
::alert[Import will write all the records it finds in the bucket to the table. If a duplicate record is encountered, it will simply overwrite it. Please be sure that your S3 data does not contain any duplicates based on the Key(s) of the new table you define.]{header="Note:"}
29+
30+
The second import is also not advisable since if you created a new vCustOrders table in step 1, the second Import attempt would not be able to replace the existing table, and would fail.

‎content/relational-migration/data migration/index5.en.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
title : "SQL Transformation Patterns for DynamoDB"
3+
weight : 34
4+
---
5+
6+
## Shaping Data with SQL
7+
8+
Let's return to the web app and explore some techniques you can use to shape and enrich your relational
9+
data before importing it to DynamoDB.
10+
11+
1. Within the web app, refresh the browser page.
12+
2. Click on the Querying tab.
13+
3. Notice the set of SQL Sample buttons below the SQL editor.
14+
4. Click button one.
15+
The OrderLines table has a two-part primary key, as is common with DynamoDB. We can think of the returned dataset as an Item Collection.
16+
5. Repeat by clicking each other sample buttons. Check the comment at the top of each query, which summarizes the technique being shown.
17+
18+
![SQL Samples](/static/images/relational-migration/sparse.png)
19+
20+
Notice the final two sample buttons. These demonstrate alternate ways to combine data from multiple tables.
21+
We already saw how to combine tables with a JOIN operator, resulting in a denormalized data set.
22+
23+
The final button shows a different approach to combining tables, without using JOIN.
24+
You can use a UNION ALL between multiple SQL queries to stack datasets together as one.
25+
When we arrange table data like this, we describe each source table as an entity and so the single DynamoDB
26+
table will be overloaded with multiple entities. Because of this, we can set the partition key and sort key
27+
names to generic values of PK and SK, and add some decoration to the key values so that it's clear what type
28+
of entity a given record represents.
29+
30+
![Stacked entities](/static/images/relational-migration/stacked.png)

‎content/relational-migration/data migration/index6.en.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
title : "Custom VIEWs"
3+
weight : 35
4+
---
5+
6+
## Challenge: Create New Views
7+
8+
The SQL editor window is provided so that you have an easy way to run queries and
9+
experiment with data transformation techniques.
10+
11+
Using the sample queries as a guide, see how many techniques you can combine in a single query.
12+
Look for opportunities to align attributes across the table so that they can be queried by a GSI.
13+
Consider using date fields in column two, so that they become Sort Key values, and be queryable with
14+
DynamoDB range queries.
15+
16+
1. When you have a SQL statement you like, click the CREATE VIEW button.
17+
2. In the prompt, enter a name for your new view. This will add a CREATE VIEW statement to the top of you query.
18+
3. Click RUN SQL to create the new view.
19+
4. Refresh the page, and your view should appear as a button next to the vCustOrders button.
20+

‎content/relational-migration/index.en.md‎

Lines changed: 19 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -29,31 +29,30 @@ You can review this guidance when considering and planning your own relational m
2929

3030
[AWS Documentation: Relational Migration to DynamoDB Developer Guide](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/migration-guide.html)
3131

32-
[//]: # (## Workshop overview)
3332

34-
[//]: # (The workshop includes a Cloud9 developer workstation and an S3 bucket for staging data.)
33+
## Workshop overview
3534

36-
[//]: # (The source database for migration is a deployed a MySQL instance, running on EC2. )
35+
The workshop provides a MySQL instance running on EC2, a Cloud9 developer workstation,
36+
and an S3 bucket for staging data.
3737

38-
[//]: # (Run a script to deploy the sample set of MySQL tables and data.)
39-
40-
[//]: # (You will create a serverless API and Python Lambda function that)
41-
42-
[//]: # (performs database read and write operations against the relational database. )
43-
44-
[//]: # (A sample web app provides a GUI interface as a test harness for the serverless API. )
45-
46-
[//]: # ()
47-
[//]: # (An end-to-end migration script is run that performs a SQL query, transforms results to DynamoDB JSON and writes to Amazon S3, then starts a DynamoDB Import job.)
48-
49-
[//]: # ()
50-
[//]: # (Developer challenge: Tour various SQL data modeling techniques, then combine them into a final single-table transformation. )
51-
52-
[//]: # (Developer challenge: write a new set of data access functions that point to DynamoDB. )
53-
54-
[//]: # ()
38+
You will create a serverless API and Python Lambda function that
39+
performs database read and write operations against the relational database,
40+
and then deploy a new version that performs read and write operations against DynamoDB.
5541

42+
A sample web app is provided that:
43+
* Acts as a GUI test harness for the serverless API
44+
* Converts tables and indexes into suggested DynamoDB tables and GSIs
45+
* Has a SQL editor with a set of sample queries, and hints on how to combine tables
46+
* Performs read and write operations to both MySQL and DynamoDB
47+
and provides hints and suggestions for building a migration
5648

49+
You will run a set of scripts that:
50+
* Deploys a set of sample MySQL tables, views, and data.
51+
* Converts MySQL table metadata to a DynamoDB table definition.
52+
* Converts the results of a SQL query into DynamoDB JSON format, and stores in the Amazon S3 bucket.
53+
* Perform a full migration by running a SQL query, transforming results to DynamoDB JSON, writing to Amazon S3, then starting a DynamoDB Import job.
5754

55+
Developer challenge: Tour various SQL data modeling techniques, then combine them into a custom VIEW for a single-table transformation.
56+
Developer challenge: Write a new set of data access functions that point to DynamoDB.
5857

5958

0 commit comments

Comments
 (0)