Update to version v1.2.0

Author: aassadza

.github/ISSUE_TEMPLATE/bug_report.md

@@ -17,9 +17,9 @@ Steps to reproduce the behavior.
 A clear and concise description of what you expected to happen.
 
 **Please complete the following information about the solution:**
-- [ ] Version: [e.g. v1.0.0]
+- [ ] Version: [e.g. v1.1.0]
 
-To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0122) - Discovering Hot Topics using Machine Learning. Version v1.0.0".
+To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0122) - Discovering Hot Topics using Machine Learning. Version v1.1.0".
 
 - [ ] Region: [e.g. us-east-1]
 - [ ] Was the solution modified from the version published on this repository?

.gitignore

@@ -42,6 +42,7 @@ pids
 lib-cov
 
 # Coverage directory used by tools like istanbul
+coverage-reports/
 coverage
 *.lcov
 
@@ -186,6 +187,9 @@ coverage.xml
 .hypothesis/
 .pytest_cache/
 
+# linting, scanning configurations, sonarqube
+.scannerwork/
+
 # Translations
 *.mo
 *.pot

CHANGELOG.md

@@ -5,11 +5,19 @@
   The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
   and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+  ## [1.2.0] - 2020-10-29
+  ### Added
+  - New and simplified interactive Amazon QuickSight dashboard that is now automatically generated through an AWS CloudFormation deployment and that customers can extend to suit their business case
+
+  ### Updated
+  - Updated to AWS CDK v1.69.0
+  - Consolidate Amazon S3 access Log bucket across the solution. All access log files have a prefix that corresponds to the bucket for which they are generated
+
   ## [1.1.0] - 2020-09-29
   ### Updated
   - S3 storage for inference outputs to use Apache Parquet
-  - Add paritioning to AWS Glue tables
-  - Update to CDK v1.63.0
+  - Add partitioning to AWS Glue tables
+  - Update to AWS CDK v1.63.0
   - Update to AWS SDK v2.755.0
 
   ## [1.0.0] - 2020-08-28

NOTICE.txt

@@ -34,6 +34,7 @@ pytest-cov - MIT license
 pytest - MIT license
 requests - Apache-2.0
 sinon - BSD license
+tenacity under the Apache License Version 2.0
 ts-jest - MIT license
 ts-node - MIT license
 twitter-lite - MIT License

README.md

@@ -1,10 +1,14 @@
 ## Discovering Hot Topics using Machine Learning
 
-With so much customer sentiment available for analysis today, understanding the contextualization of the most relevant topics can be difficult at scale. Separating the signal from the noise requires analysis that goes beyond basic aggregation of sentiment analysis. Diving deeper and truly understanding the conversation at scale can help organizations to succeed in the market, identify new opportunities and react quickly.
+The Discovering Hot Topics Using Machine Learning solution helps you identify the most dominant topics associated with your products, policies, events, and brands. Implementing this solution helps you react quickly to new growth opportunities, address negative brand associations, and deliver higher levels of customer satisfaction.
 
-The Discovering Hot Topics Using Machine Learning solution addresses the problem of organizing large-scale customer feedback analytics by automating ingesting digital assets and performing near real-time analysis using machine learning algorithms. Organizations can gain insight about new product launches, service announcements, public relations, crisis management, and changes to company policies that impact their customers.
+The solution uses machine learning algorithms to automate digital asset (text and image) ingestion and perform near real-time topic modeling, sentiment analysis, and image detection. The solution then visualizes these large-scale customer analyses using an Amazon QuickSight dashboard. This guide provides step-by-step instructions to building a dashboard that provides you with the context and insights necessary to identify trends that help or harm you brand.
 
-This solution ingests text and images from online discourse and performs topic and sentiment analyses, and detect unsafe content in images. The default input data source for the solution is Twitter, but it can be extended to ingest other social media platforms in addition to any stream from an enterprise’s internal systems. The output of this inference is organized and visualized in a dashboard for users to consult and analyze. 
+The solution performs the following key features:
+* **Performs topic modeling to detect dominant topics**: identifies the terms that collectively form a topic from within customer feedback
+* **Identifies the sentiment of what customers are saying**: uses contextual semantic search to understand the nature of online discussions
+* **Determines if images associated with your brand contain unsafe content**: detects unsafe and negative imagery in content
+* **Helps customers identify insights in near real-time**: you can use a visualization dashboard to better understand context, threats, and opportunities almost instantly
 
 For an overview and solution deployment guide, please visit [Discovering Hot Topics using Machine Learning](https://aws.amazon.com/solutions/implementations/discovering-hot-topics-using-machine-learning)
 
@@ -22,14 +26,15 @@ Deploying this solution with the default parameters builds the following environ
 * Application Integration – Event based architecture approach through the use of AWS Events Bridge
 * Storage and Visualization – A combination of Kinesis Data Firehose, S3 Buckets, Glue, Athena and QuickSight
 
-Once the solution is deployed, use QuickSight to create a dashboard like the one below.
+After you deploy the solution, use the included Amazon QuickSight dashboard to visualize the solution's machine learning inferences. The image to the right is an example
+visualization dashboard featuring a dominant topic list, donut charts, weekly and monthly trend graphs, a word cloud, a tweet table, and a heat map.
 
 <p align="center">
   <img src="source/images/dashboard.png">
   <br/>
 </p>
 
-This is an example Amazon QuickSight dashboard built by the solution. The first row of visuals in the dashboard shows the aggregation of all the dominant topics detected, and the second row drills down to the most dominant topic '000'. The bottom left corner of the image demonstrates that selecting a specific phrase (in this example, machine learning) in the word cloud filters the data for the related donut chart and table.
+The first row of visuals in the dashboard shows the aggregation of all the dominant topics detected, and the second row drills down to the most dominant topic '000'. The bottom left corner of the image demonstrates that selecting a specific phrase (in this example, machine learning) in the word cloud filters the data for the related donut chart and table.
 
 ## 1. Build the solution
 
@@ -48,17 +53,26 @@ chmod +x ./run-all-tests.sh
 
 * Configure the bucket name of your target Amazon S3 distribution bucket
 ```
-export DIST_OUTPUT_BUCKET=my-bucket-name # bucket where customized code will reside
-export VERSION=my-version # version number for the customized code
+export DIST_OUTPUT_BUCKET=my-bucket-name
+export VERSION=my-version
 ```
-_Note:_ You would have to create an S3 bucket with the prefix 'my-bucket-name-<aws_region>'; aws_region is where you are testing the customized solution. Also, the assets in bucket should be publicly accessible.
 
 * Now build the distributable:
 ```
 cd <rootDir>/deployment
 chmod +x ./build-s3-dist.sh
-./build-s3-dist.sh $DIST_OUTPUT_BUCKET $SOLUTION_NAME $$VERSION
+./build-s3-dist.sh $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION $CF_TEMPLATE_BUCKET_NAME QS_TEMPLATE_ACCOUNT
+
+```
+* Parameter details
 ```
+$DIST_OUTPUT_BUCKET - This is the global name of the distribution. For the bucket name, the AWS Region is added to the global name (example: 'my-bucket-name-us-east-1') to create a regional bucket. The lambda artifact should be uploaded to the regional buckets for the CloudFormation template to pick it up for deployment.
+$SOLUTION_NAME - The name of This solution (example: discovering-hot-topics-using-machine-learning)
+$VERSION - The version number of the change
+$CF_TEMPLATE_BUCKET_NAME - The name of the S3 bucket where the CloudFormation templates should be uploaded
+$QS_TEMPLATE_ACCOUNT - The account from which the Amazon QuickSight templates should be sourced for Amazon QuickSight Analysis and Dashboard creation
+```
+
 
 * Deploy the distributable to an Amazon S3 bucket in your account. _Note:_ you must have the AWS Command Line Interface installed.
 ```

deployment/build-s3-dist.sh

@@ -26,18 +26,23 @@
 set -e
 
 # Important: CDK global version number
-cdk_version=1.63.0
+cdk_version=1.69.0
 
 # Check to see if input has been provided:
-if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
+if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ] || [ -z "$4" ] || [ -z "$5" ] || [ -z "$6" ]; then
     echo "Please provide all required parameters for the build script"
-    echo "For example: ./build-s3-dist.sh solutions trademarked-solution-name v1.0.0"
+    echo "For example: ./build-s3-dist.sh solutions trademarked-solution-name v1.2.0 template-bucket-name template_account_id solutions"
     exit 1
 fi
 
 bucket_name="$1"
 solution_name="$2"
 solution_version="$3"
+template_bucket_name="$4"
+template_account_id="$5"
+dist_quicksight_namespace="$6"
+
+dashed_version="${solution_version//./$'_'}"
 
 # Get reference for all important folders
 template_dir="$PWD"
@@ -126,6 +131,18 @@ do
 
     replace="s/%%VERSION%%/$solution_version/g"
     sed -i -e $replace $file
+
+    replace="s/%%TEMPLATE_BUCKET_NAME%%/$template_bucket_name/g"
+    sed -i -e $replace $file
+
+    replace="s/%%TEMPLATE_ACCOUNT_ID%%/$template_account_id/g"
+    sed -i -e $replace $file
+
+    replace="s/%%DIST_QUICKSIGHT_NAMESPACE%%/$dist_quicksight_namespace/g"
+    sed -i -e $replace $file
+
+    replace="s/%%DASHED_VERSION%%/$dashed_version/g"
+    sed -i -e $replace $file
 done
 
 echo "------------------------------------------------------------------------------"

deployment/cdk-solution-helper/index.js

@@ -34,22 +34,66 @@ fs.readdirSync(global_s3_assets).forEach(file => {
         if (fn.Properties.Code.hasOwnProperty('S3Bucket')) {
             // Set the S3 key reference
             let artifactHash = Object.assign(fn.Properties.Code.S3Bucket.Ref);
-            artifactHash = artifactHash.replace('AssetParameters', '');
+            // console.debug(`Old artificatHash is ${artifactHash}`);
+            artifactHash = artifactHash.replace(/[\w]*AssetParameters/g, '');
             artifactHash = artifactHash.substring(0, artifactHash.indexOf('S3Bucket'));
+            // console.debug(`New artificatHash is ${artifactHash}`);
             const assetPath = `asset${artifactHash}`;
             fn.Properties.Code.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}.zip`;
 
             // Set the S3 bucket reference
             fn.Properties.Code.S3Bucket = {
                 'Fn::Sub': '%%BUCKET_NAME%%-${AWS::Region}'
             };
+        } else {
+            // console.debug(`Here is the fn dump ${JSON.stringify(fn)}`);
         }
     });
 
+    // Clean-up nested template stack dependencies
+    const nestedStacks = Object.keys(resources).filter(function(key) {
+        return resources[key].Type === 'AWS::CloudFormation::Stack'
+    });
+
+    nestedStacks.forEach(function(f) {
+        const fn = template.Resources[f];
+        fn.Properties.TemplateURL = {
+            'Fn::Join': [
+                '',
+                [
+                    'https://s3.',
+                    {
+                        'Ref' : 'AWS::URLSuffix'
+                    },
+                    '/',
+                    `%%TEMPLATE_BUCKET_NAME%%/%%SOLUTION_NAME%%/%%VERSION%%/${fn.Metadata.nestedStackFileName}`
+                ]
+            ]
+        };
+
+        const params = fn.Properties.Parameters ? fn.Properties.Parameters : {};
+        const nestedStackParameters = Object.keys(params).filter(function(key) {
+            if (key.search(/[\w]*AssetParameters/g) > -1) {
+                return true;
+            }
+            return false;
+        });
+
+        nestedStackParameters.forEach(function(stkParam) {
+            fn.Properties.Parameters[stkParam] = undefined;
+        });
+    });
+
     // Clean-up parameters section
     const parameters = (template.Parameters) ? template.Parameters : {};
     const assetParameters = Object.keys(parameters).filter(function (key) {
-        return key.includes('AssetParameters');
+        console.debug(`key to analyze ${key}`);
+        if (key.search(/[\w]*AssetParameters/g) > -1) {
+            // console.debug('Pattern match');
+            return true;
+        }
+        // console.debug('Pattern did not match');
+        return false;
     });
     assetParameters.forEach(function (a) {
         template.Parameters[a] = undefined;

source/bin/discovering-hot-topics-app.ts

@@ -19,5 +19,6 @@ import { DiscoveringHotTopicsStack } from '../lib/discovering-hot-topics-stack';
 const app = new cdk.App();
 new DiscoveringHotTopicsStack(app, 'discovering-hot-topics-using-machine-learning', {
     description: '(SO0122) - Discovering Hot Topics using Machine Learning. Version %%VERSION%%',
-    solutionID: 'SO0122'
+    solutionID: 'SO0122',
+    solutionName: 'discovering-hot-topics-using-machine-learning'
 });

source/cdk.json

@@ -1,3 +1,6 @@
 {
-  "app": "npx ts-node bin/discovering-hot-topics-app.ts"
+  "app": "npx ts-node bin/discovering-hot-topics-app.ts",
+  "context": {
+    "quicksight_source_template_arn": "arn:aws:quicksight:us-east-1:%%TEMPLATE_ACCOUNT_ID%%:template/%%DIST_QUICKSIGHT_NAMESPACE%%_%%SOLUTION_NAME%%_%%DASHED_VERSION%%"
+  }
 }