Fix issue with robots.txt deploy

CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## DMPTool Releases
 
+### v5.5.9
+- Updated config/robots.txt 
+
 ### v5.58
 - Added custom override for `idsc.miami.edu` and `miami.edu`
 

config/robots.txt

@@ -1,9 +1,27 @@
-# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
+# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
 #
-# To ban all spiders from the entire site uncomment the next two lines:
-# User-Agent: *
-# Disallow: /
+# This file is copied into place during the Capistrano deploy process.
 
 # Prevent spiders from downloading PDF files.
 User-agent: *
-Disallow: /pdf/
+Crawl-delay: 5
+Disallow: /rails/active_storage/
+Disallow: /narratives/*.pdf
+
+# Explicitly name certain bots to ensure they are blocked from downloading PDF files.
+User-agent: Amazonbot
+User-agent: Amzn-SearchBot
+User-agent: AmazonProductDiscoverybot
+User-agent: Bytespider
+User-agent: GPTBot
+User-agent: OAI-SearchBot
+User-agent: ClaudeBot
+User-agent: CCBot
+User-agent: Googlebot
+User-agent: Meta-ExternalAgent
+User-Agent: PetalBot
+User-agent: Turnitin
+User-agent: TurnitinBot
+Crawl-delay: 10
+Disallow: /rails/active_storage/
+Disallow: /narratives/*.pdf