This is the official list of user agents that are regarded as robots/spiders by COUNTER Metrics. It includes the list of AI-related crawlers from the ai.robots.txt repository.
Web crawlers and other bots have the potential to inflate usage statistics. Only genuine, user-driven usage should be reported in COUNTER Reports and Standard Views of COUNTER Reports. This is a non-exclusive list that can be used to determine whether a request is performed by a bot.
This list is open for extension and anyone can make a new pull request with new additions to this list. When proposing new bots it is important to include the bot name and/or pattern, as well as a description of the bot activity.
Merges of changes to the branch will automatically trigger a new official release by the end of the day. A tag with the date is automatically made when a change is released.
The list is available in a JSON format as well as in a simplified .txt file.
We recommend case-insensitive matching, to catch any case-specific iterations of bots. For example, if we want to match 'bot', we can be fairly certain that 'BoT' is to be matched as well. Case insensitive matching thus reduces the number of configs to take into consideration.