Recent searches
Search options
@timbray @jasonkoebler's article alludes to @cory's site https://github.com/ai-robots-txt/ai.robots.txt but unfortunately doesn't link to it.
GitHubGitHub - ai-robots-txt/ai.robots.txt: A list of AI agents and robots to block.A list of AI agents and robots to block. Contribute to ai-robots-txt/ai.robots.txt development by creating an account on GitHub.
@underlap @timbray @jasonkoebler looks like there’s a small typo! https://github.com/ai-robots-txt/ai.robots.txt 
GitHubGitHub - ai-robots-txt/ai.robots.txt: A list of AI agents and robots to block.A list of AI agents and robots to block. Contribute to ai-robots-txt/ai.robots.txt development by creating an account on GitHub.
@cory @timbray @jasonkoebler Edited - thanks!
@timbray A lot of the bots simply don't care about robots.txt and ignore it completely.
Blocking directly in the server based on the user-agent is, at least for me, far more effective. Ultimate-nginx-bad-bot-blocker works for me. There is also one for apache.