Trapping misbehaving bots in an AI Labyrinth

[Rui Carlos]

Citação

[...] While Cloudflare has several tools for identifying and blocking unauthorized AI crawling, we have found that blocking malicious bots can alert the attacker that you are on to them, leading to a shift in approach, and a never-ending arms race. So, we wanted to create a new way to thwart these unwanted bots, without letting them know they’ve been thwarted.

To do this, we decided to use a new offensive tool in the bot creator’s toolset that we haven’t really seen used defensively: AI-generated content. When we detect unauthorized crawling, rather than blocking the request, we will link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them. But while real looking, this content is not actually the content of the site we are protecting, so the crawler wastes time and resources. 

As an added benefit, AI Labyrinth also acts as a next-generation honeypot. No real human would go four links deep into a maze of AI-generated nonsense. Any visitor that does is very likely to be a bot, so this gives us a brand-new tool to identify and fingerprint bad bots, which we add to our list of known bad actors. Here’s how we do it…

[...]

Fonte: https://blog.cloudflare.com/ai-labyrinth/

 

Tendo em conta a forma como muitos AI crawlers estão frequentemente a ignorar as boas práticas na área, por exemplo, ignorando os robots.txt ou criando cargas absurdas sob os sites, parece-me uma boa solução para quem usa a Cloudflare.

(E para quem não usa Cloudflare, há também soluções como o Nephentes que podem instalar localmente.)