Protect your website from bots

You can't stop bots from reaching your website, but you can hold back the harmful ones: put a protection service in front of the site (a web application firewall or WAF, like the one Cloudflare offers), limit how many requests a single address can make in a short time and protect the forms and the admin login. The goal isn't zero bots, but making sure the bad ones don't do damage or cost you money.

A bot is any program that visits websites automatically, and not all of them are bad. Google's or Bing's put you in the search engines; others, the crawlers from AI companies, read your content to train models or answer questions. The robots.txt file is for telling them what they can visit, but it's a "please" sign: the polite ones respect it and the malicious ones ignore it. What often happens at the start is blocking everything and disappearing from Google. The clues that you have a problem: strange traffic, a slow server and attempts to get into the admin panel.

What kinds of bots showed up on my website as soon as I published it and how I stopped them without being an expert I explain in "Where the attacks came from when I published my website". My advice for starting out: before blocking anything, look at your stats for a few days and find out who's really visiting you. Blocking blindly is the fastest way to throw out the good visitors too.