Deny access to the server and all its ports completely except for CloudFlare subnets - https://www.cloudflare.com/ips/ and your IP or the subnet from which you administer the server. If you have already been caught by these Internet scanners (censys, fofa etc..), then after performing the above actions, your server's IP will disappear from the search results in 3-7 days. But it is better to change the IP. In addition, you can give 444 or 403 to those who connect to http / https with a host header different from yours.
As for crawlers and bots, well, it depends on which ones you are protecting yourself from, firstly, you need to ban idiots who use default user agents like Go-http-client, libcurl, python and so on, this can be done through the WAF rules in CloudFlare itself. Secondly, you need to use Super BotFight Mode, JS detection. Thirdly, enable the JS challenge in the settings for those countries from which you get the most crap, such as India, Brazil, China, etc. WAF in CloudFlare itself does a good job (Cloudflare OWASP Core Ruleset + Cloudflare Managed Ruleset), you can additionally configure the paranoia level. All this is configured in Security > WAF > Managed rules. Well, for those who are especially anxious, you can also install a WAF on the backend, for example, the same ModSecurity, open source + updated regularly.
As for crawlers and bots, well, it depends on which ones you are protecting yourself from, firstly, you need to ban idiots who use default user agents like Go-http-client, libcurl, python and so on, this can be done through the WAF rules in CloudFlare itself. Secondly, you need to use Super BotFight Mode, JS detection. Thirdly, enable the JS challenge in the settings for those countries from which you get the most crap, such as India, Brazil, China, etc. WAF in CloudFlare itself does a good job (Cloudflare OWASP Core Ruleset + Cloudflare Managed Ruleset), you can additionally configure the paranoia level. All this is configured in Security > WAF > Managed rules. Well, for those who are especially anxious, you can also install a WAF on the backend, for example, the same ModSecurity, open source + updated regularly.
