Comment bannir un crawler web en utilisant fail2ban

J'utilise nginx et, si je ne me trompe pas, je suis toujours attaqué par les robots d'exploration.

J'ai essayé de configurer fail2ban mais l'adresse IP ne peut pas être détectée par fail2ban.

La raison pour laquelle il n'est pas détecté est qu'il semble qu'il s'agisse d'un visiteur légitime. Voici l'exemple de journal :

116.73.68.36 - - [19/Jul/2013:23:57:47 +0800] "GET /sites/default/files/download/rhenz23/it4cai.zip HTTP/1.1" 206 14628884 "http://www.mysite.com/php/5297/computer-aided-instruction.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 14510149 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16357796 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/tovi/online_lot_reservation_system.zip HTTP/1.1" 206 14850935 "http://www.mysite.com/php/4088/online-lot-reservation.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17217908 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/sanbunna9/online_gues_house.zip HTTP/1.1" 206 17594389 "http://www.mysite.com/php/5235/online-guest-house.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:50 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17070214 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/welmarie/online_product_reservation_system.zip HTTP/1.1" 206 15074810 "http://www.mysite.com/php/3969/online-product-reservation-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1_0.zip HTTP/1.1" 206 15232701 "http://www.mysite.com/php/4094/online-membership-and-billing-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 15555605 "http://www.mysite.com/php/4171/online-management-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16379516 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 17671134 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:59 +0800] "GET /sites/default/files/download/malyn30/socialnetworkingsite.zip HTTP/1.1" 206 16711108 "http://www.mysite.com/php/3971/sample-simple-social-networking-site.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:04 +0800] "GET /sites/default/files/download/Franziholic/franzdarylduetes.zip HTTP/1.1" 206 17718916 "http://www.mysite.com/php/5408/reyans-burger-online-ordering-system-using-php.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:07 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 16876180 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:08 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 16685045 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:15 +0800] "GET /sites/default/files/download/jkev/psits_voting_system.zip HTTP/1.1" 206 17465518 "http://www.mysite.com/php/5442/drag-and-drop-voting-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:32 +0800] "GET /sites/default/files/download/may_ann/onlineschedulingsystem.zip HTTP/1.1" 206 12997278 "http://www.mysite.com/php/scheduling-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"

BTW, j'ai testé mes paramètres fail2ban en utilisant la commande suivante et cela fonctionne bien.

fail2ban-regex /var/log/ispconfig/httpd/mysite.com/yesterday-access.log /etc/fail2ban/filter.d/apache-badbots.conf

En fait, il a détecté deux adresses IP avec le log suivant :

5.9.23.42 - - [19/Jul/2013:07:52:26 +0800] "GET / HTTP/1.1" 200 11220 "-" "Mozilla/3.0 (compatible; Indy Library)"
124.122.67.67 - - [19/Jul/2013:02:43:30 +0800] "GET / HTTP/1.1" 200 59663 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
124.122.67.67 - - [19/Jul/2013:02:43:40 +0800] "-" 400 0 "-" "-"

Alors comment se fait-il que fail2ban ne le détecte pas ? S'agit-il vraiment d'un bot ?

J'utilise également cloudflare pour éviter ce type de bot, mais je dois bloquer manuellement l'adresse IP ou changer le paramètre en "Je suis attaqué". Cela fonctionne bien si je suis toujours en train de surveiller mon serveur. Mais qu'en est-il si je suis loin de mon ordinateur ? Alors comment puis-je empêcher ce robot ?

Aidez-moi, s'il vous plaît.

Mise à jour :

Voici mes paramètres jail.local :

[nginx-badbots]
enabled  = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https"]
logpath = /var/log/ispconfig/httpd/mysite.com/access.log
bantime = 86400 # 1 day
maxretry = 1