1.根据Apache Or Nginx 日志中的时间段来统计查看各种蜘蛛的大致爬取数据,
sed -n '/2013:13/,/2013:15:/p' www.access.log | \ egrep -i 'bot|crawler|slurp|spider' | grep "HTTP" | \ awk '{print $1,$2,$3,$4,$5,$6,$7}' | head -n 20
1.根据Apache Or Nginx 日志中的时间段来统计查看各种蜘蛛的大致爬取数据,
sed -n '/2013:13/,/2013:15:/p' www.access.log | \ egrep -i 'bot|crawler|slurp|spider' | grep "HTTP" | \ awk '{print $1,$2,$3,$4,$5,$6,$7}' | head -n 20