什么情况下user_agent 会出现Oodlebot/1.0？
请教大虾，什么情况下user_agent 会出现Oodlebot/1.0？oodle bot.......... 从名字来分析，应该是Oodle的爬虫。如果自己编程直接用HTTP来访问，可以设置任意字符串作为user-agent。 第一次听说oodle，呵呵 OodleBot (aka. oodlebot/1.0) is the name of our crawler, and we've let it loose on the Internet to search for classified listings.
When OodleBot finds a listing, it tries to understand what the listing is all about. Once the information is gathered, it strips all the contact info from the listing and puts the listing into our search engine index, where we then show the listing to all our users and partners with links back to the page it was found on.
OodleBot tries very hard to make sure it has the freshest and most accurate information available without hurting the site it's crawling from downloading too much data too frequently.
Instead of letting OodleBot crawl your site, we'll happily accept a feed to download all your listings in just one simple file. Please see our feed guidelines and FAQ.
We work hard to house train our little bot, but if you find that OodleBot has been behaving badly on your site, please contact us at [email protected] and we will make sure to teach it a lesson ASAP.
If you want to make sure OodleBot doesn't stray into sensitive areas on your site, you can guide it with your robots.txt file using the User-Agent "oodlebot".
Read more: http://www.oodle.com/info/oodlebot.html#ixzz2USoC4fYw Oodlebot 蜘蛛访问网站的时候。
页:
[1]
