Spider List

This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.

I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list.

This page is located on the UBB.Wiki as well.

The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents

The List:
Code
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
[email protected]
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)[email protected]
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind
80legs=008
Yahoo=Yahoo! Slurp
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Crawler.de=Crawler
EntireWeb.com=Speedy Spider
Gigabot=Gigabot
Google Sitemaps=Google-Sitemaps/1.0
Inktomi=Slurp
MSN=MSNBOT
Naver.com=Yeti/
NorthernLight.com=Gulliver
Searchme=Charlotte
StackRambler(ru)=StackRambler
TAMU.edu=IRLbot
Twiceler=Twiceler-0.9
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
Yahoo=Yahoo Slurp
Excite=Architext spider
Infoseek=Ultraseek
Gigablast=GigaBot
Google-Image=Googlebot-Image
Snap=Snapbot
WISEnut=ZyBorg


Last Update: 2011.06.14
Posted on March 12th, 2008 · Updated on June 14th, 2011

Comments

( Posted)
Request a Quote
Please feel free to Request a Quote at any time.

By requesting a quote you allow us to adjust pricing for your product on a per-project basis. Note that most of the prices on our pricing page are just our "common" pricing and can vary on a per project basis. It will also allow us to add any per project discounts where we see fit (non-profit organizations, affiliations, etc).

If your "type" of project isn't listed in our menu's to the left, please Request a Quote; we can do much more than what is listed however building pages for it all would take a great while.
Financing Available
Hosting
We utilize StableHost for the hosting of our all of our VPS'. They're also #1 host that we recommend to every one of our clients, friends, and family. For more Web Hosting recommendations, please see our Recommended Hosts page!

StableHost Web Hosting

(Use coupon code "VNC" for discounts at checkout)

For EMail hosting we recommend G Suite (Formerly Google Apps and Google for Business).