Spider List · Blog
This is by far not a complete list of spiders; that would be most impossible to accomplish. Instead, this is a list of popular spiders that I have come across over several communities.
I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list.
This page is located on the UBB.Wiki as well.
The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents
The List:
Last Update: 2011.06.14
Posted on March 12, 2008 · Updated on June 14, 2011
I will be updating this list from time to time so please check back often. I'll be sure to put the date of the *latest* update under the list.
This page is located on the UBB.Wiki as well.
The Spider Listing can be added to your forums by going to:
Control Panel > Display Options > General > Primary > Who's Online > Search Engine Agents
The List:
Code
Accelovation=heritrix
Ahrefs=AhrefsBot
Alexa=ia_archiver
AltaVista=Mercator
Altavista=Scooter
AllTheWeb=FAST-WebCrawler
Amazon=AMZNKAssocBot
Anzwers=AnzwersCrawl
AllTheWeb=crawler@fast
Ask=Ask Jeeves
Ask=teoma_agent1
Ask=Teoma
Atomz=Atomz
AskJeeves=AskJeeves
Babaloo=BabalooSpider
BaiDuSpider=BaiDuSpider
BecomeBot=BecomeBot
Bing=bingbot/2.0
BoardReader=BoardReader
Boitho=boitho.com
Cityreview=Cityreview Robot
Cuil=Twiceler
CyberPatrol=CyberPatrol
del.icio.us Thumbnails=del.icio.us-thumbnails
DepSpid=DepSpid
DMOZ=Robozilla
DNS Mine=DNSMine
Dontbuylists=DBLBot
Dotbot=Dotbot
Entireweb=Speedy Spider
Excite=ArchitextSpider
Exalead=Exabot
Factbites=Factbot
Fast=FAST
Fast(AllTheWeb)=FAST-WebCrawler
Fast(AllTheWeb)=crawler@fast
Gigablast=Gigabot
Google=Googlebot
Google Mobile=Googlebot-Mobile
Google Images=Googlebot-Image
Google Adsense=Mediapartners-Google
Infoseek=InfoSeek sidewinder
InfoSeek=UltraSeek
Kalooga=Kalooga
libwww=libwww-perl
Looksmart=MantraAgent
LookSmart=FurlBot
Lycos=Lycos_Spider_(T-Rex)
Magpie=magpie-crawler
MJ12bot=MJ12bot
Microsoft Research=MSRBOT
MSN Media=msnbot-media
MSN=msnbot
MSN Pic-Search=PSbot
Naver=Yeti/
NaverBot=NaverBot
NetSeer=Teemer
NorthernLight=Gulliver
noXtrum=noxtrumbot
Nutch=nutchsearch
Omgili=omgilibot
OpenAcoon=openacoon
Pagebull=Pagebull
Panscient=panscient.com
Picsearch=psbot
PrivacyFinder=PrivacyFinder
ScoutJet=ScoutJet
ScrubTheWeb=Scrubby
Search Me=Charlotte
Seznam=SeznamBot
Search Hippo=Fluffy the Spider
Shelob=shelob
Similarpages=Similar
SingingFish=asterias
Snapbot=Snapbot
SnapPreviewBot=SnapPreviewBot
Snoopy=Snoopy
Sogou=sogou web spider
StackRambler=StackRambler
StumbleUpon=StumbleUpon.com
TinEye=TinEye
TAMU=IRLbot
Teoma=TeomaAgent
TMCrawler=TMCrawler
Voila=Voila.fr
Voila=Voila
Voila=VoilaBot
Voyager=Voyager
Walhello=appie
WebAlta=WebAlta Crawler
Webcrawler=WebCrawler
Web Data Centre=WebDataCentreBot
Whois=SurveyBot
Wisenut=Zyborg
Worio=woriobot
Yahoo! China=Yahoo! Slurp China
Yahoo! Germany=Yahoo! DE Slurp
Yahoo!=Yahoo! Slurp
Yahoo! Blogs=Yahoo-blogs/v3.9
Yahoo! MM=Yahoo-MMcrawler
YaCy=yacybot
Yandex=Yandex
Yanga=Yanga
YodaoBot=YodaoBot
Yoono=yoofind
80legs=008
Yahoo=Yahoo! Slurp
Ask.com=Ask Jeeves
Ask.com=teoma_agent1
Crawler.de=Crawler
EntireWeb.com=Speedy Spider
Gigabot=Gigabot
Google Sitemaps=Google-Sitemaps/1.0
Inktomi=Slurp
MSN=MSNBOT
Naver.com=Yeti/
NorthernLight.com=Gulliver
Searchme=Charlotte
StackRambler(ru)=StackRambler
TAMU.edu=IRLbot
Twiceler=Twiceler-0.9
WebAlta=WebAlta
Webcrawler.com=WebCrawler
Whois.sc=SurveyBot
Yahoo=Yahoo Slurp
Excite=Architext spider
Infoseek=Ultraseek
Gigablast=GigaBot
Google-Image=Googlebot-Image
Snap=Snapbot
WISEnut=ZyBorgLast Update: 2011.06.14
Posted on March 12, 2008 · Updated on June 14, 2011
