Search Engine's seem to get "stuck" on my site...

There are various places in which search engines can get stuck, such as the calendar (as it increases indefinitely); there are some robots.txt entries that you can add such as:

robots.txt
Code
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php/ubb/calendar.html
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php/ubb/showday.html
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile.html
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers.html
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php/ubb/online.html
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php/ubb/search.html
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php/ubb/faq.html
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies.html
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php/ubb/markallread.html
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
Disallow: /forum/ubbthreads.php/ubb/newuser.html


Be sure to edit the path to your forums from your web root; mine for example is /forum but yours may vary.

Several items in this list aren't just bot traps, but places where SE's shouldn't be in the first place as it will just show an access denied error (as a SE is just a guest).

These include:
new user/registration page
Mark all Read
Clear cookies
View privacy
FAQ
search
who's online
show profile (you can remove this setting if you want SE's trolling your member data)
show members
calendar (including show day's events)
Posted on June 22nd, 2007

Comments

( Posted)
Request a Quote
Please feel free to Request a Quote at any time.

By requesting a quote you allow us to adjust pricing for your product on a per-project basis. Note that most of the prices on our pricing page are just our "common" pricing and can vary on a per project basis. It will also allow us to add any per project discounts where we see fit (non-profit organizations, affiliations, etc).

If your "type" of project isn't listed in our menu's to the left, please Request a Quote; we can do much more than what is listed however building pages for it all would take a great while.
Financing Available
Hosting
We utilize StableHost for the hosting of our all of our VPS'. They're also #1 host that we recommend to every one of our clients, friends, and family. For more Web Hosting recommendations, please see our Recommended Hosts page!

StableHost Web Hosting

(Use coupon code "VNC" for discounts at checkout)

For EMail hosting we recommend G Suite (Formerly Google Apps and Google for Business).