What's a Yahoo Slurp Spider?

[aesop]

When you view "who's online", there are many "slurp spiders" and guests. So, what's a slurp spider?



Just curious

 

[Stephan Whelan]

The Yahoo Slurp spider is the Yahoo "bot" that crawls websites to provide results for the Yahoo Search engine.

 

[aesop]

Oh, I see. Pretty wierd, it looks at peoples profiles and threads, and for a substantial amount of time. It's not like it's only a few seconds, it's more like a few minutes.

 

[Stephan Whelan]

It will spider EVERYTHING on the site - doesn't matter what it is.

It actually doesn't spend more than a few seconds on each page but due to the way that the forum software tracks user "timeouts" it appears that it spends a few minutes on each page.

 

[aesop]

And it appears as a guest, so there's actually less PEOPLE on the site than it actually appears. Right? So does that mean when there was a record of 500 and some users on the site, there was actually a large amount of slurp spiders combined with registered users and actual people guests?

 

[Ricochet]

I whould be surprised if there where more than 3 bots crawling this site at a time. Its more likely that the 'timeouts' where set to a larger span... like 30 minutes or so.


....or there have really been more than 500 users online

 

[trux]

I whould be surprised if there where more than 3 bots crawling this site at a time. Its more likely that the 'timeouts' where set to a larger span... like 30 minutes or so.

When a SE spider visits a site, it often loads huge number of pages almost concurrently and since spiders usually do not accept and return any cookies or even session id's, each of the page requests appears as if it was made by another visitor - hence the huge visitor number. Some forums or other web applications are capable to filter out such false hits from the visitor statistics, but apparently vBulletin does not.

 

[Ricochet]

You are right about the cookies. Didnt think of it.