World-Wide Web (Or "robot", "crawler") A program that
document and recursively retrieving some or all the documents
that are referenced in it. This is in contrast with a normal
web browser operated by a human that doesn't automatically
The
algorithm used to pick which references to follow
strongly depends on the program's purpose.
Index-building
spiders usually retrieve a significant proportion of the
references. The other extreme is spiders that try to validate
the references in a set of documents; these usually do not
retrieve any of the links apart from redirections.
problems with spiders.
(2001-04-30)