spider, wanderer

a computer program that prowls the internet looking for publicly accessible resources that can be added to a database; the database can then be searched with a search engine