As Paul explains it, spidering “is when a computer scours the Web, retrieving information about the pages and related links it finds, then stores the information in a database” that is used to build results for user search queries.

continue reading.....