Well a Search Engine is a service
that uses robots or spiders. What these programs do is continually
"crawl" the web looking for new or updated pages. A
real surfers dream! but they go from URL to URL until they have
visited every site on the Internet.
At the site, an agent records the
full text of the pages (home and sub pages) within the site. sounds
like alot of work but it don't stop there! it then go and searches
every external link and follows them to. This how the Search Engine
can find your site
Following these external links is
how search engines are able to find your site regardless of whether
or not you register your URL with them. Submitting your URL, however,
does speed up the process. It notifies an agent to visit and index
your site instead of waiting for it to eventually locate you through
one of your external links.
Robots will then revisit your site
periodically to refresh the recorded information. The revisiting
of links is the reason why some search engines don't require you
to inform them of dead links. Eventually, their robot would try
unsuccessfully to update the information on a dead link and realize
it no longer exists.
Finally, an easy way to tell whether
a Web index is a search engine as opposed to another type of directory
is by the information it requires when adding your URL. A true
search engine will only need the Web address. The indexing agent
takes care of the rest.
