February 1st, 2006
The googlebot can find pages not directly linked
Dell’s confidential specs for future Dell notebooks were discovered and distributed by Google. Elinor Mills from News.com here writes a web primer on how to avoid the problem by using a robot.txt file and by not linking to sensitive materials.
However, did you know that the Googlebot can find pages that have no direct link from the home page of a web site? That’s what a Google engineer said at a search conference a couple of years ago.
If I find the reference I will post it, but the gist of it was that Google has the technology so that it can find and catalog information on a server without having to follow links.
And you would expect Google to have such technology since its mission is to index and copy all of the world’s information. Not "all linked information" but "all information."
- - -
Here is an interesting discussion on this topic.
Tom Foremski reports on the business and culture of Silicon Valley at the intersection of technology and media. He also writes at Silicon Valley Watcher. See his full profile and disclosure of his industry affiliations.
Subscribe to Tom Foremski: IMHO via Email alerts or RSS.
Follow on Twitter:
| 02/04/06
