Re: [PHPLIB] Problems when submiting a PHPLib site to Altavista From: garyb <email protected>
Date: 01/27/00

I've checked out this url. I don't think it will solve my problem because
robots.txt will act on directories and files . What i want is to avoid to be
indexed with a session id in my url...

Generally, but not certainly, search engines will not index URLs with a GET
variable string. Altavista seems to be the major exception.

Look at the REMOTE_HOST if you are doing DNS lookups. Altavista at least is
up front about their domain name. You might also look at the
HTTP_USER_AGENT. Some search engines spoof Mozilla there, others don't.

Many sites maintain a list of search engine IP addresses, and handle them
specially - such as feeding them a special index page, or feeding them a
nothing page, depending on the desired result. Of course, the search engines
switch their IP addresses around regularly to avoid this.

There is a running technology war going on between those who want to abuse
the search process, and the search engines. The rest of us are caught in the
middle. Ref: http://www.searchengine-news.com/ I used to get their
newsletter, and it was quite informative of what submission criteria were
most effective in a given month.

G
-
PHP3 Base Library Mailing List. Send messages to <phplib <email protected>>.
To unsubscribe, send "unsubscribe" to <phplib-request <email protected>> in
the body, not the subject, of your message.