Printer Friendly Version
Email this thread to a friend
|
Featured Web Site Template |
|
| Member |
Message |
Dinkar
Moderator
Joined: Aug 12, 2001
# Posts: 4298
|
Posted: 06/30/2008 09:57 am
Can you please explain to me how to disallow the page in the robot.txt file.
First of all, the file is robots.txt and not robot.txt
Create a text file and name it as robots.txt
Then add the following code:
Then upload the file to your site in web root directory.
HTH.
|
 |
JimBot
Administrator
Joined: Jul 20, 2003
# Posts: 234
|
Posted: 06/30/2008 04:58 pm
Errr.... Use either the meta tag or the robots.txt file, not both.
[ Message was edited by: g1smd 06/30/2008 05:18 pm ]
|
|
Dinkar
Moderator
Joined: Aug 12, 2001
# Posts: 4298
|
Posted: 06/30/2008 10:53 pm
Why not?
|
|
freeflyer
Joined: Aug 06, 2007
# Posts: 197
|
Posted: 07/02/2008 12:06 am
i havent had experience of pages coming back by simply doing what google recommends.
Request the url removed fro google via the webmaster tools, and then block it with robots.txt . It works for me, and results are gone even after six months. I checked the supplemental and they're not there either. It even works with wildcard requests..
|
|
g1smd
Moderator
Joined: Jul 28, 2002
# Posts: 10178
|
Posted: 07/02/2008 12:30 pm
*** Why not? ***
The robots.txt entry tells the bot to not access the page at all. If it does not access the page, it will never see the meta tag.
Pages that use the meta tag are spidered by Google, but those pages do not appear in the SERPs, if noindex in included.
Pages that are blocked by robots.txt can appear in Google SERPs as a URL-only entry if the URL is found in a link somewhere else.
Yahoo goes one better and creates a title for that blocked entry from the anchor text of the link pointing to the blocked page.
|
|
freeflyer
Joined: Aug 06, 2007
# Posts: 197
|
Posted: 07/03/2008 02:24 am
thats fine if you're using static pages.. unfortunately not so easy on a large dynamic site.
|
|
Dinkar
Moderator
Joined: Aug 12, 2001
# Posts: 4298
|
Posted: 07/04/2008 08:06 am
The robots.txt entry tells the bot to not access the page at all. If it does not access the page, it will never see the meta tag.
Yep, you are right. But bots are programs and sometimes they behave strangely.
It may happen that bots do both - check robots.txt file and read robots meta tag.
Now say, File_A.html is disallowed in robots.txt file and noindex robots meta tag is NOT used in File_A.html
Then bots may get confused as there is no robots meta tag in a file and it means bots can index it.
So it's better to use both. There is no harm using both.
|
|
textlink969
Joined: Jul 06, 2008
# Posts: 25
|
Posted: 07/06/2008 10:53 am
just submit a new site map through google site map
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
