Printer Friendly Version
Email this thread to a friend
|
7yr old PR4 Website, 300$ per month (In: I Want to Sell My Website)
50 Partners Needed For Media Trading Website (In: I Want to Sell My Website)
Established website sale:78,000$/y profit:2100$/m (In: I Want to Sell My Website)
webmasters seo tools website is for sale (In: I Want to Sell My Website)
Website for Sale (In: I Want to Sell My Website)
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
godwinseo
Joined: Sep 24, 2005
# Posts: 2
|
Posted: 2005-Sep-24 14:19
Hi
I need 2 create a robot.txt file for a site woth 40000 web pages.how do i do that?is there any tools that you wud recommend?plz help me and its very urgent
thank u
godwin
|
 |
Logan
Joined: Aug 14, 2002
# Posts: 3749
|
Posted: 2005-Sep-24 15:49
for starters its robots.txt
Do you want to exclude all 40000 web pages from all of the search engines? If so this can simply be done at the root level. You can also specify exclusion by directory or individual url.
It is not necessary to exclude a whole site by individual url, so defining what you actually want to do is necessary.
Its just a .txt file so its not a very popular function as you describe for 'automated tools'.
|
|
godwinseo
Joined: Sep 24, 2005
# Posts: 2
|
Posted: 2005-Sep-24 17:36
Hi Logan
thanks for ur kind interest.the scenario is like this.for small sites i just manually check the pages indexed by google and see if there are any pages that are not working anymore.Then i include such pages in robots.txt file.Now how do i do this for a site which has 40k pages.i certainly cannot afford to do it manually.so wats the way to do this?
plz help me out of this.
thank u
godwin
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10446
|
Posted: 2005-Sep-24 21:28
Google only reads the first fifty lines in the robots.txt file.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
