Printer Friendly Version Print this thread
Email this thread to a friend eMail this thread to a friend
  • Robots crawling (In: Members Lounge)
  • Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
    • Featured Web Site Template

    Hundreds More at Free Site Templates.com!

    Web Site Partners
    Sponsored Links
    Jet City Software
     
    Whos Here ?
    Reflects user activity within the last 5 minutes
    Moderator(s): Prowler, jcokos
    Member Message

    g1smd
    Staff
    Joined: Jul 28, 2002
    # Posts: 10418

    View the profile for g1smd Send g1smd a private message

    Posted: 2007-Jul-27 01:34
    Edit Message Delete Message Reply to this message

    The strange "hacking" case involving robots.txt, DMCA, The WayBack Machine and a specious lawsuit that could have had wide reaching implications had it succeeded...

    http://www.theregister.co.uk/2007/07/26/wayback_firm_suit/

    Thankfully it failed.



    david68
    Joined: May 16, 2005
    # Posts: 144

    View the profile for david68 Send david68 a private message

    Posted: 2007-Jul-27 14:21
    Edit Message Delete Message Reply to this message

    At least the wayback machine obeys robots.txt (I checked their site and my site is blocked as I disallowed all bots not specified as being allowed). Several bots don't obey it and it's really annoying. I end up blocking them through htaccess regardless on their motive just because their aren't being polite.



    g1smd
    Staff
    Joined: Jul 28, 2002
    # Posts: 10418

    View the profile for g1smd Send g1smd a private message

    Posted: 2007-Jul-27 20:57
    Edit Message Delete Message Reply to this message

    The point here is that for a short time, The WayBack Machine was not obeying robots.txt completely.



    Prowler
    Staff
    Joined: Aug 14, 2000
    # Posts: 1788

    View the profile for Prowler Send Prowler a private message

    Posted: 2007-Jul-28 09:18
    Edit Message Delete Message Reply to this message

    >>The point here is that for a short time, The WayBack Machine was not obeying robots.txt completely.

    For some technical reasons. Good Post g1smd. smile




    Curt
    Joined: Eons Ago
    # Posts: 3735

    View the profile for Curt Send Curt a private message

    Posted: 2007-Aug-12 10:45
    Edit Message Delete Message Reply to this message

    Wow! Guess all these bad bot spiders who don't play by the rules could find themselves in legal hot water some day for not obeying the robots.txt directives.


    You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
    1. You have not yet logged in, or registered properly as a member
    2. You are a member, but no longer have posting rights.
    3. This is a private forum, for which you do not have permissions.

    If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

    If you cannot find this message, click here to Re-Send it.

    If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

    Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

    New posts Forum is locked
    © 1995  ·  iWeb, Inc  ·  DBA JimWorld Productions