robots.txt parser php class


  1. if(preg_match(“/^$rule/”, $parsed[‘path’])) return false;

    fails if rule=/*/mac/help.mspx

  2. @Deepak,
    thanks for the feedback,

    some shell-wildcards like * are a modifier in php regex, that causes it to fail with a warning. I can use fnmatch(), which handles shell wildcards but it doesn’t work on non-posix systems. do offer a preg_match based replacement but that requires rewriting the class here and there.

    I noticed a few more bugs in it so I am going to put a new version on the blog next week.

Leave a Reply

Your email address will not be published. Required fields are marked *