I am working with a client who is working with me and another agency. I recommended to the client that since their campaign landing pages are being used for generating leads and since there may be issues with the content having little value or even potentially duplicate  content,  they create a folder for me to place their landing pages in that we would then disallow bots to crawl these pages within a folder using the robots.txt file. The other agency came back and said they recommend against using a robot.txt file for this and we should just let the bots crawl those pages. I then decided to do some investigating into this and found out that even Google gives two opinions, in one place Google says use a robots.txt file and in yet another place says no do not use a robots.txt file.

Quote from this url:
http://www.google.com/support/webmasters/bin/answer.py?answer=66359
“Google no longer recommends blocking crawler access to
duplicate content on your website, whether with a robots.txt file or other
methods.” ……… A better solution is to allow search engines to crawl
these URLs, but mark them as duplicates by using the rel=”canonical”
link element

Then on a different page Google says this:

Quote from this url:
http://www.google.com/support/webmasters/bin/answer.py?answer=35769
“Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.”

OK Google what is it to be? Do we use the robots.txt file or do we not use the robots.txt file for a situation like this? I have found myself a bit more confused than I was before and am going to continue to look for some answers from Bingahoo to see how they handle all of this robots.txt and canonical url stuff. If anyone else has any recommendations on what they feel is the best way to handle content that may have little value or could cause duplication issues please leave feedback