How to implement robots.txt / sitemap.xml / crossdomain.xml in Adobe CQ / AEM

Use Case: 

• Some time you want to implement robots.txt or any web related configuration in CQ.
• Some time you need to have different configuration of robots for different environment 

Solution: 

You can directly created web related configuration in CQ. For that do following,

1) Go to CRXDE or CRXDE light, Or you can directly put them in your CVS under jcr_root folder. You can create different version of robots.txt based environment and domain name.

Then you can configure sling rewriter (org.apache.sling.jcr.resource.internal.JcrResourceResolverFactoryImpl.xml) under /apps/sling/config to redirect to correct robots.txt or any site specific configuration. More information about configuration can be obtained from here http://www.wemblog.com/2012/10/how-to-work-with-configurations-in-cq.html

For Prod something like resource.resolver.virtual="[/robots.txt:/robots-prod.txt] and for all other env

 resource.resolver.virtual="[/robots.txt:/robots-qa.txt]"

For site specific configuration you can use something like

/apps/map.publish/www-robots.txt

     jcr:primaryType = "sling:Mapping" (that's the type when you create a new node)

     sling:internalRedirect = "/content/robots-prod.txt"

     sling:match = "http/www.SITE.com/robots.txt"

You might have to do some tuning on dispatcher for make this work. Feel free to ask any question.