User-agent: * Allow: / Allow: /media/wysiwyg/ Allow: /media/category/ Disallow: /eng Disallow: /esp #Sitemap: ## DEVELOPMENT RELATED SETTINGS ## Do not crawl development files and folders: CVS, svn directories and dump files Disallow: CVS Disallow: .svn Disallow: .idea Disallow: .sql Disallow: .tgz ## Do not crawl Magento admin page Disallow: /admin/ Disallow: /gestion/ ## Do not crawl common Magento technical folders Disallow: /app/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /lib/ Disallow: /pkginfo/ Disallow: /shell/ Disallow: /var/ ## MAGENTO SEO IMPROVEMENTS ## Do not crawl sub category pages that are sorted or filtered. Disallow: /*?dir* Disallow: /*?dir=desc Disallow: /*?dir=asc Disallow: /*?limit=all Disallow: /*?mode* Disallow: /*?___from_store=* Disallow: /*?cat=* Disallow: /*?q=*