# Allow all crawlers access to the entire site User-agent: * Disallow: # Block all web crawlers from the /private/ directory User-agent: * Disallow: /privacy/ /terms/ /admin/