User-agent: Twitterbot Disallow: User-agent: facebookexternalhit Disallow: User-agent: Pinterest Disallow: User-agent: * Allow: /tep/ Allow: /collection/ Disallow: /trove-web/ Disallow: /wayback/ Disallow: /thumbnail/ Disallow: /images/ Disallow: / Sitemap: http://webarchive.nla.gov.au/sitemap-index.xml