Reading robots.txt, fetching the page without JavaScript, checking structured data…