No Must configure a customized search ahead of the crawl. You are able to do all sorts of searches in real-time and Look at code or text modifications in-between crawls. Check if robots.txt timed out or blocked URLs. Also, confirm if a URL was in the sitemap or if AhrefsSiteAuditBot https://www.botsaas.io/ai-chatbot-advantages/