Every month, Skymem robot crawls milions of web pages focus on business data.
The robot only analyzes public web pages and navigates at a slow pace. It never visits more than 1 page every 5 seconds for a website to make sure other visitors aren’t slowed down by the crawling.
Yes. The robot strictly respect robots.txt, both disallow and allow rules If for whatever reason you wish to disable it on your website, add the two following lines into to your robots.txt file:
User-agent: SkymemWebCrawlerIf you think that Skymem robot is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact us at info@skymem.com.