If spocrawl showed up in your logs, here's what it was doing.
It's a research crawler run by August SEO in Spokane. It reads the websites of Spokane-area businesses for research on the local business web, and to find the local businesses spokaneseo.app checks.
It reads your pages, and nothing else
Your robots.txt, your sitemap, and the pages of your site, starting from the homepage. It reads the HTML and skips images, PDFs and downloads.
When a page writes its words with JavaScript, spocrawl may open that page in a headless browser, as a visitor's browser would. The page's scripts and stylesheets load; images, video and fonts don't. That happens for a few pages of a site at most.
It never logs in, submits a form or adds anything to a cart. It reads at most 300 pages of a site per visit, and most small business sites have far fewer.
What it records are the business facts a site publishes about itself: the name, contact details, services, the area it serves, and what the site is built with.
It takes its time
One page at a time per site, at least a second and a half apart. If your robots.txt sets a Crawl-delay, it waits that long instead, up to 30 seconds.
It comes back in passes, usually about once a month. On a repeat visit it sends If-None-Match and If-Modified-Since, so a page that hasn't changed costs your server almost nothing.
It does what your robots.txt says
It follows a User-agent: spocrawl group if your robots.txt has one, and the * group if it doesn't.
It won't follow the links on a page that carries either of these tags:
<meta name="robots" content="nofollow"> <meta name="spocrawl" content="nofollow">
If your robots.txt answers with a server error or can't be reached, it leaves your whole site alone that visit.
How to make it stop
Add this to your robots.txt. It takes effect on spocrawl's next visit:
User-agent: spocrawl Disallow: /
Or ask here. No email needed, just the website:
How to recognize it
Every request carries this user agent:
spocrawl/0.1 (+https://spokaneseo.app/bot/spocrawl)It runs from one machine on a home connection in Spokane, so its IP can change.
Match on the user agent.