FAMMRO Crawler for website operators

Shops use FAMMRO as the assistant in the chat on their website

For this the crawler reads websites a shop enters in its console, usually its own

The pages become the knowledge the assistant uses to answer customer questions

The content is stored only in the knowledge of that one shop

On its own it reads a website again at most every 6 hours

It identifies itself the same way in every request

User-Agent: Mozilla/5.0 (compatible; FAMMRO-Crawler/1.0; +https://fammro.com/crawler)
From: support@fammro.com

In robots.txt its name is FAMMRO-Crawler

Before every run it reads the robots.txt of your website

It follows the group for FAMMRO-Crawler, or if there is none, the group for all crawlers

It does not fetch disallowed addresses, including sitemaps and shop APIs

It observes Crawl-delay, up to 2 seconds between two requests

It caps longer Crawl-delay values at 2 seconds

It starts with 6 simultaneous requests

If your server returns 429 or 503, it fetches fewer pages at once

It loads up to 10 MB per response and 30 MB per sitemap and stops at anything larger

These lines in your robots.txt block it from your whole website

User-agent: FAMMRO-Crawler
Disallow: /

To block single areas, add lines of their own to the same group

User-agent: FAMMRO-Crawler
Disallow: /checkout
Crawl-delay: 2

New rules apply from the next run

Questions and notes about the crawler to support@fammro.com

Please include your domain and when the requests happened