ShaveIndexBot
ShaveIndexBot is the crawler of ShaveIndex, an independent database of wet-shaving products: shaving soaps and creams, aftershaves, colognes, razors, brushes and blades. It reads the public product information that manufacturers publish in their own online shops, so that ShaveIndex can list their products and link back to the manufacturer.
Who operates it
ShaveIndex · Roman Pustina · Währinger Straße 99/8, 1180 Vienna, Austria
Contact: contact@shaveindex.com · Legal notice
The catalogue is public at shaveindex.com.
How to recognise it
User-Agent: ShaveIndexBot/1.0 (+https://shaveindex.com/bot; contact@shaveindex.com)
Every request is signed with Web Bot Auth
(HTTP Message Signatures, RFC 9421, Ed25519) and carries Signature-Agent: "https://shaveindex.com".
Our public key directory: https://shaveindex.com/.well-known/http-message-signatures-directory, key ID 8D-Y345yPww6k5px2qqiO77licAzq-nYMB-QrH5GGvk.
What it reads
- Manufacturers' own shops only. Retailers' shops are not visited unless the retailer has agreed.
- Public product data: product name, description, ingredients, sizes and variants, price and availability, and the product image.
On Shopify stores this is the public product list (
/products.json), on WooCommerce the public Store API, on other shops the product pages listed in the shop's sitemap. Where a shop shows ingredients only on the product page, that page is read too, and the brand's “About” page once for a short description of the brand. - Never: customer accounts, carts, checkout, forms, logins or anything behind a password. It does not try to get around any protection.
How often
- One request at a time, at least 1.5 seconds apart.
- A shop's product list is read between every 6 hours and every 14 days, adapting to how often the shop actually changes – most shops are read every few days and need one to three requests.
- Product images are downloaded once; links to product pages are checked now and then so that ShaveIndex never links to a page that is gone.
- It reads
robots.txtand follows it, includingCrawl-delay, and it waits when a server answers429or asks it to withRetry-After.
How to stop it
Add this to your robots.txt, or write to us – we remove a shop on request:
User-agent: ShaveIndexBot Disallow: /
Deutsch
ShaveIndexBot ist der Crawler von ShaveIndex, einer unabhängigen Datenbank für Nassrasur-Produkte. Er liest nur die öffentlichen
Produktangaben in den eigenen Shops der Hersteller, höchstens eine Anfrage alle 1,5 Sekunden, hält sich an robots.txt und wartet, wenn ein
Server darum bittet. Jede Anfrage ist mit Web Bot Auth signiert. Ausschluss über robots.txt (siehe oben) oder per Mail an
contact@shaveindex.com.