PtaKaroBot

What it fetches

Pages of websites on our source list, such as a brand's spec pages or a shop's product pages; the nightly crawl reads nothing else. A run we start by hand may also read another website we allow for that run alone, such as the Internet Archive (web.archive.org) for saved copies of those pages. It does not crawl the web at large, follow links to other websites, or copy images.

A fact counts only when it comes from the website it is credited to: if a page redirects to another website, no fact is taken from it. What it reads is turned into prices and specs by fixed rules, each kept with the time it was read and an address on the website it is credited to (the page it was read from, or another page of that website, such as the product's own page), or, for a saved copy at the Internet Archive, the copy's address and the time the Archive saved it; see how we check prices.

How it behaves

• It reads robots.txt again in every crawl job (the night's crawl runs a separate job for each source it reads), before it asks your website for a page. A single page we fetch by hand, outside a crawl job, may use a copy read in the last 24 hours; it never uses an older one. It fetches a page only if the rules allow it for PtaKaroBot or *. It groups the rules as RFC 9309 says: a Crawl-delay or Sitemap line between User-agent lines does not split the group.
• It sends one request at a time to a website, at least 3 seconds apart; once a crawl job has read your robots.txt, its requests are also at least your Crawl-delay apart. If the Crawl-delay is longer than 120 seconds, that crawl job reads only robots.txt and asks for no page.
• It retries a failed request at most twice. If you answer 429 or 503, it waits as long as your Retry-After header asks, up to 60 seconds, before it asks your website again. If you keep answering 429 or 503 after that, or ask it to wait longer than 60 seconds, no crawl job on our crawl server asks your website again until the next night's crawl starts, at 02:30 Pakistan time. A longer Retry-After is honoured in full: it does not come back before the time you gave.
• If you refuse a request (HTTP 403, or a firewall's block page) or answer it with a bot check (a CAPTCHA or a browser test), no crawl job on our crawl server asks your website again until the next night's crawl starts, at 02:30 Pakistan time. That includes the jobs for our other sources on the same website.
• It sends If-None-Match and If-Modified-Since, using the ETag and Last-Modified from its last visit, so that unchanged pages can answer 304.
• It follows redirects itself, checks robots.txt again for each new address, and never follows a redirect from https to plain http.

It never routes around a refusal, a bot check or a rate limit: it uses no fetching service, proxy or other user agent to get past one, and a page it could not read is reported to us and left out. After a refusal or a bot check, the next night's crawl asks once more; to keep it away, disallow PtaKaroBot in robots.txt (below) or write to us.

How to limit or block it

Add a group for PtaKaroBot to your robots.txt, for example:

User-agent: PtaKaroBot
Crawl-delay: 10
Disallow: /checkout/

To block it completely, use Disallow: /. Its next crawl job reads the change before it asks your website for a page; a single page we fetch by hand may still go by a copy of your robots.txt read up to 24 hours earlier.

Contact

To ask us to stop, to change how often we visit, or to send us a price feed instead: hello@ptakaro.com.