JobsDogBot.

What it reads.

• The public feed of open jobs that an employer's hiring software publishes for its careers page, and nothing else on the site.
• Job postings only: the title, the employer, the place, the pay where the employer lists it, the dates, the link and the description. It keeps no separate field for a recruiter's name or contact; a description is kept as the employer wrote it.
• It never signs in, fills in a form, applies, sends cookies or follows a redirect.

How often.

• Each employer's listing is checked on its own schedule: at most six times on a weekday when its jobs changed in the past week, twice on a quiet weekday, and once a week when nobody on Jobs.Dog searches where its jobs are. Checks are spread out, never on the hour.
• It asks only for what changed since its last visit (If-None-Match and If-Modified-Since), so an unchanged listing costs one short answer.
• A scheduled run makes at most two requests at a time to one host, and one at a time to a host whose robots.txt sets a Crawl-delay, which it waits out up to 60 seconds. Some hosts get a pause of a second between requests even when their robots.txt names none. When a person opens a job whose description it did not keep, it reads that one listing's feed again, under the same host rules.
• It honours Retry-After and waits longer after each failure, up to a day. A host that refuses three requests in a row is left alone for 24 hours.
• A listing that answers not found for 7 days, or has had no open job for 90 days, is no longer read.

What it does with them.

Jobs.Dog shows a job to the people whose search it matches, with a link to the employer's own posting, where they read more and apply.

How to ask it to stop.

Add a group for it to your robots.txt. JobsDogBot reads it once a day per host and follows the group that names it, or the * group when none does:

User-agent: JobsDogBot
Disallow: /

A change to your robots.txt reaches it within a day, and a listing it may not read is left alone for a day before it looks at your robots.txt again. If your robots.txt cannot be read because of a server error, JobsDogBot stays away for an hour and tries again.

Or write to us. By hand, we stop reading your listing and delete the jobs we hold from it. A job a person already saved to their own Jobs.Dog stays theirs.