About our crawler

civilservicejobs.app helps people find Civil Service roles they're suited to and prepare honest applications. To do that, our crawler reads the public job listings on Civil Service Jobs.

How it behaves

• It identifies itself as civilservicejobs.app-bot and links to this page. It never pretends to be a browser.
• It visits once a day, early in the morning UK time, and makes one request at a time with a pause of several seconds between requests.
• It reads the listing and only fetches adverts it hasn't seen before. A typical day is a few dozen requests.
• It follows robots.txt, and stops immediately on any refusal or block. It doesn't retry until a person has reviewed it.
• It doesn't store recruiter names, email addresses or phone numbers.
• We never republish advert text. We show a short summary and always link to the official advert.

Want it to stop, or slow down?

If you run Civil Service Jobs and want the crawler to stop or change its behaviour, add a rule for civilservicejobs.app-bot to robots.txt, or contact us. We'll respond promptly, and we'd welcome the chance to use an official data feed instead.