How we evaluate
Every job on botjobs has passed two separate assessments by a language model. What the model is asked, how we check it, why it holds up, and how it keeps pace with new models.
An AI suitability score from 0 to 100 for the task profile in the job ad: how much of the advertised core work a frontier AI agent could carry out today.
Not a measured automation rate, not the share of a person that can be replaced, and not proof that the position will disappear. A score of 80 does not mean “80 % replaceable”.
What is assessed
We assess the core tasks described in the job ad itself: title, description, tasks and requirements, as published by the employer. We do not rate job titles or occupations in general. Two jobs with the same title can get very different scores if the work behind them differs.
The yardstick is a frontier AI agent as it exists today: expert-level language understanding, tool access to e-mail, documents, the web, databases and code execution, but no physical body and no legal personhood.
Stage 1: automation score
The model rates each job from 0 to 100 and gives exactly one sentence of reasoning that names the decisive task. That sentence is shown on every job.
Core tasks are fully digital and language-based: writing, research, entering, checking and analysing data, translation, correspondence, standard case work, reporting, programming. An agent could do most of it on its own.
Mostly digital work, with parts that need human judgement, taking responsibility or personal coordination (negotiations, leading a team, client meetings).
Mixed job: a relevant digital core, but substantial parts need physical presence, on-site appointments or legally protected activities.
The core needs physical presence, manual work, care or treatment, driving, lab work or point-of-sale selling, or a profession an agent cannot legally hold.
Fixed rules on top of the scale
- If the description is empty or says nothing, the model rates conservatively by title and may give at most 60.
- Legal licensing or liability requirements (admission to the bar, medical licence, chartered engineer, statutory audit) cap the score at 69, however text-heavy the work is.
- If regular physical presence is the core of the job (reception, warehouse, production, care, field service), the score is capped at 30.
- Only jobs scoring 70 or higher go on to stage 2.
Stage 2: role check
A high score says the tasks are digital. It does not yet say an agent could hold the job. So a second, independent pass asks a single question: could an autonomous agent fill this role as a whole?
The answer is "no" if the role at its core depends on:
- sales, customer relationships, account management or acquisition, even in the back office and even if most of it runs through e-mail and CRM
- leading people or carrying disciplinary responsibility
- liability, signing authority, a legally protected profession (tax advice, auditing, legal advice, medicine) or representing the company externally
- physical presence: field service, on-site appointments, lab, production, warehouse, reception, care, driving
- negotiating with suppliers or partners
- looking after, training or instructing people
The answer is "yes" only if the core of the role is processing information and a human would merely approve or sign off: case work, bookkeeping, data maintenance and analysis, reporting, documentation, writing, translation, research, programming, testing. The rule for doubtful cases is fixed: if in doubt, no.
Only jobs that pass both stages are listed. Today that is 2,032 jobs. Jobs that fail are hidden, never deleted, and paid premium ads of the source portals are never touched.
How results are checked
Identity check in every batch
Jobs are assessed in small batches. For every job the model must echo the beginning of its title next to the ID. If a single echo does not match, the whole batch is discarded and assessed again, so a score can never end up on the wrong job.
Strict output format
The model must return exactly one result per job in a fixed JSON format. Anything else counts as a failure and is retried; nothing is filled in by guesswork.
Content fingerprint
Each assessment is stored with a fingerprint of the ad text and the model that produced it. A job is reassessed only when its text changes, and the same job published on several portals is assessed once.
Two independent questions
Score and role check use different instructions and are run separately. A job must convince twice.
Human spot checks
Borderline groups (for example back-office roles close to sales, or controlling) are reviewed by people, and systematic mistakes are corrected across the whole set.
Published reasoning
Every listed job shows its score and its one-sentence reasoning, and all of it is available as open JSON. Anyone can check a single verdict and tell us when it is wrong.
Why this is plausible
The approach follows the method that established research on AI and work uses: break jobs down into tasks and rate how exposed each task is to language models. The best-known example is "GPTs are GPTs" (Eloundou, Manning, Mishkin and Rock, 2023), which rated occupational tasks with both human annotators and GPT-4 and found that the model's ratings largely agreed with the humans'. The International Labour Organization uses a similar task-based method with a language model in its global studies on generative AI and jobs.
botjobs goes one step closer to reality: it rates concrete, currently open job ads instead of occupation averages, so the result reflects what employers actually ask for today.
The rules are deliberately conservative: hard caps for physical work and legally protected professions, a separate role check, and "if in doubt, no". The listed share is therefore a lower bound rather than an inflated headline: of all jobs analysed, only a small part ends up on botjobs.
Every verdict comes with its reason and the data is open, so the method can be checked from the outside, job by job.
Keeping pace with new models
Models improve every few months. A score that was right in spring can be too low in autumn. This is how botjobs deals with that.
- The yardstick is defined as "a frontier agent today", not as a fixed model. When the frontier moves, the bar moves with it.
- Every score records the assessing model and the fingerprint of the ad, so old and new assessments can be told apart and compared.
- The assessment itself is done by a fast, smaller model (Claude Haiku), steered by the rules above. That keeps tens of thousands of assessments affordable; the strict rules and the second stage make up for the smaller model.
- A fixed reference set of jobs, reassessed with every new model and every rule change, so that drift becomes visible as a number instead of a feeling.
- A version number for the rules on every score, published with the data.
- Regular full reassessment of all listed jobs with the current model, and a second model as cross-check for borderline cases between 60 and 80.
- A measured monthly series of the AI quota, built from these runs, to replace the modelled months shown so far.
What the score does not say
The score rates whether an AI agent could technically do the core tasks today. It is not a forecast that the job will disappear, and it does not say whether an employer will actually use an agent; cost, law, trust and organisation decide that.
The assessment is made by a language model. It is a strong, transparent indicator, not a verdict. Borderline cases can tip either way between two runs; that is one reason for the planned reference set.
Spotted a wrong verdict? Write to hello@botjobs.org