RoleLens
An open-source agent that reads every new job ad in Sweden and sends me the few worth applying to.
I built RoleLens for my own job search. Good roles often hide behind titles I would never search for, so RoleLens reads the whole ad, compares the work with my profile and sends me only the roles worth a closer look.
It reads every new vacancy on Platsbanken and on eight kinds of company career sites. Ranking and rule checks come first and cost almost nothing, so a model only judges the ads that are left. It runs on a schedule on a small Linux server and is open source under the MIT licence.
In its first month, RoleLens and two companion scouts read about 37,000 job ads and opportunities and sent me 685.
Visit github.com/Aminhashemi-su/RoleLens
The project
Job alerts match titles and keywords. They send you roles that happen to contain the word "engineer" and miss the one that describes your work under a different name. RoleLens starts from a structured profile, reads the full advertisement and asks two separate questions: is this the kind of work I want, and can I realistically get this job?
What I built
- Discovery from Platsbanken's JobStream feed plus built-in collectors for Teamtailor, Varbi, Greenhouse, Lever, Ashby, SmartRecruiters, Workday and SAP SuccessFactors. If one source fails, the others still run.
- Ranking before any AI call, using role vocabulary, Gemini embeddings and JobTech skill data, combined with reciprocal rank fusion. The top 15% goes on, plus a random sample so ranking mistakes stay visible.
- Rule checks that drop clear mismatches, such as people-manager roles or mandatory fluent Swedish, before a model sees them.
- An AI judge (Gemini on Vertex AI) that scores batches of up to 10 ads against a strict JSON schema. GPT-5 mini on Azure OpenAI answers only when Vertex AI cannot. The model scores; plain Python rules make the final decision.
- Two scores per job. Career Fit rates the work itself. Opportunity rates whether this vacancy is realistic now. A 95/30 result means the work is right but something in the ad blocks me.
- 216 offline tests that run on Linux, macOS and Windows in GitHub Actions, with no network or API key.
How it runs
One scheduled run, one lock and one SQLite database. Each ad is stored once with a content fingerprint, so an unchanged ad is never paid for twice and an edited one is judged again. A run with no matches and no problems sends nothing; a failed source or an exhausted budget always sends an alert.
Problems I hit and fixed
- A 200 OK that was a failure. One provider returned valid JSON for only 3 of 20 jobs. RoleLens now requires every job ID back exactly once, keeps missing jobs queued and retries once. That provider was dropped.
- Scores that would not repeat. The same input moved one score by 20 points. Jobs near the notification line are now judged twice and decided on the average.
- Bugs in my own code. Replaying stored AI answers through the real decision code found two errors in my Swedish-language rules. Rule changes are now replayed before every release.
- Keyword search hid the best jobs. On one day, tuned searches found only 2 of 20 relevant roles. The full JobStream feed replaced search, and ranking decides what gets read.
Choosing the model by measurement
The provider choice is a public benchmark in the repository, not a preference. Expected answers are labelled before any model runs, and stored answers can be replayed through the real decision code.
- Round 1 removed a provider from automatic routing after it returned only 3 of 20 job IDs behind a 200 OK.
- Round 2: Gemini beat GPT-5 mini on weighted error (5 vs 8) and ranking accuracy (0.93 vs 0.78), at about 2.7 times the cost. I chose Gemini for recall, because a missed good job costs more than the extra AI spend, and GPT-5 mini became the fallback.
Tuning with real data
- Keeping the top 15% of ranked ads kept 101 of 102 known good matches.
- A cheap first-read model that only rejects clear mismatches settled about 75% of the ads it saw without losing a match. AI cost per ad fell from $0.0037 to $0.0010.
- Two identical re-judgements disagreed on 6 of 40 matches, all close to the threshold. That set the band where jobs are judged twice.
The first month in production
RoleLens runs next to two companion scouts on the same server: one for a second job profile and one for internships and thesis projects. These numbers come from their databases for 29 August to 24 September 2026.
- About 37,000 job ads and opportunities stored, counting ads that two scouts share once.
- About 12,300 AI evaluations. Rule-based rejections and results reused from another scout are not counted.
- 685 roles sent to me, under 2% of everything read.
- About $11 of recorded AI spend for RoleLens and the internship scout, about $0.0012 per evaluation in RoleLens. The third scout logs tokens, not dollars.
- 2,393 scheduled runs with no one watching. 96.8% finished fully and 8 failed outright, most of them in the first days.
- 356 employer job feeds checked with about 105,000 HTTP requests, to find roles that never reach Platsbanken.
A production run
45 ads selected and 45 judged in 5 batches, with no fallback calls and nothing left unresolved. The run took about 3 minutes and 24 seconds and sent 3 matches to Telegram, so I read 3 ads instead of 45.
What I would do differently
The ranking cut and first-read thresholds were tuned on one profile's data. They work as defaults, and the README says so. Before RoleLens serves many people, each profile needs its own calibration.
Stack
- Language
- Python 3.11+, standard library only at runtime
- State
- SQLite (jobs, rankings, evaluations, retry queue, cost records)
- Job sources
- Arbetsförmedlingen JobStream and JobSearch, plus eight career-site platforms
- Ranking
- Role vocabulary + gemini-embedding-001 + JobTech JobAd Enrichments, reciprocal rank fusion
- First read
- Gemini 2.5 Flash-Lite on Vertex AI
- Judge
- Gemini 3.8 Flash on Vertex AI, strict JSON schema
- Fallback
- GPT-5 mini on Azure OpenAI, same schema, only when Vertex AI cannot answer
- Decision
- Deterministic Python rules
- Delivery
- Scheduled on a Linux VPS, delivered to Telegram
- Quality
- 216 offline tests, public replay benchmark, CI on Linux, macOS and Windows. Version 2.0