AI Hiring Matcher
A resume-to-job matcher that audits its own training label before trusting it
The problem
This started as a Datathon submission: a plain classifier over seven categorical fields, with no text signal at all. This is the rebuild. A real retrieval-based matcher replaces the blind categorical classifier, and the training data gets an honest audit before anything gets trained on it.
Architecture
Fairness audit
Matching
The label is gender-biased
Before trusting the dataset's Best Match label as a training target, a fairness audit checks its selection rate by demographic group, directly on the raw data, with no model involved. Best Match is 1 for 61.3% of male rows versus 35.4% of female rows overall. Per job role, the gap is far more extreme and not uniform. Some roles favor men, others favor women, with gaps up to 89 percentage points (Journalist: 6.2% female versus 95.6% male).
Across all 102 (job role, gender) groups, the rates are sharply bimodal: 53 groups sit at 20% or below, 49 at 80% or above, and none fall in between. That's the signature of the label having been sampled per group with a fixed probability near 0.1 or 0.9, not something that reflects actual qualifications. Age, race, ethnicity, experience level, and certifications show no comparable signal. This bias is specific to gender and specific to role.
The direct design consequence: the classifier never receives gender, race, or ethnicity as a feature, even though gender is by far the strongest signal for the label. Training on a protected attribute to predict a match built on that attribute would just reproduce the bias the audit exists to catch.
How the matcher works
Resumes and job descriptions are embedded with sentence-transformers (all-MiniLM-L6-v2, local, no API key) and ranked by cosine similarity against a 51-job catalog. Evaluated as retrieval (does a resume recover its own job role among the 51 known jobs), it gets 63.1% Recall@1, 88.2% Recall@5, and an MRR of 0.745. This is a closed-set problem: ranking among 51 known jobs, not generalizing to job postings the model has never seen.
Example request
Real output, generated from the model trained in this repository.
curl -X POST http://localhost:8000/match \
-H "Content-Type: application/json" \
-d '{"resume": "Proficient in Python, SQL, Machine Learning, with senior-level experience in the field. Holds a master's degree. Skilled in delivering results and adapting to dynamic environments.", "top_n": 3}'{
"matches": [
{"job_role": "Software Engineer", "similarity": 0.564, "skill_overlap": 0.0, "best_match_proba": 0.480},
{"job_role": "AI Specialist", "similarity": 0.530, "skill_overlap": 0.25, "best_match_proba": 0.484},
{"job_role": "Machine Learning Engineer", "similarity": 0.484, "skill_overlap": 0.125, "best_match_proba": 0.487}
]
}The classifier's honest limits
A separate logistic regression predicts Best Match from cosine similarity and skill overlap. Its F1 score on the positive class is about 0.08, barely above chance. That's not a bug to fix by tuning hyperparameters. It's the expected result of deliberately withholding the attribute (gender) that actually drives the label. Improving that F1 would mean feeding the model the protected attribute the audit exists to catch, which defeats the point.
Drift monitoring
A separate check compares real logged requests against a training-time reference built the same way a live request gets scored. It runs a per-column statistical test and refuses to run below 100 logged requests, since drift tests are unreliable on small samples.
Testing
28 tests cover the fairness audit, the matcher's ranking and retrieval metrics, and the prediction pipeline, run with pytest. Ruff and mypy pass cleanly. There's no CI configured, so all of this runs locally rather than on every push.
Limitations
- ▸Closed-set retrieval only. The matcher ranks among the 51 jobs seen during training. It doesn't generalize to job postings it hasn't seen.
- ▸The Best Match classifier is weak by design, see above. That's a deliberate tradeoff, not an oversight, but it means the classifier isn't actually useful for ranking beyond what the retrieval step already does.
- ▸Drift alerts need real traffic. Below 100 logged requests, the check won't run at all.
- ▸No CI. Tests and lint run locally, not automatically on push.