Add Kaggle silver training data, retrain model, reorganize data directory

- Add 2,483 Kaggle resumes with Gemini-extracted labels + BIO tagging + noise augmentation
- Retrain model (25 epochs): entity F1 97.77%, clean F1 99.18%
- Reorganize training/data/ into sources/, gold/ subdirectories
- Add implementation guide (docs/implementation_guide.md)
- Remove one-off scripts (build_long_resumes, convert_dataturks, label_resume_pdfs)
- Fix Windows UTF-8 encoding in structured_postprocess and train_ner
- Add --extra-train flag to train_ner.py for loading additional training files
- Export updated ONNX + quantized model

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

README.md

@@ -35,12 +35,12 @@ Fine-tuned DistilBERT model for extracting structured information from resume te
 
 ## Latest NER benchmark
 
-Latest retrain on RTX 3080 with noise-augmented data (3x augmentation, 25 epochs):
+Latest retrain on RTX 3080 with noise-augmented data + Kaggle silver labels (25 epochs):
 
-- entity F1: **97.62%**
-- structured micro F1: **98.16%**
-- long resume micro F1: **91.7%** (resumes >512 tokens)
-- noisy resume F1: **72.67%** (OCR/scraped text)
+- entity F1: **97.77%**
+- structured micro F1: **97.88%**
+- clean resume F1: **99.18%**
+- noisy resume F1: **69.24%** (OCR/scraped text)
 
 These numbers come from entity-level exact-match evaluation with `seqeval` and the structured extraction benchmark.
 
@@ -127,6 +127,8 @@ Section detection (`training/section_detector.py`) fills entities the NER model
 
 All post-processing rules are config-driven for cross-language portability.
 
+For a detailed implementation walkthrough of the full pipeline, see [`docs/implementation_guide.md`](docs/implementation_guide.md).
+
 ## Training data
 
 Current checked-in dataset snapshot in `training/data/` was built from mixed sources:
@@ -136,14 +138,25 @@ Current checked-in dataset snapshot in `training/data/` was built from mixed sou
 - 12 manual resume templates across tech and non-tech domains
 - 50 hand-crafted long resumes (>512 tokens) for chunked inference training
 - 93 gold-labeled resume-resource PDFs (hand-annotated, 100% match rate)
+- 2,483 Kaggle resumes with Gemini-extracted silver labels + BIO tagging
 - 2x noise augmentation for OCR robustness (separator swaps, char corruption, case changes)
 
+Data layout:
+
+```
+training/data/
+├── ner_train.json          # Main training set
+├── ner_val.json            # Validation split
+├── kaggle_train.json       # Kaggle BIO training (silver + noise-augmented)
+├── gold/                   # Hand-annotated evaluation data
+├── sources/                # Raw source data (CSV, JSONL, etc.)
+└── resume_resource/        # Source PDFs
+```
+
 Current public rebuild path uses:
 
 - `training/generate_from_structured.py` — orchestrates all sources + augmentation
-- `training/convert_dataturks.py` — DataTurks converter
 - `training/manual_resumes.py` — short manual templates
-- `training/build_long_resumes.py` — 50 long dense resumes
 - `training/build_gold_labels.py` — gold-labeled PDF resumes
 - `training/noise_augment.py` — noise augmentation for OCR robustness
 
@@ -191,11 +204,10 @@ python -m training.benchmark_structured --model-dir .
 
 Latest internal structured benchmark on current validation set:
 
-- overall micro F1: **98.16%**
-- macro field F1: **98.65%**
-- clean-resume micro F1: **99.31%**
-- noisy-resume micro F1: **72.67%**
-- long resume (>512 tokens) micro F1: **91.69%**
+- overall micro F1: **97.88%**
+- macro field F1: **98.47%**
+- clean-resume micro F1: **99.18%**
+- noisy-resume micro F1: **69.24%**
 
 This benchmark uses in-repo structured post-processing for both gold spans and model predictions. Section-aware chunked inference handles resumes exceeding the 512-token context window. These numbers are intended for internal regression tracking, not external leaderboard claims.
 
@@ -240,9 +252,9 @@ Full details: [`docs/export.md`](docs/export.md)
 
 Current ONNX export was validated against PyTorch on same sample input:
 
-- PyTorch vs ONNX logits: `allclose=True`
+- PyTorch vs ONNX logits: `allclose=True` (max diff: 0.000013)
 - PyTorch vs ONNX predictions: `argmax_equal=True`
-- PyTorch vs quantized ONNX predictions: minor diff on punctuation tokens (expected for INT8)
+- PyTorch vs quantized ONNX predictions: minor diff (expected for INT8)
 
 This is main safety check when training and export happen in separate environments.
 
@@ -250,10 +262,8 @@ This is main safety check when training and export happen in separate environmen
 
 - `training/train_ner.py` — training loop
 - `training/generate_from_structured.py` — dataset builder orchestrating all sources + augmentation
-- `training/convert_dataturks.py` — DataTurks converter
-- `training/manual_resumes.py` — short manual templates
-- `training/build_long_resumes.py` — 50 long dense resume templates (>512 tokens)
 - `training/build_gold_labels.py` — gold-labeled PDF resume builder
+- `training/manual_resumes.py` — short manual templates
 - `training/noise_augment.py` — noise augmentation for OCR robustness
 - `training/labels.py` — shared label definitions
 - `training/dataset_utils.py` — dedupe / split / manifest helpers
@@ -262,13 +272,15 @@ This is main safety check when training and export happen in separate environmen
 - `training/structured_postprocess.py` — config-driven post-processing and grouping
 - `training/section_detector.py` — section-aware entity extraction
 - `training/benchmark_structured.py` — structured benchmark with chunked inference
+- `training/analyze_structured_errors.py` — per-resume error analysis
 - `training/synthetic_assets.py` — synthetic source assets/helpers
 - `training/synthetic_formats.py` — 8 resume format builders (A-H)
-- `training/label_resume_pdfs.py` — PDF extraction and auto-labeling
 - `training/export_onnx.py` — CLI wrapper for ONNX export
 - `training/validate_onnx.py` — PyTorch vs ONNX parity check
 - `training/quantize_onnx.py` — ONNX quantization helper
 - `training/requirements-export.txt` — separate ONNX export env
+- `run_inference.py` — single-resume inference script
+- `docs/implementation_guide.md` — detailed pre/post processing implementation guide
 - `docs/export.md` — ONNX export and validation notes
 
 ## Limitations

docs/implementation_guide.md

@@ -0,0 +1,347 @@
+# Resume NER: Pre and Post Processing Implementation Guide
+
+This document explains the full inference pipeline from raw resume text to structured output, covering all pre-processing, model inference, and post-processing steps driven by `resume_config.json`.
+
+## Pipeline Overview
+
+```
+Raw PDF/Text
+    |
+    v
+[1. Pre-processing]  ← resume_config.json → pre_processing
+    |
+    v
+[2. Tokenization]    ← distilbert-base-cased tokenizer
+    |
+    v
+[3. NER Inference]   ← DistilBERT token classification (27 labels)
+    |
+    v
+[4. Span Assembly]   ← BIO → character-offset spans
+    |
+    v
+[5. Section Detection]  ← Rule-based gap-filling for SKILLS, CERTS, LANGUAGES
+    |
+    v
+[6. Post-processing]  ← resume_config.json → post_processing
+    |
+    v
+Structured JSON output
+```
+
+---
+
+## 1. Pre-processing (`text_preprocess.py`)
+
+Config section: `resume_config.json → pre_processing`
+
+Normalizes raw PDF extraction artifacts before the model sees the text. All rules are config-driven.
+
+### Steps (in order):
+
+1. **CRLF normalization** - Convert `\r\n` and `\r` to `\n`
+
+2. **Dash normalization** (`normalize_dashes: true`)
+   - Replace em-dash `—` and en-dash `–` with hyphen `-`
+   - Configured via `dash_replacements` map
+
+3. **Bullet normalization** (`normalize_bullets: true`)
+   - Replace unicode bullets (`●`, `•`, `▪`, `■`, `▸`, `►`, `‣`, `⁃`) with `"- "`
+   - Characters listed in `bullet_chars`, replacement in `bullet_replacement`
+
+4. **Multi-space collapse** (`collapse_multi_spaces: true`)
+   - Reduce runs of 2+ spaces to single space
+
+5. **Label stripping** (`strip_labels: ["Phone:", "Email:"]`)
+   - Remove literal prefixes like "Phone:" or "Email:" that add noise
+
+6. **Skill table expansion** (`expand_skill_tables: true`)
+   - Detects two-column "Category: skill1, skill2" tables common in resumes
+   - Expands them into flat lists for better NER tagging
+   - Recognizes categories from `skill_table_categories` list
+   - Limits: `table_prose_max_words: 15`, `table_continuation_max_chars: 60`
+
+### Usage:
+
+```python
+from training.text_preprocess import preprocess_resume_text
+
+# Uses resume_config.json from current directory
+clean_text = preprocess_resume_text(raw_text)
+
+# Or with explicit config path:
+from training.text_preprocess import ResumeTextPreprocessor
+pp = ResumeTextPreprocessor("/path/to/model_dir")
+clean_text = pp.preprocess(raw_text)
+```
+
+---
+
+## 2. Tokenization & Chunking
+
+Model max sequence length: **512 tokens** (DistilBERT).
+
+For resumes exceeding 512 tokens, section-aware chunking is used (`benchmark_structured.py → chunked_predicted_spans`):
+
+1. Split text at `\n\n` (paragraph) boundaries
+2. Greedily group consecutive sections into chunks that fit within 512 tokens
+3. Run inference on each chunk independently
+4. Map character offsets back to original text
+
+This preserves entity context within natural resume sections (Experience, Education, Skills).
+
+---
+
+## 3. NER Inference
+
+Model: `distilbert-base-cased` fine-tuned for token classification.
+
+**27 BIO labels:**
+
+| Entity | B-tag | I-tag | Description |
+|--------|-------|-------|-------------|
+| NAME | 1 | 2 | Person's full name |
+| EMAIL | 3 | 4 | Email address |
+| PHONE | 5 | 6 | Phone number |
+| LOCATION | 7 | 8 | City, state, country |
+| COMPANY | 9 | 10 | Employer name |
+| TITLE | 11 | 12 | Job title |
+| DATE | 13 | 14 | Employment/education dates |
+| DEGREE | 15 | 16 | Academic degree |
+| INSTITUTION | 17 | 18 | School/university |
+| FIELD | 19 | 20 | Field of study |
+| SKILL | 21 | 22 | Technical/professional skill |
+| CERT | 23 | 24 | Certification |
+| LANGUAGE | 25 | 26 | Spoken language |
+
+Tag `0` = O (outside any entity).
+
+### Subword alignment:
+
+The tokenizer splits words into subword tokens. During training:
+- First subword of a word: gets the word's BIO label
+- Continuation subwords: B-X converts to I-X, other labels propagate
+- Special tokens ([CLS], [SEP], [PAD]): label = -100 (ignored in loss)
+
+---
+
+## 4. Span Assembly
+
+Convert BIO predictions back to character-offset spans:
+
+```python
+@dataclass
+class Span:
+    label: str      # Entity type (NAME, COMPANY, etc.)
+    text: str       # Extracted text
+    start: int      # Character offset start
+    end: int        # Character offset end
+    score: float    # Confidence (1.0 for argmax)
+```
+
+Rules:
+- B-X starts a new span
+- I-X continues the current span (including whitespace gaps between subwords)
+- O or different entity type closes the current span
+
+---
+
+## 5. Section Detection (`section_detector.py`)
+
+Rule-based gap-filling that runs AFTER NER. Catches entities the model missed using section context:
+
+- Detects section headers (SKILLS, CERTIFICATIONS, LANGUAGES, EDUCATION) by keyword matching
+- Within detected sections, extracts untagged text as entities
+- Especially useful for skills lists that the model partially tags
+
+---
+
+## 6. Post-processing (`structured_postprocess.py`)
+
+Config section: `resume_config.json → post_processing`
+
+Transforms raw spans into clean structured JSON.
+
+### 6.1 Span Merging
+
+```json
+"span_merge_max_gap": 3,
+"span_merge_labels": ["TITLE", "COMPANY"]
+```
+
+Adjacent spans of same type (TITLE or COMPANY) separated by <= 3 characters are merged. Handles cases where the model splits "Senior Software Engineer" into multiple spans.
+
+### 6.2 Entity Validation Rules
+
+Each entity type has validation rules in `entity_rules`:
+
+**COMPANY:**
+- `min_length: 4` — reject spans shorter than 4 chars
+- `gazetteer_bypass: true` — known companies from `companies.json` skip length check
+- `strip_trailing_state_code: true` — remove trailing US state codes ("Acme Inc. CA" → "Acme Inc.")
+
+**TITLE:**
+- `min_length: 2`
+- `exceptions: ["VP", "PA", "RN", "MD", "DO", "QA"]` — short titles that are valid
+
+**SKILL:**
+- `min_length: 4`
+- `uppercase_bypass: true` — short all-caps skills (AWS, GCP) pass
+- `exceptions: ["Go", "R", "C", "C#", "F#", "D"]` — valid short skills
+- `blocked_words` — language proficiency descriptors ("native", "fluent", "bilingual") filtered out
+- `aliases` — normalize variants ("nodejs" → "node.js", "cpp" → "c++")
+
+**EMAIL:**
+- `require: "@"` — must contain @
+- `reject_patterns: ["//", "www."]` — filter URLs misclassified as emails
+- `strip_prefixes: ["Esq.", "Dr.", ...]` — remove honorifics attached by OCR
+
+**DATE:**
+- `min_length: 3`
+- `date_words` list validates month names
+- `present_words: ["present", "current"]` — recognized as end-date markers
+
+### 6.3 Text Cleanup
+
+```json
+"space_collapse_pairs": [
+  [" . ", "."],
+  [" + + ", "++"],
+  [" # ", "#"],
+  [" ,", ","]
+]
+```
+
+Fixes tokenizer-induced spacing artifacts in extracted text (e.g., "C + +" → "C++").
+
+### 6.4 Seniority Inference
+
+Determines career level from title keywords and experience duration:
+
+```json
+"seniority_keywords": {
+  "Executive": ["cto", "ceo", ...],
+  "Senior": ["senior", "sr.", "lead", "director", ...],
+  "Junior": ["junior", "intern", "trainee", ...]
+}
+```
+
+Fallback by years of experience:
+```json
+"seniority_by_years": { "Staff": 15, "Senior": 8, "Mid": 3, "Junior": 0 }
+```
+
+### 6.5 Country Detection
+
+1. Phone prefix matching (`phone_country_prefixes`)
+2. Location span matching against `city_country_map.json` (317 cities)
+3. US state code detection (`us_states` list)
+4. Country name aliases ("usa" → "United States")
+
+### 6.6 Experience Years Calculation
+
+- Parse start/end dates from DATE spans
+- `max_experience_months: 600` — cap at 50 years
+- `present_words` treated as current date
+
+---
+
+## Structured Output Format
+
+```json
+{
+  "personal": {
+    "name": "string",
+    "email": "string",
+    "phone": "string",
+    "location": "string"
+  },
+  "experience": [
+    {
+      "title": "string",
+      "company": "string",
+      "start_date": "string",
+      "end_date": "string"
+    }
+  ],
+  "education": [
+    {
+      "degree": "string",
+      "field": "string",
+      "institution": "string"
+    }
+  ],
+  "skills": ["string"],
+  "certifications": ["string"],
+  "seniority": "Executive|Principal|Staff|Senior|Mid|Junior",
+  "country": "string",
+  "experience_years": number
+}
+```
+
+---
+
+## Training Configuration
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | `distilbert-base-cased` |
+| Max sequence length | 512 |
+| Epochs | 25 |
+| Batch size | 8 |
+| Learning rate | 3e-5 |
+| Weight decay | 0.01 |
+| Warmup steps | 20 |
+| Metric for best model | entity_f1 |
+| Noise augmentation | 2x multiplier |
+
+### Training Data Sources
+
+| File | Records | Description |
+|------|---------|-------------|
+| `ner_train.json` | ~3,647 | Synthetic + manual + DataTurks (with noise augmentation) |
+| `kaggle_train.json` | ~7,449 | Kaggle resumes: 2,483 clean + 4,966 noise-augmented |
+
+### Evaluation
+
+| File | Records | Description |
+|------|---------|-------------|
+| `ner_val.json` | 652 | Validation split |
+| `gold/resume_resource_gold.json` | 93 | Hand-annotated gold standard |
+
+---
+
+## Quick Start: Running Inference
+
+```python
+import torch
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+from training.benchmark_structured import chunked_predicted_spans
+from training.structured_postprocess import StructuredPostProcessor
+
+# Load model
+tokenizer = AutoTokenizer.from_pretrained("path/to/model")
+model = AutoModelForTokenClassification.from_pretrained("path/to/model")
+model.eval()
+postprocessor = StructuredPostProcessor("path/to/model")
+
+# Run pipeline
+from training.text_preprocess import ResumeTextPreprocessor
+pp = ResumeTextPreprocessor("path/to/model")
+clean_text = pp.preprocess(raw_resume_text)
+
+_, spans = chunked_predicted_spans(clean_text, model, tokenizer)
+result = postprocessor.build_structured_resume_from_spans(spans, clean_text)
+```
+
+---
+
+## File Reference
+
+| File | Role |
+|------|------|
+| `resume_config.json` | All pre/post processing rules |
+| `label_config.json` | Label ↔ ID mappings |
+| `city_country_map.json` | City → country lookup |
+| `training/data/companies.json` | Company name gazetteer |
+| `training/data/titles.json` | Job title gazetteer |

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47161c540cc9b6b9f5e6be734efc29e650539d1d4eb897f53063d86ab1bfec76
+oid sha256:dbd306d5b166d786a0aa3ce4fff045a07617b6e43181d5a014fa154bc8bcb5bb
 size 260859036

onnx/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20d09f456686cd72701e2189e388fb3e90c1cec14d4c1dd6c93dddf9d7a98cfb
+oid sha256:6b0e3d608be7ce113f9eb25396c87d05da211e1cd1a6a88762f6fba67b83e06c
 size 260958312

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0839099137e08b8799a96501c65f17a3bb0dc6a1e167b7622056eb052418da14
+oid sha256:c2423a3b24dcaca62686156276c6f0a970737639c22531d6c3e02a54c700a427
 size 65622482

training/build_gold_labels.py

@@ -47,7 +47,7 @@ def find_span_in_text(span_text: str, full_text: str) -> bool:
 
 
 def main():
-    texts = json.load(open(Path(__file__).parent / "data" / "resume_resource_texts.json"))
+    texts = json.load(open(Path(__file__).parent / "data" / "sources" / "resume_resource_texts.json"))
     annotations = load_all_annotations()
 
     print(f"Resumes: {len(annotations)}")
@@ -119,7 +119,7 @@ def main():
     for label, count in label_counts.most_common():
         print(f"  {label:15s}: {count}")
 
-    out_path = Path(__file__).parent / "data" / "resume_resource_gold.json"
+    out_path = Path(__file__).parent / "data" / "gold" / "resume_resource_gold.json"
     with open(out_path, "w") as f:
         json.dump({"data": examples}, f)
     print(f"\nWrote {len(examples)} examples to {out_path}")

training/convert_dataturks.py

@@ -1,147 +0,0 @@
-"""Convert DataTurks resume NER dataset to BIO format and merge with existing data."""
-
-from __future__ import annotations
-
-import json
-from collections import Counter
-from pathlib import Path
-
-try:
-    from training.dataset_utils import dedupe_examples, stable_split_examples, write_dataset
-    from training.labels import ID2LABEL, LABEL2ID
-except ModuleNotFoundError:
-    from dataset_utils import dedupe_examples, stable_split_examples, write_dataset
-    from labels import ID2LABEL, LABEL2ID
-
-DATA_DIR = Path(__file__).parent / "data"
-
-LABEL_MAP = {
-    "Name": "NAME",
-    "Email Address": "EMAIL",
-    "Location": "LOCATION",
-    "Companies worked at": "COMPANY",
-    "Designation": "TITLE",
-    "Skills": "SKILL",
-    "Degree": "DEGREE",
-    "College Name": "INSTITUTION",
-    "Graduation Year": "DATE",
-    "Years of Experience": "DATE",
-}
-
-
-def convert_dataturks_sample(item: dict, source_id: str | None = None) -> dict | None:
-    content = item.get("content", "")
-    annotations = item.get("annotation") or []
-    if not content or not annotations:
-        return None
-
-    tokens = content.split()[:512]
-    if len(tokens) < 10:
-        return None
-
-    char_to_word = {}
-    pos = 0
-    for idx, word in enumerate(tokens):
-        start = content.find(word, pos)
-        if start == -1:
-            start = pos
-        for char_idx in range(start, start + len(word)):
-            char_to_word[char_idx] = idx
-        pos = start + len(word)
-
-    labels = ["O"] * len(tokens)
-    for annotation in annotations:
-        ann_labels = annotation.get("label", [])
-        points = annotation.get("points", [])
-        if not ann_labels or not points:
-            continue
-
-        our_label = LABEL_MAP.get(ann_labels[0])
-        if not our_label:
-            continue
-
-        for point in points:
-            tagged_words = sorted({
-                char_to_word[char_idx]
-                for char_idx in range(point.get("start", 0), point.get("end", 0) + 1)
-                if char_idx in char_to_word
-            })
-            for offset, word_idx in enumerate(tagged_words):
-                if word_idx < len(labels) and labels[word_idx] == "O":
-                    labels[word_idx] = f"B-{our_label}" if offset == 0 else f"I-{our_label}"
-
-    tagged = sum(1 for label in labels if label != "O")
-    if tagged < 3:
-        return None
-
-    return {
-        "tokens": tokens,
-        "ner_tags": [LABEL2ID[label] for label in labels],
-        "metadata": {
-            "source": "dataturks",
-            "source_id": source_id or "dataturks:unknown",
-            "group_id": source_id or "dataturks:unknown",
-        },
-    }
-
-
-def main() -> None:
-    data = []
-    decode_errors = 0
-    with open(DATA_DIR / "dataturks_raw.json") as f:
-        for line_no, line in enumerate(f, start=1):
-            line = line.strip()
-            if not line:
-                continue
-            try:
-                data.append(json.loads(line))
-            except json.JSONDecodeError:
-                decode_errors += 1
-                print(f"Skipped invalid JSON line {line_no}")
-
-    converted = []
-    converted_skipped = 0
-    for idx, item in enumerate(data):
-        result = convert_dataturks_sample(item, source_id=f"dataturks:{idx}")
-        if result:
-            converted.append(result)
-        else:
-            converted_skipped += 1
-
-    with open(DATA_DIR / "ner_train.json") as f:
-        train_payload = json.load(f)
-    with open(DATA_DIR / "ner_val.json") as f:
-        val_payload = json.load(f)
-
-    combined, duplicates_removed = dedupe_examples(train_payload["data"] + val_payload["data"] + converted)
-    train, val = stable_split_examples(combined, train_ratio=0.8)
-    manifest = {
-        "builder": "convert_dataturks.py",
-        "sources": {
-            "dataturks_raw_lines": len(data),
-            "dataturks_decode_errors": decode_errors,
-            "dataturks_converted": len(converted),
-            "dataturks_skipped": converted_skipped,
-            "duplicates_removed": duplicates_removed,
-        },
-    }
-    write_dataset(train, val, DATA_DIR, manifest=manifest)
-
-    print(f"DataTurks raw: {len(data)} samples")
-    print(f"DataTurks converted: {len(converted)}")
-    print(f"Combined unique: {len(combined)}")
-    print(f"Train: {len(train)}, Val: {len(val)}")
-
-    label_counts = Counter()
-    for example in combined:
-        for tag_id in example["ner_tags"]:
-            label = ID2LABEL[tag_id]
-            if label != "O":
-                label_counts[label[2:]] += 1
-    print("Label distribution:")
-    for label, count in label_counts.most_common():
-        print(f"  {label:15s}: {count}")
-
-
-if __name__ == "__main__":
-    main()

training/data/kaggle_train.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:905ebc8ef723a3ad1d07e28b26c1bd7e8416896c3570d9dbb30d2b49572cc4d7
+size 54797265

training/data/sources/kaggle_resumes.csv

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15188f30175c6cb3268754c4ec7bd861abd38b94e742ba009a7977163c4e94ef
+size 15112023

training/generate_from_structured.py

@@ -239,7 +239,7 @@ def main():
         from build_long_resumes import build_examples as build_long_examples
 
     dataturks = []
-    with open(DATA_DIR / "dataturks_raw.json") as f:
+    with open(DATA_DIR / "sources" / "dataturks_raw.json") as f:
         for line in f:
             line = line.strip()
             if line:
@@ -255,7 +255,7 @@ def main():
     long_resumes = build_long_examples()
 
     resume_resource = []
-    rr_path = DATA_DIR / "resume_resource_gold.json"
+    rr_path = DATA_DIR / "gold" / "resume_resource_gold.json"
     if rr_path.exists():
         with open(rr_path) as f:
             resume_resource = json.load(f)["data"]

training/label_resume_pdfs.py

@@ -1,124 +0,0 @@
-"""Extract text from resume PDFs and assign BIO gold labels.
-
-Uses kreuzberg for PDF extraction and the current NER model for initial
-label predictions, then applies heuristic corrections.
-
-Run: python -m training.label_resume_pdfs
-"""
-
-import asyncio
-import json
-from pathlib import Path
-
-import torch
-from kreuzberg import extract_file
-from transformers import AutoModelForTokenClassification, AutoTokenizer
-
-from training.benchmark_structured import chunked_predicted_spans
-from training.labels import LABEL2ID, ID2LABEL
-
-PDF_DIR = Path("training/data/resume_resource")
-OUT_PATH = Path("training/data/resume_resource_labeled.json")
-
-
-async def extract_all(pdf_dir: Path) -> list[tuple[str, str]]:
-    pdfs = sorted(pdf_dir.glob("*.pdf"))
-
-    async def extract_one(p):
-        try:
-            result = await extract_file(p)
-            return p.stem, result.content
-        except Exception:
-            return p.stem, None
-
-    results = await asyncio.gather(*[extract_one(p) for p in pdfs])
-    return [(name, text) for name, text in results if text and len(text.split()) >= 20]
-
-
-def spans_to_bio(tokens: list[str], spans: list, text: str) -> list[int]:
-    labels = [0] * len(tokens)
-    char_to_token = {}
-    pos = 0
-    for i, tok in enumerate(tokens):
-        idx = text.find(tok, pos)
-        if idx == -1:
-            idx = pos
-        for c in range(idx, idx + len(tok)):
-            char_to_token[c] = i
-        pos = idx + len(tok)
-
-    for span in spans:
-        token_indices = set()
-        for c in range(span.start, span.end):
-            if c in char_to_token:
-                token_indices.add(char_to_token[c])
-        token_indices = sorted(token_indices)
-        if not token_indices:
-            continue
-        b_label = LABEL2ID.get(f"B-{span.label}", 0)
-        i_label = LABEL2ID.get(f"I-{span.label}", 0)
-        for j, ti in enumerate(token_indices):
-            if labels[ti] == 0:
-                labels[ti] = b_label if j == 0 else i_label
-
-    return labels
-
-
-def main():
-    print("Loading model...")
-    tokenizer = AutoTokenizer.from_pretrained(".")
-    model = AutoModelForTokenClassification.from_pretrained(".")
-    model.eval()
-
-    print("Extracting PDFs...")
-    extracted = asyncio.run(extract_all(PDF_DIR))
-    print(f"Extracted {len(extracted)} resumes")
-
-    examples = []
-    skipped = 0
-    for name, text in extracted:
-        text = text.strip()
-        _, pred_spans = chunked_predicted_spans(text, model, tokenizer)
-        tokens = text.split()
-        if len(tokens) < 20:
-            skipped += 1
-            continue
-        ner_tags = spans_to_bio(tokens, pred_spans, text)
-        tagged = sum(1 for t in ner_tags if t != 0)
-        if tagged < 5:
-            skipped += 1
-            continue
-        examples.append({
-            "tokens": tokens,
-            "ner_tags": ner_tags,
-            "metadata": {
-                "source": "resume_resource",
-                "source_id": f"resume_resource:{name}",
-                "group_id": f"resume_resource:{name}",
-            },
-        })
-
-    print(f"Labeled: {len(examples)}, Skipped: {skipped}")
-
-    token_counts = [len(e["tokens"]) for e in examples]
-    print(f"Token counts: min={min(token_counts)}, max={max(token_counts)}, avg={sum(token_counts)//len(token_counts)}")
-
-    from collections import Counter
-    label_counts = Counter()
-    for e in examples:
-        for tag in e["ner_tags"]:
-            label = ID2LABEL[tag]
-            if label != "O":
-                label_counts[label[2:]] += 1
-    print("\nLabel distribution:")
-    for label, count in label_counts.most_common():
-        print(f"  {label:15s}: {count}")
-
-    output = {"data": examples}
-    with open(OUT_PATH, "w") as f:
-        json.dump(output, f)
-    print(f"\nWrote {len(examples)} examples to {OUT_PATH}")
-
-
-if __name__ == "__main__":
-    main()

training/structured_postprocess.py

@@ -20,12 +20,12 @@ class Span:
 class StructuredPostProcessor:
     def __init__(self, model_dir: str | Path):
         self.model_dir = Path(model_dir)
-        with open(self.model_dir / "resume_config.json") as fh:
+        with open(self.model_dir / "resume_config.json", encoding="utf-8") as fh:
             self.config = json.load(fh)
         companies_path = self.model_dir / "companies.json"
         self.companies = set()
         if companies_path.exists():
-            with open(companies_path) as fh:
+            with open(companies_path, encoding="utf-8") as fh:
                 data = json.load(fh)
             self.companies = {company.lower() for companies in data.values() for company in companies}
         self.multi_word_skills = {skill.lower() for skill in self.config.get("multi_word_skills", [])}

training/tagging.py

@@ -1,5 +1,8 @@
 from __future__ import annotations
 
+from training.labels import LABEL2ID
+from training.structured_postprocess import Span
+
 
 def normalize_token(text: str) -> str:
     return text.strip().rstrip(",.;:|")
@@ -53,3 +56,28 @@ def tag_exact_words(tokens: list[str], labels: list[str], text: str, label_type:
                     labels[idx] = f"B-{label_type}" if j == 0 else f"I-{label_type}"
             return i + len(words)
     return start_from
+
+
+def spans_to_bio(tokens: list[str], spans: list[Span], text: str) -> list[int]:
+    labels = [0] * len(tokens)
+    char_to_token = {}
+    pos = 0
+    for i, tok in enumerate(tokens):
+        idx = text.find(tok, pos)
+        if idx == -1:
+            idx = pos
+        for char_idx in range(idx, idx + len(tok)):
+            char_to_token[char_idx] = i
+        pos = idx + len(tok)
+
+    for span in spans:
+        token_indices = sorted({char_to_token[char_idx] for char_idx in range(span.start, span.end) if char_idx in char_to_token})
+        if not token_indices:
+            continue
+        b_label = LABEL2ID.get(f"B-{span.label}", 0)
+        i_label = LABEL2ID.get(f"I-{span.label}", 0)
+        for offset, token_idx in enumerate(token_indices):
+            if labels[token_idx] == 0:
+                labels[token_idx] = b_label if offset == 0 else i_label
+
+    return labels

training/train_ner.py

@@ -40,13 +40,25 @@ BASE_MODELS = {
 }
 
 
-def load_ner_data(split: str) -> tuple[list, dict]:
+def load_ner_data(split: str, extra_train_files: list[str] | None = None) -> tuple[list, dict]:
     path = DATA_DIR / f"ner_{split}.json"
     if not path.exists():
         raise FileNotFoundError(f"Missing {path}. Generate training data first.")
-    with open(path) as f:
+    with open(path, encoding="utf-8") as f:
         payload = json.load(f)
-    return payload["data"], payload["meta"]
+    data = payload["data"]
+    meta = payload["meta"]
+
+    if split == "train" and extra_train_files:
+        for extra in extra_train_files:
+            extra_path = DATA_DIR / extra
+            if extra_path.exists():
+                with open(extra_path, encoding="utf-8") as f:
+                    extra_payload = json.load(f)
+                data.extend(extra_payload["data"])
+                print(f"  + {extra}: {len(extra_payload['data'])} records")
+
+    return data, meta
 
 
 def tokenize_and_align(examples, tokenizer, label_list, max_length=512):
@@ -132,6 +144,8 @@ def main() -> None:
     parser.add_argument("--lr", type=float, default=3e-5)
     parser.add_argument("--max-length", type=int, default=512)
     parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--extra-train", nargs="*", default=["kaggle_train.json"],
+                        help="Extra training JSON files in data/ dir")
     args = parser.parse_args()
 
     set_global_seed(args.seed)
@@ -139,7 +153,7 @@ def main() -> None:
     model_name = BASE_MODELS[args.base_model]
     output_dir = OUTPUT_DIR / args.base_model
 
-    train_data, meta = load_ner_data("train")
+    train_data, meta = load_ner_data("train", extra_train_files=args.extra_train)
     val_data, _ = load_ner_data("val")
     label_list = meta["label_list"]
     label2id = {label: idx for idx, label in enumerate(label_list)}