v2: update model card — add BA class, new metrics, usage example

README.md

@@ -14,9 +14,10 @@ tags:
 - jobs
 - developer-classification
 - data-analyst-classification
-- dev-plus-da
+- business-analyst-classification
+- dev-plus-da-plus-ba
 - r95
-- v1
+- v2
 base_model: cointegrated/rubert-tiny2
 metrics:
 - precision
@@ -27,22 +28,24 @@ model-index:
   results:
   - task:
       type: text-classification
-      name: Developer/Data Analyst vs Other Binary Classification
+      name: Developer / Data Analyst / Business Analyst vs Other Binary Classification
     metrics:
     - type: roc_auc
-      value: 0.9790
+      value: 0.9815
     - type: precision
-      value: 0.8797
+      value: 0.9219
     - type: recall
-      value: 0.9509
+      value: 0.9506
 ---
 
-# dev_da_roles_1 - Developer + Data Analyst Classifier
+# dev_da_roles_1 — Developer + Data Analyst + Business Analyst Classifier
 
-Binary job-vacancy classifier: detects **developer or Data Analyst** roles (`tech`) versus **other** roles (`other`).
+Binary job-vacancy classifier: detects **developer, Data Analyst, or Business Analyst** roles (`tech`) versus **other** roles (`other`).
 
 Built on top of [`cointegrated/rubert-tiny2`](https://huggingface.co/cointegrated/rubert-tiny2), a compact BERT model for Russian and English text.
 
+> **v2** — extends v1 by adding Business Analyst to the positive class and using a longer input context (384 tokens / 2000 chars). Precision improved from 0.880 → 0.922.
+
 ## Task Definition
 
 The positive class (`tech`) is defined as:
@@ -59,6 +62,7 @@ The positive class (`tech`) is defined as:
 - ML / AI / Data Scientist
 - Mobile
 - Data Analyst
+- Бизнес аналитик (Business Analyst)
 
 Team leads and management roles are intentionally excluded from the positive class.
 
@@ -73,19 +77,29 @@ Team leads and management roles are intentionally excluded from the positive cla
 
 | Metric | Value |
 |---|---:|
-| ROC AUC | 0.9790 |
-| Precision @ threshold | 0.8797 |
-| Recall @ threshold | 0.9509 |
-| Best threshold | 0.3765 |
+| ROC AUC | 0.9815 |
+| Precision @ threshold | 0.9219 |
+| Recall @ threshold | 0.9506 |
+| Best threshold | 0.8791 |
 | Target recall | 0.95 |
+| Best epoch | 7 |
+
+**Recall by key category (held-out test set):**
 
-Best epoch: **8**. Training `pos_weight`: **2.4813**.
+| Category | Recall |
+|---|---:|
+| Backend | 0.984 |
+| Frontend | 1.000 |
+| Mobile | 1.000 |
+| ML / AI / Data Scientist | 0.976 |
+| Data Analyst | 0.916 |
+| Business Analyst | 0.895 |
 
 ## Inference Parameters
 
-- `max_length`: **256** tokens
-- Vacancy text is formed as `title + description`, with description truncated to **1200 characters**
-- Decision threshold for class `tech`: **0.3765**
+- `max_length`: **384** tokens
+- Vacancy text: `title + " . " + description`, description truncated to **2000 characters**
+- Decision threshold for class `tech`: **0.8791**
 
 ## Usage
 
@@ -94,23 +108,30 @@ import torch
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
 MODEL_ID = "AndreiTolmachev/dev_da_roles_1"
-THRESHOLD = 0.3765
+THRESHOLD = 0.8791
 
 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
 model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).eval()
 
-def is_developer_or_data_analyst(title: str, description: str = "") -> bool:
-    text = (title + " " + description[:1200]).strip()
-    enc = tokenizer(text, truncation=True, max_length=256, return_tensors="pt")
+def is_tech_role(title: str, description: str = "") -> bool:
+    text = f"{title.strip()} . {description[:2000].strip()}"
+    enc = tokenizer(text, truncation=True, max_length=384, return_tensors="pt")
     with torch.no_grad():
         logits = model(**enc).logits
     prob_tech = torch.softmax(logits, dim=-1)[0, 1].item()
     return prob_tech >= THRESHOLD
 
-print(is_developer_or_data_analyst(
-    "Data Analyst",
-    "SQL, Python, dashboards, product metrics, A/B tests..."
-))
+# Developer
+print(is_tech_role("Backend Python Developer", "FastAPI, PostgreSQL, Docker, Kubernetes..."))
+
+# Data Analyst
+print(is_tech_role("Data Analyst", "SQL, Python, dashboards, product metrics, A/B tests..."))
+
+# Business Analyst
+print(is_tech_role("Бизнес аналитик", "Сбор требований, UML, BPMN, работа с командой разработки..."))
+
+# Manager — should return False
+print(is_tech_role("Project Manager", "Agile, управление командой, планирование спринтов..."))
 ```
 
 ## Architecture
@@ -125,20 +146,29 @@ print(is_developer_or_data_analyst(
 ## Training
 
 - Dataset: internal job-vacancy dataset (`vacancies_labeled.csv`), labeled by an LLM pipeline
-- Loss: weighted cross-entropy
-- Threshold selected for target recall = **0.95**
-- Positive class includes developer roles and Data Analyst, excluding team leads
+- Train/test split: 85% / 15%, stratified by role and team_lead flag
+- Loss: weighted cross-entropy (`pos_weight` = 2.115)
+- Optimizer: AdamW, `lr=2e-5`, linear warmup 10%, grad clip 1.0
+- Early stopping: patience=3 on F1 at target recall ≥ 0.95
+- Threshold selected to achieve target recall = **0.95**
 
 ## Limitations
 
 - Trained primarily on Russian-language IT job vacancies; quality on other domains/languages is not guaranteed.
 - Team lead and management roles are treated as `other` by design.
-- Description is truncated to 1200 characters before tokenization.
-- The model intentionally groups developers and Data Analysts into one positive class; it does not distinguish between them.
+- Description is truncated to 2000 characters before tokenization.
+- The model groups developers, Data Analysts, and Business Analysts into one positive class; it does not distinguish between them.
+- Data Analyst recall is ~0.92: vacancies with heavy business/finance framing may be missed.
 
 ## Version
 
-Hub tag: `v1.0-dev-da-r95`
+Hub tag: `v2.0-dev-da-ba-r95`
+
+**Changelog vs v1:**
+- Added Business Analyst (`Бизнес аналитик`) to positive class
+- Input context extended: `max_length` 256→384, description 1200→2000 chars
+- Precision improved: 0.880 → 0.922
+- `lr` lowered to 2e-5, batch size 32→24 to accommodate longer sequences
 
 ## License