| ---
|
| language:
|
| - ru
|
|
|
| pipeline_tag: sentence-similarity
|
|
|
| tags:
|
| - russian
|
| - pretraining
|
| - embeddings
|
| - tiny
|
| - feature-extraction
|
| - sentence-similarity
|
| - sentence-transformers
|
| - transformers
|
|
|
| license: mit
|
| base_model: cointegrated/rubert-tiny2
|
|
|
| ---
|
|
|
| ## Быстрый Bert для Semantic text similarity (STS) на CPU
|
|
|
| Быстрая модель BERT для расчетов компактных эмбеддингов предложений на русском языке. Модель основана на [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) - имеет аналогичные размеры контекста (2048), ембеддинга (312) и быстродействие.
|
|
|
|
|
| ## Использование модели с библиотекой `transformers`:
|
|
|
| ```python
|
| # pip install transformers sentencepiece
|
| import torch
|
| from transformers import AutoTokenizer, AutoModel
|
| tokenizer = AutoTokenizer.from_pretrained("sergeyzh/rubert-tiny-sts")
|
| model = AutoModel.from_pretrained("sergeyzh/rubert-tiny-sts")
|
| # model.cuda() # uncomment it if you have a GPU
|
|
|
| def embed_bert_cls(text, model, tokenizer):
|
| t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
|
| with torch.no_grad():
|
| model_output = model(**{k: v.to(model.device) for k, v in t.items()})
|
| embeddings = model_output.last_hidden_state[:, 0, :]
|
| embeddings = torch.nn.functional.normalize(embeddings)
|
| return embeddings[0].cpu().numpy()
|
|
|
| print(embed_bert_cls('привет мир', model, tokenizer).shape)
|
| # (312,)
|
| ```
|
|
|
| ## Использование с `sentence_transformers`:
|
| ```Python
|
| from sentence_transformers import SentenceTransformer, util
|
|
|
| model = SentenceTransformer('sergeyzh/rubert-tiny-sts')
|
|
|
| sentences = ["привет мир", "hello world", "здравствуй вселенная"]
|
| embeddings = model.encode(sentences)
|
| print(util.dot_score(embeddings, embeddings))
|
| ```
|
|
|
| ## Метрики
|
| Оценки модели на бенчмарке [encodechka](https://github.com/avidale/encodechka):
|
|
|
| | Модель | STS | PI | NLI | SA | TI |
|
| |:---------------------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
|
| | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 0.862 | 0.727 | 0.473 | 0.810 | 0.979 |
|
| | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 0.845 | 0.737 | 0.481 | 0.805 | 0.957 |
|
| | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 0.815 | 0.723 | 0.477 | 0.791 | 0.949 |
|
| | **sergeyzh/rubert-tiny-sts** | 0.797 | 0.702 | 0.453 | 0.778 | 0.946 |
|
| | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 0.793 | 0.704 | 0.457 | 0.803 | 0.970 |
|
| | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 0.794 | 0.659 | 0.431 | 0.761 | 0.946 |
|
| | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 0.750 | 0.651 | 0.417 | 0.737 | 0.937 |
|
|
|
| **Задачи:**
|
|
|
| - Semantic text similarity (**STS**);
|
| - Paraphrase identification (**PI**);
|
| - Natural language inference (**NLI**);
|
| - Sentiment analysis (**SA**);
|
| - Toxicity identification (**TI**).
|
|
|
| ## Быстродействие и размеры
|
|
|
| На бенчмарке [encodechka](https://github.com/avidale/encodechka):
|
|
|
| | Модель | CPU | GPU | size | dim | n_ctx | n_vocab |
|
| |:---------------------------------|----------:|----------:|----------:|----------:|----------:|----------:|
|
| | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 149.026 | 15.629 | 2136 | 1024 | 514 | 250002 |
|
| | [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts) | 42.835 | 8.561 | 490 | 768 | 512 | 55083 |
|
| | [sergeyzh/rubert-mini-sts](https://huggingface.co/sergeyzh/rubert-mini-sts) | 6.417 | 5.517 | 123 | 312 | 2048 | 83828 |
|
| | **sergeyzh/rubert-tiny-sts** | 3.208 | 3.379 | 111 | 312 | 2048 | 83828 |
|
| | [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) | 43.314 | 9.338 | 532 | 768 | 512 | 69382 |
|
| | [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru) | 42.867 | 8.549 | 490 | 768 | 512 | 55083 |
|
| | [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) | 3.212 | 3.384 | 111 | 312 | 2048 | 83828 |
|
|
|
|
|
|
|
| |
