Instructions to use ReliableAI/UCCIX-Llama2-13B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ReliableAI/UCCIX-Llama2-13B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ReliableAI/UCCIX-Llama2-13B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ReliableAI/UCCIX-Llama2-13B")
model = AutoModelForCausalLM.from_pretrained("ReliableAI/UCCIX-Llama2-13B")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ReliableAI/UCCIX-Llama2-13B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ReliableAI/UCCIX-Llama2-13B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReliableAI/UCCIX-Llama2-13B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ReliableAI/UCCIX-Llama2-13B

SGLang

How to use ReliableAI/UCCIX-Llama2-13B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ReliableAI/UCCIX-Llama2-13B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReliableAI/UCCIX-Llama2-13B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ReliableAI/UCCIX-Llama2-13B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ReliableAI/UCCIX-Llama2-13B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ReliableAI/UCCIX-Llama2-13B with Docker Model Runner:
```
docker model run hf.co/ReliableAI/UCCIX-Llama2-13B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model Card for UCCIX-Llama2-13B

The UCCIX-Llama2-13B Large Language Model (LLM) is an Irish-English bilingual model, capables of understanding both languages and outperforms much larger models on Irish language tasks. The model is based on Llama 2-13B, with vocabulary expansion to include native Irish tokens, and additional continued pre-training on our collection of ~520M Irish tokens (available at https://huggingface.co/datasets/ReliableAI/Irish-Text-Collection).

UCCIX is a pioneering effort on the development of first-ever open-source Irish-based LLM. You can find more details at: https://arxiv.org/abs/2405.13010

Access the instruction-tuned version at https://huggingface.co/ReliableAI/UCCIX-Llama2-13B-Instruct, and interact with it live at: https://aine.chat

Run the model

Run the model with the transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "ReliableAI/UCCIX-Llama2-13B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             device_map="auto",
                                             dtype=torch.float16 # optional, load in 16-bit precision mode to reduce memory usage
)
model.eval()

input = "I love the environment."
input_ids = tokenizer(input, return_tensors="pt")["input_ids"]
generated_token_ids = model.generate(
    inputs=input_ids,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.6,
    top_p=1,
)[0]
generated_text = tokenizer.decode(generated_token_ids)

Notice

As a pioneering effort, the UCCIX model does not have any moderation mechanisms at the moment. We anticipate collaborating with the community to refine the model's adherence to restrictions so that it can be implemented in settings that demand moderated outcomes.

Citation

@misc{tran2024uccix,
      title={UCCIX: Irish-eXcellence Large Language Model}, 
      author={Khanh-Tung Tran and Barry O'Sullivan and Hoang D. Nguyen},
      year={2024},
      eprint={2405.13010},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}