GOAT-V 482M · C4 1× Chinchilla · seed 0 (Flax / Orbax — RESUME)
Original Flax/Orbax checkpoint from a flaxchat training run on TPU v6e-8. Includes model state + optimizer state + metadata, so training can be resumed exactly from this point (294,362 steps = full 1× Chinchilla).
For inference / fine-tuning in PyTorch, use the converted port: .
Architecture
GOAT-V (yat-full): yat attention with V projection kept (no Q/K), no value embeddings, YatNMN-Softplus MLP. Identical architecture to ; this repo differs only in pretraining corpus (C4 vs FineWeb-Edu).
| Params | 482.3M |
| Depth / n_embd / heads | d=22 / 1408 / 22 |
| Tokens | 9.65B (1× Chinchilla) |
| Data | allenai/c4 (en) |
| Final loss | 2.9694 (smoothed 3.0007) |
| Seed | 0 |
| Hardware | TPU v6e-8 (europe-west4-a, TRC), fp32, ~20.6h |
| Training script | scripts/launch_yat_full_d22_c4_1x.py |
Layout
Resume from this checkpoint
See flaxchat training scripts for the exact restore + continuation pattern.
License
Apache 2.0.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support