MMA LLM
Built and trained a 35M-parameter Llama-style autoregressive language model from scratch in PyTorch, implementing multi-head attention, RoPE embeddings, RMSNorm, and SwiGLU activations. Trained a custom 16k-vocabulary BPE tokenizer, completed general pretraining on 8M tokens, and continued domain pretraining on verbalized UFC fight data. The model generalized to held-out prompts and generated syntactically correct, contextually coherent fight summaries.
PyTorchLlamaBPERoPERMSNormSwiGLU