From 164ab81e27470c6cb948bea3e4bf2e64bffa33b9 Mon Sep 17 00:00:00 2001 From: Ferdinand Mom <47445085+3outeille@users.noreply.github.com> Date: Thu, 19 Dec 2024 09:24:14 +0100 Subject: [PATCH] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index becb743..a037779 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ In the spirit of [NanoGPT](https://github.com/karpathy/nanoGPT), we created Picotron: The minimalist & most-hackable repository for pre-training Llama-like models with [4D Parallelism](https://arxiv.org/abs/2407.21783) (Data, Tensor, Pipeline, Context parallel). It is designed with simplicity and **educational** purposes in mind, making it an excellent tool for learning and experimentation. ![](assets/banière.png) -- The code itself is simple and readable: train.py, model.py and \[data|tensor|pipeline|context\]_parallel.py are all under **300** lines of code. +- The code itself is simple and readable: `train.py`, `model.py` and `[data|tensor|pipeline|context]_parallel.py` are all under **300** lines of code. -- Performance is not the best but okay-ish, and still under active development. We observed 38% MFU on a LLaMA-2-7B model using 64 H100 GPUs and nearly 50% MFU on the SmolLM-1.7B model with 8 H100 GPUs. +- Performance is not the best but okay-ish, and still under active development. We observed 38% MFU on a LLaMA-2-7B model using 64 H100 GPUs and nearly 50% MFU on the SmolLM-1.7B model with 8 H100 GPUs. Benchmarks will come soon # Install @@ -43,4 +43,4 @@ pip install -e . - [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) - [FairScale](https://github.com/facebookresearch/fairscale) -- [LitGPT](https://github.com/Lightning-AI/lit-gpt) \ No newline at end of file +- [LitGPT](https://github.com/Lightning-AI/lit-gpt)