GitHub – Deepseek-ai/DeepSeek-V3

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language design with 671B total criteria with 37B activated for each token. To attain effective reasoning and economical training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were completely validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training goal for more powerful efficiency. We pre-train DeepSeek-V3 on 14.8 trillion varied and premium tokens, followed by Supervised Fine-Tuning and Reinforcement Learning phases to fully harness its abilities. Comprehensive assessments expose that DeepSeek-V3 outshines other open-source models and accomplishes efficiency equivalent to leading closed-source models. Despite its outstanding efficiency, DeepSeek-V3 needs just 2.788 M H800 GPU hours for its full training. In addition, its training procedure is extremely steady. Throughout the entire training procedure, we did not experience any irrecoverable loss spikes or perform any rollbacks.

2. Model Summary

Architecture: Innovative Load Balancing Strategy and Training Objective

– On top of the effective architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free technique for load balancing, which minimizes the performance destruction that emerges from encouraging load balancing.
– We examine a Multi-Token Prediction (MTP) objective and show it beneficial to design performance. It can likewise be used for speculative decoding for inference velocity.

Pre-Training: Towards Ultimate Training Efficiency

– We develop an FP8 mixed accuracy training structure and, for the first time, confirm the feasibility and effectiveness of FP8 training on an exceptionally large-scale design.
– Through co-design of algorithms, structures, and hardware, we conquer the communication bottleneck in cross-node MoE training, almost attaining complete computation-communication overlap.
This considerably enhances our training performance and minimizes the training expenses, allowing us to even more scale up the model size without additional overhead.
– At an affordable cost of only 2.664 M H800 GPU hours, we finish the pre-training of DeepSeek-V3 on 14.8 T tokens, producing the currently greatest open-source base model. The subsequent training stages after pre-training require only 0.1 M GPU hours.

Post-Training: Knowledge Distillation from DeepSeek-R1

– We present an innovative methodology to distill thinking abilities from the long-Chain-of-Thought (CoT) model, particularly from among the DeepSeek R1 series models, into basic LLMs, particularly DeepSeek-V3. Our pipeline elegantly includes the confirmation and reflection patterns of R1 into DeepSeek-V3 and notably improves its reasoning efficiency. Meanwhile, we also maintain a control over the output design and length of DeepSeek-V3.

3. Model Downloads

The total size of DeepSeek-V3 designs on Hugging Face is 685B, which includes 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights. **

To ensure optimal performance and versatility, we have partnered with open-source neighborhoods and hardware suppliers to provide multiple ways to run the model locally. For step-by-step assistance, check out Section 6: How_to Run_Locally.

For developers aiming to dive deeper, we advise exploring README_WEIGHTS. md for information on the Main Model weights and the Multi-Token Prediction (MTP) Modules. Please note that MTP assistance is currently under active advancement within the community, and we welcome your contributions and feedback.

4. Evaluation Results

Base Model

Standard Benchmarks

Best outcomes are shown in vibrant. Scores with a space not exceeding 0.3 are considered to be at the exact same level. DeepSeek-V3 attains the very best performance on most standards, specifically on mathematics and code jobs. For more evaluation information, please examine our paper.

Context Window

Evaluation results on the Needle In A Haystack (NIAH) tests. DeepSeek-V3 carries out well throughout all context window lengths up to 128K.

Chat Model

Standard Benchmarks (Models bigger than 67B)

All designs are evaluated in a configuration that limits the output length to 8K. Benchmarks including less than 1000 samples are evaluated several times utilizing differing temperature level settings to derive robust results. DeepSeek-V3 stands as the best-performing open-source model, and likewise displays competitive efficiency against frontier closed-source designs.

Open Ended Generation Evaluation

English open-ended conversation evaluations. For AlpacaEval 2.0, we utilize the length-controlled win rate as the metric.

5. Chat Website & API Platform

You can talk with DeepSeek-V3 on DeepSeek’s official website: chat.deepseek.com

We likewise supply OpenAI-Compatible API at DeepSeek Platform: platform.deepseek.com

6. How to Run Locally

DeepSeek-V3 can be released locally utilizing the following hardware and open-source neighborhood software:

DeepSeek-Infer Demo: We provide an easy and lightweight demonstration for FP8 and BF16 reasoning.
SGLang: Fully support the DeepSeek-V3 model in both BF16 and FP8 inference modes, with Multi-Token Prediction coming quickly.
LMDeploy: Enables effective FP8 and BF16 reasoning for local and cloud release.
TensorRT-LLM: Currently supports BF16 reasoning and INT4/8 quantization, with FP8 assistance coming soon.
vLLM: Support DeepSeek-V3 model with FP8 and BF16 modes for tensor parallelism and pipeline parallelism.
AMD GPU: Enables running the DeepSeek-V3 design on AMD GPUs via SGLang in both BF16 and FP8 modes.
Huawei Ascend NPU: Supports running DeepSeek-V3 on Huawei Ascend gadgets.
Since FP8 training is natively embraced in our framework, we just offer FP8 . If you need BF16 weights for experimentation, you can utilize the supplied conversion script to carry out the improvement.

Here is an example of converting FP8 weights to BF16:

Hugging Face’s Transformers has actually not been directly supported yet. **

6.1 Inference with DeepSeek-Infer Demo (example just)

System Requirements

Note

Linux with Python 3.10 only. Mac and Windows are not supported.

Dependencies:

Model Weights & Demo Code Preparation

First, clone our DeepSeek-V3 GitHub repository:

Navigate to the inference folder and install dependencies listed in requirements.txt. Easiest way is to use a package manager like conda or uv to create a new virtual environment and install the dependences.

Download the model weights from Hugging Face, and put them into/ path/to/DeepSeek-V 3 folder.

Model Weights Conversion

Convert Hugging Face design weights to a particular format:

Run

Then you can chat with DeepSeek-V3:

Or batch inference on an offered file:

6.2 Inference with SGLang (advised)

SGLang presently supports MLA optimizations, DP Attention, FP8 (W8A8), FP8 KV Cache, and Torch Compile, providing modern latency and throughput performance amongst open-source frameworks.

Notably, SGLang v0.4.1 completely supports running DeepSeek-V3 on both NVIDIA and AMD GPUs, making it an extremely versatile and robust option.

SGLang also supports multi-node tensor parallelism, enabling you to run this model on numerous network-connected makers.

Multi-Token Prediction (MTP) remains in development, and development can be tracked in the optimization strategy.

Here are the launch directions from the SGLang group: https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3

6.3 Inference with LMDeploy (suggested)

LMDeploy, a flexible and high-performance inference and serving structure customized for big language models, now supports DeepSeek-V3. It offers both offline pipeline processing and online implementation capabilities, perfectly incorporating with PyTorch-based workflows.

For comprehensive step-by-step instructions on running DeepSeek-V3 with LMDeploy, please describe here: InternLM/lmdeploy # 2960

6.4 Inference with TRT-LLM (recommended)

TensorRT-LLM now supports the DeepSeek-V3 design, using accuracy choices such as BF16 and INT4/INT8 weight-only. Support for FP8 is presently in development and will be released quickly. You can access the custom branch of TRTLLM specifically for DeepSeek-V3 support through the following link to experience the new features directly: https://github.com/NVIDIA/TensorRT-LLM/tree/deepseek/examples/deepseek_v3.

6.5 Inference with vLLM (suggested)

vLLM v0.6.6 supports DeepSeek-V3 inference for FP8 and BF16 modes on both NVIDIA and AMD GPUs. Aside from basic strategies, vLLM provides pipeline parallelism permitting you to run this model on several makers linked by networks. For in-depth assistance, please refer to the vLLM directions. Please do not hesitate to follow the enhancement plan too.

6.6 Recommended Inference Functionality with AMD GPUs

In cooperation with the AMD group, we have actually attained Day-One support for AMD GPUs using SGLang, with full compatibility for both FP8 and BF16 precision. For detailed assistance, please describe the SGLang directions.

6.7 Recommended Inference Functionality with Huawei Ascend NPUs

The MindIE framework from the Huawei Ascend neighborhood has effectively adapted the BF16 variation of DeepSeek-V3. For detailed assistance on Ascend NPUs, please follow the instructions here.

7. License

This code repository is accredited under the MIT License. Using DeepSeek-V3 Base/Chat models goes through the Model License. DeepSeek-V3 series (consisting of Base and Chat) supports industrial use.