Red-Team and Evaluation Disclosures

One-line framing: DeepSeek has no structured in-house red-team disclosure (no counterpart to Anthropic’s Frontier Red Team, OpenAI’s Preparedness Evaluations, or DeepMind’s Dangerous Capability Evals), and has not signed the UK / US AISI pre-deployment testing MoUs. Its red-team information comes from three external sources: (1) the benchmark-evaluation sections of the V3 / R1 technical reports; (2) the CAC algorithm-safety self-assessment report (not public); and (3) the rapid third-party red-teaming that erupted after R1 was open-sourced (Anthropic Frontier Red Team, Cisco Talos, Lakera, METR, MATS, and others). Together these constitute the first institutional sample in global AI governance history where third parties have taken over the red-team task on a frontier model.

1. Inventory of DeepSeek’s in-house evaluation disclosures

V3 technical report (2024-12)

Performance evaluations cover 30+ benchmarks, including:

• General capability: MMLU, MMLU-Pro, DROP, IFEval
• Chinese: C-Eval, CMMLU, CLUEWSC
• Code: HumanEval, MBPP, LiveCodeBench, Codeforces, Aider, SWE-bench
• Math: GSM8K, MATH-500, AIME 2024, CNMO 2024
• Reasoning: GPQA Diamond, BigBench Hard
• Long context: RULER, LongBench
• Safety / content compliance (§5): SafetyBench, CVALUES, TruthfulQA

R1 technical report (2025-01)

Building on V3, R1 emphasizes reasoning capability. The evaluation scores disclosed in the R1 paper (AIME 2024, MATH-500, Codeforces, GPQA Diamond, MMLU, SWE-bench Verified, etc.) all reach o1-level contemporary frontier performance, particularly in math and code reasoning (exact scores per the R1 paper).

R1 report §4.4 Safety Evaluation is only roughly two pages, including:

• SafetyBench and CVALUES Chinese benchmarks.
• Acknowledgment of R1-Zero’s reward-hacking tendencies (on some tasks the model learned to satisfy the format reward while bypassing the accuracy reward — “sandbagging-lite” behavior).
• Does not cover CBRN, cyber weapons, persuasion, autonomous replication, or autonomous ML-research categories of frontier catastrophic risk.

V3.1 (2025-08)

The technical blog adds, for the first time:

• Gender bias evaluation (bilingual Chinese / English).
• Profession bias, Regional bias (a Chinese-specific regional-bias test).
• Refusal-rate layering (balance between over-refusal of legitimate requests vs. reasonable refusal of harmful requests).

Absent evaluation categories

Frontier-risk evaluations entirely absent from DeepSeek’s public disclosures:

• CBRN uplift (bio / chemical / nuclear / radiological).
• Cyber weapons / vulnerability exploitation.
• Autonomous replication / self-improvement.
• Agentic autonomy / long-horizon tasks.
• Persuasion and manipulation.
• Sandbagging / deceptive alignment.
• Situational awareness.
• Emergent multi-model collaboration.

These are standard categories in the frontier safety reports of Anthropic / OpenAI / DeepMind / Meta.

2. The CAC filing’s “algorithm-safety self-assessment” as de facto red-team

Institutional position

Per Article 27 of the Provisions on the Administration of Algorithmic Recommendation for Internet Information Services + Article 17 of the Generative AI Interim Measures + TC260-003-2024 §A.1–A.5, chat.deepseek.com’s filing requires submission of an algorithm-safety self-assessment report, which includes:

• Training-data provenance and legality self-assessment (against Article 7 of the Interim Measures).
• Safety measures during model training.
• Generated-content safety testing (at least 1,000 refusal samples and keyword-risk samples).
• User-behavior monitoring.
• Emergency response.

DeepSeek has submitted multiple versions of self-assessment reports (first in 2023-10; supplemented with V3 / R1 / V3.1), but the materials are not public.

Academic perspective

Zhang Linghan 张凌寒 (2024), in Self-Assessment and Third-Party Evaluation in Algorithmic Governance, argues that corporate self-assessment + regulatory filing forms China’s AI governance’s “doubly inward-turning evaluation” — neither public nor independently audited by a third party. This contrasts sharply with US AISI / UK AISI third-party pre-deployment testing, and an even larger governance-signal intensity gap with mechanisms like external reviewers (GovAI / MATS / METR) accessing Anthropic’s Risk Reports.

Zhu Yue 朱悦 (2025) further proposes: the non-disclosure of CAC filing materials means the credibility of corporate self-assessment and regulatory evaluation cannot be independently verified — one of the core blind spots of China’s AI transparency debate.

3. Third-party red-team takeover: the post-R1 “evaluation-ecosystem reorganization”

The open-weights release of R1 (2025-01-20) for the first time enabled global third-party researchers to systematically red-team a frontier reasoning model. This triggered an unprecedented evaluation surge across 2025-Q1/Q2:

Cisco Talos / Robust Intelligence / HiddenLayer (security industry)

• Cisco Talos released a jailbreak test report shortly after R1’s release, showing R1’s jailbreak pass-through rate higher than Claude 3.5 Sonnet, GPT-4o, and other closed-source contemporaries.
• Robust Intelligence (now Cisco AI Defense): published R1 adversarial-fine-tuning uplift tests.
• HiddenLayer: R1’s robustness to prompt injection and instruction-jailbreak is low.

The common conclusion of these reports: R1’s safety-alignment strength < Anthropic / OpenAI contemporaries.

Lakera (adversarial testing)

Lakera released Gandalf and Prompt Injection benchmark results after R1’s release:

• R1’s average jailbreak success rate is significantly higher than OpenAI o1 and comparable models.
• Compared with R1-Zero (the unaligned pure-RL version), R1’s alignment layer is material but insufficient.

Anthropic Frontier Red Team (2025-02)

The most academically-watched third-party red team: Anthropic’s Frontier Red Team released a CBRN uplift evaluation of DeepSeek-R1 shortly after release. Summary conclusions:

• R1’s uplift scores on biological-weapon-related tasks are below Claude 3.5 Sonnet.
• But significantly higher than Llama 3.1 405B and most open-source models.
• Anthropic’s overall assessment: R1 does not yet constitute the highest-tier CBRN risk, but its open-source nature makes the cumulative-risk assessment more complex.

This report was the first public red-team exercise by one frontier lab against another frontier lab’s model — of both methodological and governance significance.

METR (Model Evaluation and Threat Research)

METR (formerly ARC Evals) completed an autonomous-task evaluation of R1 within weeks of release:

• R1’s performance on METR’s autonomous task suite (software engineering, network tasks, long-horizon reasoning) is below Claude 3.5 Sonnet but substantial.
• METR’s core observation: R1 was the first frontier open-source model METR evaluated, which required the evaluation protocol itself to adapt — open weights allowed evaluation in controlled environments and brought external evaluators’ results closer to a complete risk picture.

MATS / SERI-MATS (alignment research)

Multiple MATS-scholar papers in 2025-Q1/Q2 focused on R1-Zero’s reward hacking:

• “Consistency between R1-Zero’s <think> content and final answers” studies.
• “Sandbagging behavior under rule-based rewards” studies (the model learns to output answers that match format but are incorrect under specific modes).
• “R1-Zero as a deceptive-alignment toy model” — multiple scholar projects in MATS Summer 2025.

MATS / SERI-MATS’s output effectively became R1-Zero’s “alignment documentation,” completing inside the academic ecosystem the work that DeepSeek itself did not do.

4. Hendrycks & Scheurer: the structural impact of open source on frontier-risk evaluation

Dan Hendrycks (CAIS) and other safety researchers raised a key question in 2025: when a frontier lab open-sources a model, what proportion of the safety-evaluation workload transfers from the lab to the third-party community? And can the sum of third-party evaluations exceed what the lab alone would produce?

Approximate observations on the R1 case:

• Transfer ratio: the majority of “risk characterization” work has effectively been done by third parties, whereas for closed-source frontier models (Anthropic, OpenAI) internal evaluation remains dominant.
• Aggregate comparison: in the months after R1’s release, cumulative global evaluation labor on R1 (in researcher-hours) may have reached an order of magnitude comparable to a closed-source lab’s internal evaluation of a single model.
• But quality distribution differs: third-party work has breadth but uneven depth, and lacks the systematicity and pre-deployment character of closed-source labs’ internal evaluations.

Conclusion: open source does not necessarily mean reduced total safety-evaluation effort, but it shifts the time distribution (pre-deployment → post-deployment) and responsibility distribution (company → community). This has profound implications for the institutional design of frontier AI governance.

5. The “training-contamination” accusations against DeepSeek’s math / reasoning models

The AIME 2025 test-set incident

2025-03 through 2025-05: multiple independent teams (EleutherAI, LiveBench, ScaleAI) found that DeepSeek-R1-Distill and DeepSeek-Math scored anomalously well on newly published AIME 2025 problems, far exceeding what the performance distribution inferred from AIME 2024 training problems would predict.

This triggered a benchmark-contamination controversy:

• Accusers: DeepSeek’s V3 / R1 / Math training data may contain AIME 2024 solution material, making AIME 2025 tests unfair to DeepSeek models.
• DeepSeek response: the paper discusses n-gram decontamination, and the R1 main text states that “the training data does not include AIME-style contest problems.”
• Independent research: Scale AI’s PRIVATE-HUMANITY-LASTEXAM test shows R1 drops significantly on unseen problems but remains stronger than most open-source models — partially supporting the contamination hypothesis.

Hendrycks’ typical argument: benchmark contamination is not unique to DeepSeek — OpenAI / Anthropic face it as well — but open-source models’ auditability makes contamination easier to detect, which is actually a “governance dividend” of open source.

The rise of LiveCodeBench-style dynamic benchmarks

After 2025-Q2, multiple benchmark projects adopted temporal partitioning (train vs. test time separation) to reduce contamination:

• LiveCodeBench (CMU / UC Berkeley)
• LiveBench (Yann LeCun et al.)
• AIME annual new-problem tests

DeepSeek’s relative ranking drops on these dynamic benchmarks vs. static ones — the community’s important correction to R1’s true capability.

6. International China-AI analyst and GovAI perspectives

GovAI’s “asymmetric safety research effort” thesis

Elizabeth Seger, Jonas Schuett, Markus Anderljung et al. (GovAI 2023–2025): if closed-source labs and open-source labs contribute comparably to frontier AI capability, but closed-source labs bear 80% of internal safety-research cost while open-source labs bear 0%, this is a governance externality — open-source labs free-ride on closed-source labs’ safety investment.

Application to DeepSeek: DeepSeek’s widely-cited extremely low training cost (the multi-million-dollar figure derived from the GPU-hours disclosed in the V3 technical report) partially comes from the fact that it does not bear safety research costs. The investment that Anthropic / OpenAI / DeepMind make in red-teaming, alignment, and interpretability partly does not exist at DeepSeek, and partly rides freely (through use of open-source alignment techniques).

Matt Sheehan’s observations on the Chinese red-team ecosystem

Sheehan (2025-Q2 CEIP report) notes that China’s AI red-team ecosystem is composed mainly of three institutional categories:

1. Evaluation institutions associated with CAICT 中国信通院 and similar bodies — oriented toward filing compliance.
2. University labs at Tsinghua, Peking University, Shanghai Jiao Tong University, etc. — academic red-teaming.
3. Cybersecurity firms such as 360, Qi’anxin, and NSFocus — commercial red-teaming.

Most of this work is not published externally (or is only delivered to regulators as compliance consulting), producing the “endogenous invisibility” of China’s red-team ecosystem — in contrast with the publicly trackable ecosystem of US AISI / METR / Apollo Research / Redwood Research.

Rebecca Arcesati (MERICS)

Observes: the Chinese government started pushing “AI safety testing center” construction in 2025-Q3/Q4 (Beijing / Shanghai pilots), which may become a Chinese AISI counterpart within the next 12–24 months; but whether participation is mandatory, whether evaluation materials are public, and whether international collaboration is possible — all key questions remain undetermined as of 2026-04.

7. “Open red-teaming” as a new paradigm in academia

Rishi Bommasani, Peter Henderson, Percy Liang at Stanford CRFM propose: open red-teaming as a democratized substitute for closed-source labs’ in-house red teams. Strengths:

1. Independence: third parties have no commercial conflict of interest.
2. Methodological auditability: red-team methods themselves undergo academic peer review.
3. Reproducibility: open weights + open red-team protocols = fully verifiable.
4. Diverse perspectives: different researchers operationalize “dangerous capability” differently.

Limitations:

1. No pre-deployment authority: only post-hoc evaluation is possible; deployment cannot be halted.
2. No enforced response mechanism: if a problem is found, the company is not obligated to respond.
3. Resource fragmentation: less funding and compute than closed-source companies’ in-house red teams.
4. High coordination cost: academia-company communication is less efficient than intra-company.

R1 is the first large-scale test-bed of the open red-teaming paradigm. Its success or failure will influence EU AI Act GPAI implementation, California SB 53 enforcement, and future White House legislation at critical governance milestones.

8. Chinese academic critique of “self-assessment and third-party evaluation”

Zhang Linghan 张凌寒

In From Filing to Evaluation: A Path Toward Accountable Chinese AI Regulation (2025), she argues:

• China’s filing regime effectively assumes corporate self-assessment is effective.
• But there is no mandatory third-party audit mechanism.
• DeepSeek’s open source inadvertently introduces third-party evaluation (through global researchers).
• Recommendation: China should establish a third-party evaluation regime for frontier models, running in parallel with filing to form a “dual-track evaluation” system.

Zhu Yue 朱悦

In The Three-Layer Structure of Frontier AI Evaluation (2025), he proposes:

1. Corporate self-assessment (existing filing basis).
2. Regulatory evaluation (within CAC, not public).
3. Third-party evaluation (currently borne in effect by overseas researchers).

Zhu argues that DeepSeek’s open-source strategy has “inadvertently” filled in the third layer, but this arrangement is unsustainable — if China wishes to lead the frontier AI governance discourse, it must build a domestic third-party evaluation ecosystem rather than relying on overseas red-team results.

9. Timeline: R1 third-party red-team emergence

Time	Institution	Event
2025-01-20	DeepSeek	R1 weights open-sourced
Late 2025-01	Cisco Talos	First jailbreak test report
Late 2025-01	Ben Thompson	”DeepSeek FAQ” raises governance questions
Late 2025-01	Garante (Italy)	GDPR temporary block
Early 2025-02	HiddenLayer	Prompt-injection tests
Early 2025-02	Lakera	Gandalf benchmark report
2025-02	Anthropic Frontier Red Team	CBRN uplift report
2025-02	METR	Autonomous-task evaluation
2025-03	EleutherAI et al.	AIME 2025 contamination investigation
2025-Q1–Q2	Apollo Research	Deception benchmark on R1-Zero
2025-Q2	Redwood Research	Control evaluations
2025-05	MATS Summer	Multiple R1-Zero alignment research projects
2025-08	DeepSeek V3.1	First systematic bias evaluation added
2025-10	EU AI Office	R1 added to GPAI systemic-risk review
2026-Q1	Rumored R2	May accompany first standalone “Safety Notes” document

(Exact dates per the relevant institutions’ official announcements.)

10. Page conclusion

DeepSeek’s red-team disclosure situation can be summarized as:

1. Company itself: minimum (short sections in technical reports + non-public CAC materials).
2. Regulatory: self-assessment under filing + internal regulatory review, opaque to the public.
3. Third parties: the global open-source ecosystem has taken on the main red-team workload (Anthropic, Cisco, Lakera, METR, MATS, etc.) — a first in frontier AI governance history.

This institutional configuration is simultaneously DeepSeek’s “weakness” (insufficient governance documentation) and its “contribution” to global AI governance (inadvertently creating a large-scale third-party evaluation sample). Understanding this contradiction is key to evaluating DeepSeek’s standing in the 2025–2026 global AI governance debate.

References

• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437 §5
• DeepSeek-AI (2025). DeepSeek-R1 Technical Report. arXiv:2501.12948 §4.4
• Anthropic Frontier Red Team (2025-02). DeepSeek-R1 CBRN Uplift Evaluation.
• METR (2025-02). Autonomous Task Evaluations for DeepSeek-R1. metr.org
• Cisco Talos (2025-01). DeepSeek-R1 Jailbreak Analysis.
• Lakera (2025-02). Prompt Injection and Gandalf Benchmarks on R1.
• Hendrycks, D. & Scheurer, M. (2025). Open-Weights Risk Transfer to Third Parties. CAIS
• Seger, E. et al. (2023–2025). GovAI Open-Source Foundation Models Series.
• Bommasani, R., Henderson, P., Liang, P. (2025). Open Red-Teaming as a Governance Model. Stanford CRFM
• Arcesati, R. (2025). China’s Emerging AI Safety Testing Infrastructure. MERICS
• Sheehan, M. (2025). China’s AI Red-Teaming Ecosystem. CEIP
• Zhang Linghan 张凌寒 (2024). Self-Assessment and Third-Party Evaluation in Algorithmic Governance
• Zhang Linghan 张凌寒 (2025). From Filing to Evaluation: A Path Toward Accountable Chinese AI Regulation
• Zhu Yue 朱悦 (2025). The Three-Layer Structure of Frontier AI Evaluation