LangMatch

1 · Background

LLM inference cost scales with context length: longer prompts mean more tokens, higher latency, and more context pressure. Prompt-compression research shows that shrinking the input while preserving behaviour is a real efficiency lever. A second line of work shows tokenizers are not language-fair — the same meaning costs a different number of tokens in different languages — and cross-lingual prompting shows the prompt language changes not just token count but the model's behaviour, reasoning path, and format compliance.

Classical Chinese is exceptionally dense: heavy ellipsis, high information density, terse grammar. That makes it an intuitive candidate for a “natural-language compression prompt.” But the intuition does not settle two things: (1) is Wényán actually cheaper under the tokenizer, and (2) if it is, does task success survive?

2 · Objective & research questions

The key comparison is not “which language is shorter,” but: under fixed tasks and fixed model backends, does a more compressed prompt language produce a better quality–cost frontier? Only the language of the system prompt changes; the task is held constant.

• RQ1 — Under semantic equivalence, does a Wényán system prompt reliably reduce input tokens?
• RQ2 — Does any token saving come at the cost of success rate / instruction-following / accuracy?
• RQ3 — Do differences come mainly from tokenizer segmentation, or from style-induced behaviour change?
• RQ4 — Does the effect reproduce across model families (open vs. closed)?

3 · Related work

• Ancient-Chinese LLMs (C3Bench, TongGu, Fuxi, WenyanGPT) treat Classical Chinese as a first-class, separately-modelled language — but focus on understanding / generation, not token efficiency.
• Cross-lingual prompting shows the prompt language shifts model behaviour, reasoning, and format compliance — not just surface tokens.
• Token tax / tokenizer fairness shows the same information costs differently across languages — language itself is a cost variable.
• Prompt compression (LLMLingua, Selective Context) proves shrinking prompts is valuable — but via algorithmic compression, not by exploiting a language's own expressive density.

These four lines have never been combined into one testable question: can Classical Chinese act as a natural-language compression prompt? LangMatch is that controlled test — it does not train a new Wényán model, it evaluates Wényán as a system-prompt language under strict variable control.

4 · Experiment design

Three functionally-equivalent system-prompt modes — same task, same models, same output budget; only the prompt language changes:

• Models: gpt-4o, gpt-5.4, qwen3-1.7b, qwen3-4b — API and local (Transformers) backends under one pipeline
• Benchmarks: IFEval (instruction-following), MATH-500 (text-only subset, final-answer accuracy), MMLU-Pro
• Metrics: prompt / completion / total tokens; benchmark Score or SR; and Score per 1k tokens as the efficiency axis
• Budget & auditing: a unified 2048-token output budget with per-run token accounting; the runner records configured_max_tokens, finish_reason, and usage, so truncation is detected rather than silently scored

Manifest (fixed task subsets)
  → run all task × prompt-mode × model combinations
  → raw results.jsonl + per-run summaries
  → cross-run aggregation into matrix reports
  → Pareto frontier plots (score vs. total tokens)

Rigor note. An early pilot used tight caps (IFEval 192, MATH-500 64, MMLU-Pro 8) and produced truncation artifacts — most sharply, gpt-5.4 / IFEval returned empty visible output while still consuming the full completion budget, making the model look weak. The main matrix was rerun at 2048, which cleared the artifacts for gpt-4o, gpt-5.4, and qwen3-4b (0 cap-hits); qwen3-1.7b retains 4 cap-hits (finish_reason=length) and is read cautiously. All numbers below are the corrected 2048 rerun.

A follow-up 3-run study holds the language fixed (base / zh / wy) and instead varies the interaction setting — explicit_process (visible reasoning), hidden (think but don't show), compact_visible (brief visible reasoning) — to separate the effect of prompt language from that of reasoning visibility.

5 · Results

5.1 Main matrix (2048-token budget)

Overall, aggregated across benchmarks (each point one model × prompt-mode combination):

gpt-5.4 is the strongest model overall, and its earlier weak IFEval showing was confirmed to be a truncation artifact, not a real deficit. zh_compact is the most stable prompt family (best for gpt-4o, ties best for qwen3-4b); wy is a conditional gain — the single strongest overall mode on gpt-5.4.

Pareto-style score vs. total tokens, per benchmark (click any figure for full size):

IFEval — qwen3-4b hits the same score at the lowest token cost; gpt-5.4's wy variant is its cheapest.

MATH-500 — preferences split by model: gpt-4o favours zh_compact, gpt-5.4 favours wy.

MMLU-Pro — zh_compact holds up as the robust default.

IFEval

After removing the cap artifact, gpt-5.4 reaches 1.000 SR under all three modes; qwen3-4b also saturates. On this benchmark high-capacity models satisfy the task under multiple prompt languages.

MATH-500

The headline split: zh_compact is best for gpt-4o, while wy is best for gpt-5.4 — Wényán becomes a real quality–cost contender on capable models and concise reasoning tasks. qwen3-4b's best score is under wy, while zh_compact stays slightly cheaper.

MMLU-Pro

The most conservative benchmark: it does not reward Wényán as clearly. Once truncation is removed, gpt-5.4 is uniformly strong across modes; gpt-4o still prefers base, qwen3-4b still prefers zh_compact — a model-task interaction that survived the rerun.

5.2 LangMatch 3-run (interaction settings)

Language fixed; interaction setting varied. Overall SR vs. tokens under explicit, hidden, and compact-visible reasoning (IFEval panels are missing-data placeholders in this export, not negative results):

Explicit reasoning — gpt-5.4 + wy is the standout point.

Hidden reasoning — mainly shifts points left (cheaper), not up.

Compact-visible — a middle ground between cost and score.

explicit_process

hidden

compact_visible

gpt-5.4 + wy is the standout under both explicit_process and compact_visible (SR 0.833); hidden mainly reduces cost rather than raising scores; qwen3-4b shows no stable Wényán advantage.

Full result reports:

• Main matrix report (2048 budget) — 中文版
• 3-run comparison report (中文) — with the token / score table

6 · Takeaways

• Prompt-language effects are real, but strongly model- and task-dependent.
• zh_compact is the most robust default across closed and open families.
• wy is a serious candidate on strong models — its gains on gpt-5.4 survive the corrected rerun, and it is best on gpt-5.4 / MATH-500 — but it is not universal.
• In the 3-run view, reasoning visibility matters alongside language; hidden is a cost-reduction setting, not a score-maximizing one.

Working conclusion: Classical Chinese functions as a conditional prompt-compression strategy, not a universal one — worth switching on per model, not by default.

7 · Rigor & limitations

• The 2048 rerun cleared truncation artifacts for gpt-4o, gpt-5.4, and qwen3-4b; qwen3-1.7b still has 4 cap-hits and is interpreted cautiously.
• Token counts come from source-native backends (openai for GPT logs, transformers for qwen logs) — read them within-model, not as tokenizer-identical across providers.
• Prompt modes are functionally close but not perfectly translation-aligned; a style-vs-density confound is mitigated by including the Modern-Chinese compact control (zh_compact), not fully eliminated.

Early-stage — findings are preliminary observations, not a formal result.