Fix gpt-5/reasoning-model accuracy: model-aware max_tokens default (#260)
Fixes #255. ## Root cause Reasoning models (gpt-5, o-series) spend **hidden reasoning tokens from the same `max_completion_tokens` budget** as the visible answer. Lotus's flat `max_tokens=512` default starves them: 1. On non-trivial rows, gpt-5 burns the entire 512-token budget on internal reasoning (`finish_reason='length'`) 2. The completion comes back with `content=''` (or OpenAI 400s with *"Could not finish the message because max_tokens or model output limit was reached"*) 3. `filter_postprocess` finds neither output token in the empty string and **silently coerces the row to `default=True`** — bad accuracy, zero errors ## Empirical confirmation (debug branch `debug-gpt5-repro`, runs in Actions) | Scenario | reasoning tokens | result | |---|---|---| | trivial sentiment claim @512 | 64 | `Answer: True` ✓ (8/8 — easy tasks unaffected) | | hard arithmetic claim @512 | 448–512 | one row empty → defaulted → **5/6** | | same, @8000 budget | ~512 | **6/6** | | hard ZS_COT @512 | — | truncated/empty rows → **5/6** | | forced 64-token budget | 64 | `finish_reason=length`, `content=''`, sometimes OpenAI 400 | | gpt-4o-mini @512 (control) | 0 | **6/6** | On real workloads (longer docs, harder predicates) most rows blow the 512 budget, which is why gpt-5 looked uniformly broken. ## Fix - `LM` default `max_tokens` is now model-aware: **8192 for reasoning models** (detected via `litellm.supports_reasoning`), 512 otherwise. An explicit `max_tokens` always wins. `max_completion_tokens` is a cap, not a purchase — this only affects worst-case per-row cost on reasoning models. - `_get_top_choice` now **warns on `finish_reason='length'`** with a reasoning-model-specific hint, so this failure mode is never silent again. - Users can still pass `reasoning_effort="minimal"` (forwarded via kwargs) to trade reasoning depth for cost — verified working in the probes. ## Tests - New offline `tests/test_lm_defaults.py` (7 tests, no API calls) pinning the model-aware defaults; wired into the `settings` CI suite. - End-to-end verification run on the debug branch: gpt-5 with pure library defaults on the previously-failing hard set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
L
liana313 committed
bdaa5dc5d95ee32e5995000e5dfd0a1d71b12a87
Parent: 8a271b3
Committed by GitHub <noreply@github.com>
on 6/12/2026, 12:22:53 AM