SIGN IN SIGN UP

Fix gpt-5/reasoning-model accuracy: model-aware max_tokens default (#260)

Fixes #255.

## Root cause

Reasoning models (gpt-5, o-series) spend **hidden reasoning tokens from
the same `max_completion_tokens` budget** as the visible answer. Lotus's
flat `max_tokens=512` default starves them:

1. On non-trivial rows, gpt-5 burns the entire 512-token budget on
internal reasoning (`finish_reason='length'`)
2. The completion comes back with `content=''` (or OpenAI 400s with
*"Could not finish the message because max_tokens or model output limit
was reached"*)
3. `filter_postprocess` finds neither output token in the empty string
and **silently coerces the row to `default=True`** — bad accuracy, zero
errors

## Empirical confirmation (debug branch `debug-gpt5-repro`, runs in
Actions)

| Scenario | reasoning tokens | result |
|---|---|---|
| trivial sentiment claim @512 | 64 | `Answer: True` ✓ (8/8 — easy tasks
unaffected) |
| hard arithmetic claim @512 | 448–512 | one row empty → defaulted →
**5/6** |
| same, @8000 budget | ~512 | **6/6** |
| hard ZS_COT @512 | — | truncated/empty rows → **5/6** |
| forced 64-token budget | 64 | `finish_reason=length`, `content=''`,
sometimes OpenAI 400 |
| gpt-4o-mini @512 (control) | 0 | **6/6** |

On real workloads (longer docs, harder predicates) most rows blow the
512 budget, which is why gpt-5 looked uniformly broken.

## Fix

- `LM` default `max_tokens` is now model-aware: **8192 for reasoning
models** (detected via `litellm.supports_reasoning`), 512 otherwise. An
explicit `max_tokens` always wins. `max_completion_tokens` is a cap, not
a purchase — this only affects worst-case per-row cost on reasoning
models.
- `_get_top_choice` now **warns on `finish_reason='length'`** with a
reasoning-model-specific hint, so this failure mode is never silent
again.
- Users can still pass `reasoning_effort="minimal"` (forwarded via
kwargs) to trade reasoning depth for cost — verified working in the
probes.

## Tests

- New offline `tests/test_lm_defaults.py` (7 tests, no API calls)
pinning the model-aware defaults; wired into the `settings` CI suite.
- End-to-end verification run on the debug branch: gpt-5 with pure
library defaults on the previously-failing hard set.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
L
liana313 committed
bdaa5dc5d95ee32e5995000e5dfd0a1d71b12a87
Parent: 8a271b3
Committed by GitHub <noreply@github.com> on 6/12/2026, 12:22:53 AM