This might be inherent to how the models are benchmarked. Aren’t some benchmarks... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

jiehong 75 days ago | parent | context | favorite | on: Agents need control flow, not more prompts

This might be inherent to how the models are benchmarked.

Aren’t some benchmarks giving the model multiple shots at a problem and only keep the successful result if it appeared, ignoring the failure rate?

andyferris 75 days ago [–]

Good point. We need the mean, “any 1 of 10” and the “all 10 of 10” success rates in the metrics, so we can estimate reliability (the last one).

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact