Verdicts and test cases
What each verdict means, why you see every case rather than only the first failure, which two verdicts are our fault, and when an old Accepted goes stale.
The Result pane fills in after every Run and every Submit. It shows one verdict for the attempt and one line per test case.
The verdicts#
| Verdict | What happened |
|---|---|
| Accepted | Every case passed. |
| Wrong answer | Your code ran and returned something different from the expected answer. |
| Time limit exceeded | It was still running when the budget ran out. Usually an algorithm that is too slow, sometimes a loop that never ends. |
| Memory limit exceeded | It used more memory than the sandbox allows. |
| Runtime error | It threw, crashed, or read past the end of something. The exception is shown. |
| Compile error | It did not build. The diagnostics point at lines in your editor. |
| Too much output | It printed far more than a program answering this problem should. Usually a stray print inside a loop. |
| Finished | A custom run. Nothing was graded, because there was nothing to grade it against. |
| Answer too large to return | Your answer was correct in shape but too big to send back. Ours to fix, not yours. |
| Our judge failed | Something broke on our side. |
Every case is listed, not just the first failure#
Most judges say "Wrong Answer on test 14 of 20" and stop. We show what happened to all twenty, because the batch ran anyway and the difference is the whole diagnosis:
- 3 of 20 passed - the algorithm is wrong.
- 19 of 20 passed - the algorithm is right and one edge case is not handled.
Those are the same verdict and completely different problems to go and fix.
Each case shows its status and how long it took. A case that never ran shows no time, which is a different statement from a case that ran instantly.
Case statuses#
Passed and Failed are the ordinary two. Errored means that case threw. Skipped and Not reported mean the case never ran - usually because an earlier one exhausted the budget. Oversized means the case produced more output than could be returned.
Sample cases and hidden cases#
Every problem has a handful of sample cases - those are the worked examples on the Description tab, and they are fully public. Behind them sit a larger set of hidden cases, which is what Submit actually checks against.
When you fail, we reveal the first failing case - its input, the expected answer, and what your code returned - even when that case is hidden. A bare "failed on test 14" teaches nothing, and one disclosed case is a fair price for a diagnosis you can act on.
The other hidden cases stay hidden. They show as hidden rather than as three blank rows, so you
can tell redaction from a bug.
Your own output still reaches you#
Anything your program printed comes back regardless of the verdict, including for hidden cases.
That is your program's output about your own program, so there is nothing to redact - print
debugging works here exactly as you would expect.
Output is capped. A program that prints inside a hot loop will be cut off, and very loud programs are stopped outright with Too much output.
When an old Accepted goes stale#
Occasionally a problem's test cases are corrected after people have already solved it. Adding a case is routine; changing or removing one needs an admin, and when it happens your affected submissions are marked stale rather than quietly left green.
A stale submission stays in your history and still shows what you wrote. It does not count toward your solved total, because the verdict was reached against a set of cases that no longer exists. Re-submitting the same code is usually all it takes.
Related#
Was this page helpful?