Verdicts and test cases

A Run's result fills the Test result tab under the editor; a Submit's opens as its own tab over the statement. Both show one verdict for the attempt and one line per test case.

The verdicts#

Verdict	What happened
Accepted	Every case passed.
Wrong answer	Your code ran and returned something different from the expected answer.
Time limit exceeded	It was still running when the budget ran out, or a case answered correctly but slower than the problem's per-case limit. Usually an algorithm that is too slow, sometimes a loop that never ends.
Call limit exceeded	On an interactive problem, your code called the problem's API more times than one case allows. The limit is shown on the problem; it is what tells a binary search from a scan.
Memory limit exceeded	It used more memory than the sandbox allows.
Runtime error	It threw, crashed, or read past the end of something. The exception is shown.
Compile error	It did not build. The diagnostics point at lines in your editor.
Too much output	It printed far more than a program answering this problem should. Usually a stray print inside a loop.
Finished	A custom run. Nothing was graded, because there was nothing to grade it against.
Answer too large to return	Your answer was correct in shape but too big to send back. Ours to fix, not yours.
Our judge failed	Something broke on our side.

Every case is listed, not just the first failure#

Most judges say "Wrong Answer on test 14 of 20" and stop. We show what happened to all twenty, because the batch ran anyway and the difference is the whole diagnosis:

• 3 of 20 passed - the algorithm is wrong.
• 19 of 20 passed - the algorithm is right and one edge case is not handled.

Those are the same verdict and completely different problems to go and fix.

Each case shows its status and how long it took. A case that never ran shows no time, which is a different statement from a case that ran instantly.

Case statuses#

Passed and Failed are the ordinary two. Errored means that case threw. Skipped and Not reported mean the case never ran - usually because an earlier one exhausted the budget. Oversized means the case produced more output than could be returned. Too slow means the case returned the right answer but took longer than the problem's per-case time limit for your language. Call limit means the case made more API calls than the problem allows.

When a problem accepts many valid answers, a checker decides instead of a fixed expected answer, and a failed case shows the checker's reason, such as "edge 2 -> 5 is violated".

How your answer is compared#

Every problem says, under its statement, how answers are judged: exactly, in any order (and at which level), within a tolerance, or by a checker. Order-free comparisons still count duplicates, so [1,1,2] is never the same answer as [1,2,2]. When the method returns nothing, the problem says which parameter your answer is read from after it returns.

Some hidden cases are generated: built inside the sandbox from a seed at full size, often 10^5 values, because an input that large cannot be typed. If one fails and the problem reveals failing cases, it is shown as its size and seed rather than the values themselves.

Sample cases and hidden cases#

Every problem has a handful of sample cases - those are the worked examples on the Description tab, and they are fully public. Behind them sit a larger set of hidden cases, which is what Submit actually checks against.

When you fail, we reveal the first failing case - its input, the expected answer, and what your code returned - even when that case is hidden. A bare "failed on test 14" teaches nothing, and one disclosed case is a fair price for a diagnosis you can act on.

The other hidden cases stay hidden. They show as hidden rather than as three blank rows, so you can tell redaction from a bug.

Your own output still reaches you#

Anything your program printed comes back regardless of the verdict, including for hidden cases. That is your program's output about your own program, so there is nothing to redact - print debugging works here exactly as you would expect.

Output is capped. A program that prints inside a hot loop will be cut off, and very loud programs are stopped outright with Too much output.

When an old Accepted goes stale#

Occasionally a problem's test cases are corrected after people have already solved it. Adding a case is routine; changing or removing one needs an admin, and when it happens your affected submissions are marked stale rather than quietly left green.

A stale submission stays in your history and still shows what you wrote. It does not count toward your solved total, because the verdict was reached against a set of cases that no longer exists. Re-submitting the same code is usually all it takes.

Related#