How fast is my solution

Once a submission is accepted, a few different things can tell you something about its speed. All of them are measurements. None is an analysis of your code, and that distinction is the whole design.

The Performance report#

A minute or so after a submission is accepted, the Performance panel under it fills in. It answers two questions: how your solution's speed compares with the problem's reference solution, and where it sits among other people who solved the problem in your language.

What it shows:

• "1.08x the reference solution's time at n = 1,000,000" - your fastest call divided by the reference's fastest, on the same inputs.
• "Faster than 72% of the 40 people measured in Java" - and a band such as Top 25% or Middle half. If your result sits near a band's edge, you see both ends, for example Top 10% to Top 25%, rather than whichever end flatters you.

How it is measured#

• Your solution and the reference run on fresh, randomly generated inputs of the problem's size, inside the judge's sandbox, timing only the call itself - not starting the language runtime, not reading the input.
• Your solution and the reference each run twice, taking turns, on the same machine. Dividing by the reference cancels out how fast or busy the machine was. If the reference's two runs disagree, or yours do, the machine was too busy to measure fairly: the reading is thrown away and tried again later.
• The fastest call on each side is compared, because a busy machine only ever adds time.
• The measurement runs in the background, only when the judge is quiet, so it never slows down anybody's Run or Submit, and it never uses your daily runs.

How the ranking works#

• Each person counts once, by their best measured solution. Submitting the same problem thirty times does not push anybody else down.
• Within one language, and on one version of the judge. When the judge's language runtime is upgraded, measuring starts again on the new version rather than mixing two clocks.
• Ties count half. Results within 3% of each other are a tie, so a solution level with everybody else sits in the middle, not at the bottom.
• Hidden until 20 people have been measured on that problem in your language. Before that you still see the comparison with the reference, and how many people have been measured so far.

When there is no ranking#

You see	Why
Measuring performance	The measurement is queued or running. It waits for a quiet moment on the judge.
Under 500 microseconds	Your solution is so fast at this size that the timer cannot tell solutions apart, so it is not ranked.
Returned a different answer from the reference	On a fresh input your solution disagreed with the reference. Your verdict stands - it passed every test case - but it is not ranked.
The judge was too busy	Every attempt to measure found a noisy machine. No number is better than a misleading one.
Inputs too small to separate solutions	The problem has no large generated inputs, so there is nothing fair to time.
Test cases changed	The problem's tests changed after your submission was judged. It is re-checked on the new tests first.

The whole-run figures#

The Runtime and Memory tiles at the top of an accepted result are the judge's figures for the whole run: starting the language runtime and running every test case, not just your code. For Java, starting the runtime alone takes over a second, so these numbers mostly say which language you used. They are labelled Whole run for that reason.

Where a whole-run comparison is shown ("Whole run: faster than 73%"), it follows the same rules as the ranking above: each person once, ties count half, within one language, hidden below 20 people.

Among each person's best (whole run) is a histogram of those figures, with your bar marked You. It appears once a handful of solutions have been accepted in that language; before that the panel says how many it is waiting for.

Per test case shows how long each test case took inside the judge, slowest one named. It is always there, because it is about your submission alone - and "which case was slow" is often the more useful question.

Reopening a past accepted submission on the Submissions tab shows the same report and charts.

Analyse my solution#

The growth analysis is opt-in. On an accepted submission you may find an Analyse my solution button. It runs your accepted code again at four increasing input sizes, times each, and reports what happened to the clock.

You get the four sizes, the four times, and a sentence:

Your solution's runtime roughly quadrupled when the input doubled.

That sentence is true, it is falsifiable against the numbers printed beside it, and it teaches the idea better than a symbol does.

What the answers can be#

Result	What was observed
Flat / Sublinear	Time barely moved as the input grew.
Linear	Doubling the input roughly doubled the time.
Indistinguishable	Between linear and n log n - at these sizes they differ by less than the noise, so we say so rather than pick one.
Quadratic	Doubling the input roughly quadrupled the time.
Cubic or worse	It grew faster than quadratically.
Too fast to measure	Every size finished so quickly that the timings are mostly overhead. Not a verdict on your solution - the measurement could not see it.
Not enough data	Too few sizes completed to fit anything.

When the button is not there#

It is absent rather than disabled when the problem cannot be profiled. Measuring needs an input generator written by the problem's author in your language, and most problems will not have one for a long time. Rather than offer a button that spends a sandbox slot to produce a failure, we do not draw it.

It is also only offered on accepted submissions. Profiling a wrong answer measures how fast it is wrong.

It measures once#

A finished analysis is stored and shown again without re-running anything. Your source has not changed, so re-measuring would spend a run to produce the same number plus the noise of a shared machine - and would show you two different slopes on two visits, which teaches that the measurement is arbitrary.

If the problem's test cases change afterwards, the stored analysis is marked stale.

Related#