How to Read a Difficulty Curve Without Telemetry Data

You don't need an analytics pipeline to find out where your difficulty curve breaks. Three cheap sources — a local attempt log you write yourself, recordings of other people playing, and store reviews — cover most of the ground, as long as you know which of them tend to lie.

What follows is a working method rather than a theory: what to record, what to ignore, and how a handful of lines in a text file can be enough to justify rebuilding a level. Everything here runs on a text file and a spreadsheet.

The dashboard is not the missing piece

Teams without telemetry usually assume a funnel chart would answer their design questions. It mostly wouldn't. A funnel can tell you that a large share of players stopped at level 7 — something a dozen watched playtests also tell you — but it can't tell you why they stopped, and why is the only part you can act on.

What small teams actually lack is not data volume but discipline: writing down what happened while it is still fresh, in a shape you can compare against next week. Difficulty is a curve over time, and a single session tells you nothing about slope. Two sessions of the same level, a week apart, with the same notes template, already tell you something.

Three sources you already have

• Your own session notes. One line per attempt: level id, attempt number, duration, outcome, and one short observation. Written during the session, not afterwards from memory.
• Recorded playthroughs. Screen capture plus, if the player agrees, a webcam or just their voice. A player's face and hands leak more information than their answers do.
• Store reviews, comments and DMs. Unstructured, biased, and still the only source that covers players you will never meet.

None of these needs a backend, an SDK, or consent forms for identifiers you shouldn't be collecting anyway. If you do write a local log, keep it anonymous: level ids and timings, no device identifiers, no names.

Four signals that mislead

Self-reported difficulty

Ask someone how hard a level was and they answer a different question: how competent they want to appear. People who failed nine times routinely call a level fair; people who cleared it on the first try call it confusing because one icon was unclear. Treat the number they give you as a mood reading, and read the behaviour instead.

Average completion time

Averages hide exactly the shape you are hunting for. A level where half the testers finish in 40 seconds and half grind for four minutes has the same average as one where everybody takes two minutes. With small samples, always look at the spread of individual attempts, not the middle. A sorted list of ten numbers is more honest than their mean.

Your own playtime

You built the level, so you cannot experience its difficulty. This gets worse over the project: by month three you read enemy patterns and card combinations as language, not as puzzles. Any judgement of the form this is obviously easy should be deleted from your notes.

The loudest review

A detailed one-star review about an unfair wave feels like a design verdict. It is one data point with a strong voice. The useful move is to check whether the same mechanic shows up in unrelated reviews written by people who are otherwise positive. One angry review is noise; the same complaint phrased three different ways by three different people is a signal.

Signals that hold up

• Retry clustering. Not the failure rate, but where failures bunch together. Five attempts on one level and one on everything around it is a spike, and spikes are the only thing you can fix precisely.
• The quit point in a recording. People rarely quit at the moment of failure. They quit one or two attempts later, usually after a retry that started slower than the one before. That earlier attempt is where the level actually lost them.
• Repeated identical mistakes. If four players make the same wrong move in the same spot, that is not a difficulty problem; it is a teaching problem. The fix is usually in feedback or readability, not in numbers.
• Attention leaving the screen. A tester glancing at their phone mid-level is a stronger churn signal than anything they say afterwards.
• Reviews that describe mechanics. Reviews naming a specific object, wave or card are usable. Reviews describing a feeling are not, unless several of them describe the same feeling in the same place.

A thirty-line local logger is enough

If you can write a CSV file, you have telemetry — just local. Write one row per attempt and read the file yourself:

using System.Globalization;
using System.IO;
using UnityEngine;

public static class AttemptLog
{
    static readonly string FilePath =
        Path.Combine(Application.persistentDataPath, "attempts.csv");

    public static void Write(string levelId, int attempt, float seconds, string outcome)
    {
        if (!File.Exists(FilePath))
            File.AppendAllText(FilePath, "level,attempt,seconds,outcome\n");

        string line = string.Format(CultureInfo.InvariantCulture,
            "{0},{1},{2:F1},{3}\n", levelId, attempt, seconds, outcome);

        File.AppendAllText(FilePath, line);
    }
}

Call it once when a level ends. After a playtest round, the whole analysis fits in one command:

awk -F, 'NR>1 { tries[$1]++; if ($4 == "win") wins[$1]++ }
         END { for (l in tries)
                 printf "%s\t%d attempts\t%d clears\n", l, tries[l], wins[l] }' \
    attempts.csv | sort

That output — attempts and clears per level, side by side — is enough to find the spike. Everything after that comes from watching the recording of the player who produced it.

From one clue to a level revision

The useful pattern is narrow: find a level whose attempt count is out of line with its neighbours, then watch two recordings of that level and write down the first thing both players did wrong. If it is the same thing, you have a redesign target instead of a difficulty slider.

The distinction matters because the two problems have different fixes. In a wave-based tower defense like Thornguard: Tower Defense, a level that fails players who never had time to see the wave composition is an information problem — more preview time, clearer telegraphing — while a level that fails players who placed the right towers is a numbers problem. Lowering enemy health fixes the second and hides the first.

In card games the same split shows up as readability versus rules depth. A hand that takes too long to parse looks exactly like a hard puzzle in the logs: long attempts, repeated retries, players quitting mid-turn. We ran into that boundary while working on card presentation and wrote it up in Card UI Readability vs Aesthetics, and it applies directly to how a level in a casual card game such as Kart Üçlüsü reads in the first three seconds.

An hour a week is the whole routine

1. Export the attempt CSV from your own build and from any tester who will send it.
2. Run the awk line. Note every level whose attempts are more than double its neighbours'.
3. Watch only the recorded segments for those levels — not the full sessions.
4. Write one sentence per suspicious level: what the player tried, and what they seemed to believe about the rules.
5. Read new store reviews and tag them by mechanic, not by sentiment.

After four weeks you have a table of spikes over time, which is the curve you were missing.

What this method cannot do

It gives you no statistical confidence, no cohorts, and no data at all about players who quit before the level you suspect. Your tester pool is small and probably friendlier than your audience. Store reviews come almost entirely from the extremes. So use these signals to generate hypotheses and to prioritise which level to open in the editor next — not to prove that a change worked. For that, the honest test is still the next batch of playtests, with the same notes template and the same questions.