DSA vs AI: A Sealed Three-Round Test, Every Result, and 43 Puzzles Built to Be Broken

Varun Pratap Bhardwaj · AI Reliability Engineering · November 2026

Does a developer still need data structures and algorithms when an AI writes the code? So I tested it. There were three rounds, and the scoring rule was sealed before the scored runs. In round one an AI solved hard puzzles. In round two it hunted a planted bug. In round three, teams of AI agents built elevator software while the requirements changed under them. AI won all three. Then I searched the winning code for dynamic programming, heaps and graphs, and found none.

Below is how each round was built, how the referee's test days were sealed so nobody could tune them, every team's result on every version, and the code. The last third covers the repository: every test from the match, plus a library of 43 classic puzzles, each with an answer key, a real solution and two wrong answers to hunt. It all runs on Python 3, under the MIT licence.

The film is at youtu.be/fFENRZjIdto. The code, the results and the earlier runs that don't count are at github.com/qualixar/dsa-vs-ai.

Scoring worked like this. If the AI could do the job on its own, AI got the point. DSA got it only if the AI team couldn't do it alone and an architect's way of thinking, the kind DSA builds, made the difference. That rule was sealed before the scored runs, but the setup had already changed by then; I cover how further down.

Round one: two hard puzzles, a hundred answers

The exam used two hard puzzles: Trapping Rain Water and Elevator Requests II. I collected fifty answers to each and tested every answer against thousands of cases. All one hundred passed.

I kept the setup plain. One AI coding agent, no web access, a fresh session for each attempt, and the same prompt every time: "Solve this LeetCode problem in Python 3. Use the given class and method signature. Reply with only the code."

Each answer then went through four gates, in order. The puzzle's own examples came first. Then edge cases, three thousand small random cases against a slow reference that is obviously correct, and six hundred medium ones. Then the puzzle's real limits, with two seconds per case. Last came rules no correct answer can break: water is never negative, and the elevator never beats each person's own distance.

The fifty elevator answers cost $1.46 in total, an average of 2.9 cents each. The fifty rain answers averaged half a cent. Those are November 2026 prices, so they won't hold for long.

The elevator puzzle came out in August 2026. The AI's training stopped in June. That makes this particular puzzle useful: the published answer couldn't have been in that training. It doesn't settle how much related work the AI had seen.

Start with the small version. A lift moves one floor per second and starts on floor 10. People are waiting at floors 8, 13, 14, 15 and 16, with every request already known. The aim is to minimise the sum of their waiting times.

Floor 8 is closest. Visit it first, then climb, and the waits are 2, 7, 8, 9 and 10 seconds. Total: 36 seconds. Go up first and the four upstairs wait 3, 4, 5 and 6 seconds. Return for the person on 8 and that person waits 14 seconds. Total: 32.

Nearest first saves the person on 8 twelve seconds, and costs the other four people four seconds each.

Trying every possible visiting order would find the answer. I timed that approach on my machine: about five seconds for ten people, then forty-seven seconds for eleven. The estimate for twenty people is about ninety thousand years. That last figure is an extrapolation, not a run I completed. The actual puzzle allows fifteen hundred people.

An elevator passes every floor between where it is and where it's going, so it can serve people on the way. That means the served floors can form one continuous stretch, growing outward from the start, instead of a scattered set of stops.

Now the program can record the best cost for each stretch and which end the lift is at. Its next decision extends the stretch upward or downward. Dynamic programming, or DP, means saving those smaller answers and using them to build larger ones without repeating the same work.

This is the idea as code, from the reference solution in the repository. i counts the requests served below the start, j counts those served above it, and there are two tables because the lift can be at either end of the stretch.

for i in range(a + 1):
    for j in range(b + 1):
        waiting = k - i - j
        if waiting == 0:
            continue
        for cost, pos in ((at_left[i][j], left[i - 1] if i else None),
                          (at_right[i][j], right[j - 1] if j else None)):
            if cost == inf:
                continue
            if i < a:
                c = cost + (pos - left[i]) * waiting
                if c < at_left[i + 1][j]:
                    at_left[i + 1][j] = c
            if j < b:
                c = cost + (right[j] - pos) * waiting
                if c < at_right[i][j + 1]:
                    at_right[i][j + 1] = c

Look at the multiplication by waiting. Moving a distance costs that distance for every person still waiting. So the program never tracks anyone's individual arrival time, and a problem about people turns into a table you fill row by row.

For fifteen hundred people, the table has about a million rows. A laptop fills it in under a second. The AI wrote the solution in about sixteen seconds, and all fifty answers to this puzzle passed.

AI took the first point.

Andrej Karpathy offers an explanation for why this is favourable ground for AI: models become particularly strong where their work can be marked automatically. Code with a fast correctness check gives training a repeatable signal. That's his account, not something my test established.

His car-wash example describes the unevenness outside that ground: the wash is fifty metres away, and the AI recommends walking, leaving the car behind. I repeated the question fifty times on the model used for the film. Every answer said drive. That older failure no longer described what happened in these tries. The date matters: these are November 2026 results.

Round two: samples let the wrong answer through

I turned the nearest-floor idea into a complete solution. It passed all three sample cases on the puzzle page. Against three thousand tiny buildings checked by exhaustive search, it failed two hundred and thirty-five. In the worst cases it made people wait almost seventy per cent longer than necessary.

The samples on a puzzle page are there to explain the problem. Passing all three leaves most possible inputs untried.

The slow search becomes useful again when the building is small. Instead of asking it to handle the full puzzle, let it try every order on a tiny input and compare its answer with the fast solution. Because the check works a different way, it can catch a mistake that the fast solution's own logic hides.

There is also a real try-every-order check in the elevator referee. This is its central loop:

for order in itertools.permutations(dests):
    t, pos, total = 0, start, 0
    for f in order:
        t += abs(pos - f)
        total += t
        t += 1
        pos = f
    best = total if best is None else min(best, total)

It tries every order of the destinations, totals arrival times and keeps the smallest total.

This loop comes from the later simulation and scores riders already inside a lift. The extra t += 1 charges one tick for opening the doors, which the waiting-time example above didn't count. Otherwise it's the same try-every-order method.

How much do samples actually catch? I measured that too. Across the fifty real elevator answers I made two thousand single-symbol changes, forty in each answer. The samples caught 1,499. Edge cases caught 182 more. The three thousand small cases caught another 36. And 283 survived every check.

Some of those 283 change no answer at all, so a black-box test has nothing to see. The 36 matter more to me. They got through the samples and the edge cases, and only the slow check stopped them.

For the scored review test, I changed one character in a correct puzzle solution: a plus became a minus. The code should compare a candidate with the table cell at j + 1, then write to j + 1. The altered comparison reads j - 1 while the write still goes to j + 1. It compares against the wrong neighbour without crashing.

That version passed the three samples and seven hard cases. It was wrong on eight of three thousand small buildings and fourteen of six hundred medium buildings: twenty-two wrong answers altogether.

I gave a fresh AI the code, no tools, and the question of whether it was correct. It got twenty-five independent reads of the altered version and twenty-five of the correct version. Every altered version led it to the bad line. None of the correct versions drew a false alarm. Each review took about ten seconds.

That earned AI the second point. The task was narrow: a short program, a direct correctness question and one planted change. Even so, it's a reason to run the test instead of assuming a subtle bug will get past the AI.

What the research says about practice

In Chase and Simon's 1973 chess study, a beginner, an amateur and a master saw real game positions for five seconds, then reconstructed them. For the middle-game positions, they recalled about four, eight and sixteen pieces respectively. Random arrangements removed the master's large advantage. Later work found that experts retain a small edge even on random boards.

What the master remembers is arrangements: groups of pieces that make sense together because they've seen those relationships before. The 1981 work by McKeithen and colleagues found a similar pattern with programmers looking at normal and shuffled code.

That's my reason to care about the practice behind a pile of solved puzzles. These studies don't measure what any one person's practice achieved, or how well chess expertise transfers to programming.

There is more direct evidence about learning with AI. A small Anthropic study published in 2026 involved fifty-two developers, mostly junior. The group using AI averaged about fifty per cent on the subsequent quiz; the group without it averaged about sixty-seven per cent. Debugging had the biggest gap. This was one small study in one sitting, so it cannot settle the long-term effects of learning with AI.

My view: keep doing the solving and the checking. Recognising a pattern, explaining why an approach fails and finding a counterexample stay useful when typing the final answer is cheap. That's what I'd use DSA for now: judging and testing the machine's work.

Karpathy repeats a line that fits this distinction: "You can outsource your thinking, but you can't outsource your understanding." His published account of the conversation also discusses why understanding still matters when directing agents.

Round three: changing requirements and a clock

I gave each team the software job of running lifts in a thirty-floor building. A team had four AI agents: a manager, two builders and a reviewer. They wrote one Python class. The building called on_call for a hall button, on_car_request for a destination pressed inside, and step once per second. The class returned one command per lift: up, down, stay or open.

Moving a floor and opening doors were separate operations. The code had to track requests and avoid illegal movement while getting passengers to their destinations within the stated limits.

Requirements arrived in five versions:

• Version 1: run one elevator.
• Version 2: run several elevators sharing calls.
• Version 3: handle a lift breaking down mid-trip with people inside.
• Version 4: handle a fire.
• Version 5: handle a morning rush of three hundred calls in five minutes.

A team saw the next change only when its manager declared the previous version done. Earlier behaviour still had to work. The clock allowed two and a half hours of working time, and there was nobody to ask for clarification. The submitted code was whatever remained when time ran out.

A separate program, the referee, ran the acceptance checks. The teams never saw it. It delivered events, called the team's code, moved the lifts and recorded violations.

A day was an input list. This sample day has 120 people, and these are its first two events:

    ["call", 1, 27, 0],
    ["call", 33, 29, 1],

These events introduce a person travelling from floor 27 to ground at second 1, then a person travelling from floor 29 to floor 1 at second 33.

The list belongs to the referee. The controller learns about each person when the button is pressed, and a hall call initially reveals the floor and direction rather than the destination.

For each version the referee tried fifty busy days. A clean day meant no rule violations and passengers meeting that version's timing requirements. Fifty out of fifty clean days passed a version.

On every tick it checked four hard rules: the contract between building and code, the limits of the shaft, the fire rule, and no movement while a lift is out of service. On top of those sat a time limit for every passenger. I set the limits generously, because this test is about correctness, not speed. On the first version, everyone had to arrive within 270 seconds.

If anyone could have tuned the days after seeing the code, the result would mean little. So the inputs were frozen and hashed, a random salt was drawn and recorded, and the days were generated from both. Before any team ran, I measured the referee itself. It caught all five deliberately broken dispatchers, and fifteen of the seventeen planted bugs that changed what the lifts do.

There were four teams using the same AI, job and clock. Two worked alone. Each of the other two received a page of architectural notes with every change: what must remain true, a simple design, checks to run, traps and a definition of done. A separate AI wrote those notes. No person wrote or edited them.

The scoring rule was sealed first: if both teams working alone got everything right, AI would take the round. If the notes clearly made the difference, DSA took it. Anything else, nobody got the point.

Both teams working alone passed every version

Each team working alone passed all five versions. Each team with notes passed three of five. Under the agreed rule, AI took the round and the match ended AI 3, DSA 0.

One failure was simple to inspect. Version 3 required a method for a broken lift, and one team's code never got it, so all fifty days failed. The referee ran every version on what shipped. Version 5 needs no new method, so code missing version 3's method still passed it.

The other notes team failed differently. It had no fire mode, so version 4 failed on all fifty days. On version 5 it was clean on forty-nine days of fifty, and one bad day fails a version.

In these four runs, the teams that got notes did worse. Four runs can't tell you why, and they can't establish that the notes caused the difference.

The runs weren't even, and I haven't corrected for that. Working time ran from 80 to 155 minutes. One alone team finished everything in 80 minutes. The other reached the deadline before its manager signed version 5, and its code passed all five anyway. Three teams hit 21 pauses on usage limits and the fourth hit none. The clock stops during a pause.

These were also not the first attempts. Earlier versions of the setup had a money limit, a limit on rounds and a hole in the brief. Some teams stopped short under those conditions. I fixed them, and one practice team then finished all five versions before sealing the final test. None of those earlier runs count in this score. They are recorded in the repository.

Those earlier runs ended differently. Under the money cap, the first team stopped after two versions, and the rule then gave that round to DSA. So DSA did take a point, under rules I later changed. Another team stopped because two locked tests were wrong and nobody could edit them. Removing the caps helped the AI side; keep that in mind when you read the final score.

One team's code was already passing at minute twenty-three. Its manager kept checking for another two hours, and the team never saw the next change. Across the four teams, three had working code sit for an hour or more, including one that worked alone. The notes defined done but said nothing about when to stop checking.

The code used a sweep

I searched all four programs for dynamic programming, heaps and graphs, and found none. What I found was lists, sets, dictionaries, min and sorting.

That's one project, and databases and search engines have reasons to use the structures missing here. What changed between the puzzle and the building was how much the code knew in advance. The puzzle supplied the whole request list in advance. The building supplied requests as people pressed buttons. The earlier DP table could optimise a known list; it couldn't fill its rows with future passengers it hadn't been told about.

All four teams built a sweep, the family of methods known as the elevator algorithm or SCAN. Picture a lift going upward. It serves stops ahead of it and keeps that direction while work remains ahead. A new request below waits for the return journey. When the lift turns, it serves the stops in the other direction. A nearby arrival doesn't automatically pull it away from everyone already waiting farther along its route. The operating-systems account of SCAN explains the same sweep using a disk head moving between tracks.

This passing team's code contains the nearest-floor choice:

# Choose the nearest available stop when beginning or reversing a
# sweep. Continuing a sweep is handled by step(); favoring age here
# can send the car across the building past several nearby groups.
target = min((f for f, _ in targets), key=lambda f: (abs(f - here), f))

It chooses the closest target when starting or reversing a sweep, with the floor number breaking a distance tie.

The comment says where that choice applies: when a sweep starts or reverses. During a sweep, other code keeps the direction. So the nearest-floor rule that failed as a complete puzzle solution can be useful as one part of a larger policy, and you have to look at where it's used before calling it wrong.

The line counts show how much else the teams wrote. For the four programs, elevator code and test lines were respectively 259 and 1,339; 598 and 774; 277 and 1,055; 246 and 933. Every team wrote more test lines than elevator lines. Those counts describe effort, not test quality.

Every team was also handed the exact limits and rules before it started, and a referee it never saw enforced them. None of that came from the teams. Whoever set those limits decided what counted as right.

The fire (unscored)

The fire was an unscored simulation, run on each team's shipped code with the same day and alarm time. About seven minutes in, a lift had broken down. Nineteen seconds later, the alarm went off. Every lift in service had to go down to ground and open there, with no going up and no doors opening on the way.

The teams working alone reached the ground in twenty and twenty-one seconds, with no rule broken. The teams with notes never reached the fire version, so nobody had told their code what a fire is. In both buildings, the same lift climbed from floor 2 to floor 27. Their code raised an error at the alarm, and the referee logged at least fifty fire-rule violations before its log hit the cap.

The teams ran out of time before the fire version came, so their code had never been asked to handle one.

This is the whole fire rule from one passing team:

expected = [
    "stay" if not self._available[i] else ("open" if floor == 0 else "down")
    for i, floor in enumerate(floors_now)
]

It holds unavailable lifts still, opens available lifts at the ground and sends the rest down.

No person wrote a line of that code, and it handled the fire on every day it was given. I'd still keep it out of a real building: it passed fifty sampled days, and that's all it has passed.

What this doesn't show

Four runs cannot establish a general effect of architectural notes. They also cannot rank every way to organise an AI team. This was one elevator project using November 2026 models, under rules that people supplied.

The earlier runs under different limits are left out of the final score, so anyone quoting that score should mention them. The practice success was known before the final rule was sealed. You can inspect that history in the repository.

The referee has limits too. I planted seventeen behaviour-changing bugs in working code; it caught fifteen. In another team's code, it caught six of nine. Passing its checks leaves room for faults it doesn't detect, and a pass covers only the days it tried.

The fresh reader in round two tested reading only. It says nothing about an AI that runs code and writes its own checks. And the notes were written by an AI and never tested as advice, so they tell you little about what a good human architect would have added.

People read results like these as a jobs story, so here is one more limit. The Stanford Digital Economy Lab update of 12 August 2026 reports that employment for workers aged twenty-two to twenty-five in highly AI-exposed occupations stands about nineteen per cent below where it would be if it had kept pace with similarly aged workers in less-exposed occupations. It describes a gap; it doesn't establish its cause. My elevator experiment cannot turn that association into an explanation of the labour market.

The repository: what you get and how to use it

You can rerun everything above. The repository is also set up for practising what the match points at: checking an answer you didn't write. It's at github.com/qualixar/dsa-vs-ai.

The 43 puzzles

Each is a well-known LeetCode problem, reworded, with a link to the original. They cover eighteen families: arrays and hashing, two pointers, sliding windows, binary search, stacks, heaps, intervals, linked structures, trees, graphs, greedy choices, backtracking, tries, bit tricks, and dynamic programming in four forms (one-dimensional, two-dimensional, intervals and bitmasks). Graphs have the most, with six cases, among them number of islands, course schedule, network delay time and word ladder.

Every case folder has the same five files:

• README.md: the problem, the signature, the shape to notice, the idea, the complexity, how an answer gets checked, and the two bugs hidden behind a fold.
• slow.py: a brute force simple enough to believe. It is the answer key.
• fast.py: an efficient solution.
• wrong.py: gut, a tempting wrong idea, and planted, a copy of fast.py with exactly one line changed.
• cases.py: a generator of small inputs, one large input with its own check, values at the 32-bit limits, and a custom comparison where several answers can be right.

I care most about the wrong answers. In Coin Change, the gut answer takes the largest coin first. That works for most real currencies, which is why it feels right. With coins 1, 3 and 4 and an amount of 6, it uses three coins where two 3s are enough.

What the runner checks

For each case, check.py draws three thousand small random inputs, removes duplicates and asks slow.py for every answer. Then it requires three things. fast.py must agree with slow.py on every input, and finish one large input within ten seconds with an answer that a separate check accepts. gut must fail on at least one input. planted must fail on at least one input. If a wrong answer survives, the whole case fails, so nothing claimed in wrong.py goes unrun.

It is strict about types on purpose. True doesn't pass for 1, 1.0 doesn't pass for 1, and a tuple doesn't pass for a list. A crash or a call over two seconds counts as a failure. Inputs reach the ends of the 32-bit range even where LeetCode's limits are narrower, because a mask that is too narrow passes on small numbers.

python3 check.py --list
python3 check.py two-sum
python3 check.py --all

The whole library runs in a few seconds on a laptop. On my machine, the latest full run printed: "43 of 43 cases OK. 86 wrong answers each failed on at least one input."

How to use it

Hunt the bugs. Pick a case and read its README down to the fold. Run the check, which prints the smallest input that breaks each wrong answer:

python3 check.py two-sum

two-sum  [Two Sum]  2977 distinct inputs
  fast     agrees with slow on all 2977
  gut      fails 1253 of 2977
           smallest input: nums=[1, 0, 1], target=1
           got [0, 2], slow says [0, 1]
  planted  fails 132 of 2977
           smallest input: nums=[0, 1], target=0
           got [0, 0], slow says None

Work out why each one fails before you open the explanation. planted makes a good exercise on its own: open wrong.py and find the changed line.

Solve it yourself. Write your own solve function with the signature in the README, save it, and run it against the answer key:

python3 check.py coin-change --answer my_solution.py

Check your AI's answer. Give the same problem to whichever AI you use, ask for a function named solve with that signature, and save its code. Read it first, because the next line runs it on your machine. This is what a failing answer looks like, using the sample in the repository:

python3 check.py two-sum --answer docs/examples/answer_with_a_bug.py

two-sum  [Two Sum]  2977 distinct inputs
  answer   fails 541 of 2977
           smallest input: nums=[0, 1], target=0
           got [0, 0], slow says None
  large input: 0.0 s (limit 10 s), answer checked
  PROBLEM: answer disagrees with slow on 541 of 2977 inputs
  FAILED

Then work out by hand why that input breaks it, before asking anyone. A pass means no disagreement on the inputs tried. It doesn't mean the answer is correct, and each case's README says what its check leaves out.

Build a check for your own problem. Most code you'll check isn't in any library. The guide in docs/learn.md walks through it. Write down in one sentence what a right answer is. Write the dumbest program that meets that sentence: every order, every pair, every subset. Check it by hand on three tiny inputs. Generate thousands of small inputs, including empty, single, duplicate, negative and tied ones. Compare the code you want to check against it, and print the smallest disagreement.

docs/examples/own_check.py does all of that in one file, for a problem outside the library: the largest sum of a contiguous stretch of a list.

def slow(nums):
    return max(sum(nums[i:j + 1]) for i in range(len(nums)) for j in range(i, len(nums)))


def answer(nums):
    best = current = 0
    for x in nums:
        current = max(0, current + x)
        best = max(best, current)
    return best

The answer looks right, and it is right whenever some number is positive. Run it:

318 of 3000 inputs disagree
smallest failing input: [-5]
answer says 0, slow says -5

One number breaks it. I keep the smallest failing input because the bug is usually obvious within seconds. When the check finds nothing, its own message says what that's worth: "no disagreement on 3000 inputs, which is not the same as correct".

Then test the check itself. Copy a solution you trust, change one line (flip a comparison, move a + 1, swap < and <=), and run the check again. It should fail with a small input you can read. If it passes, either the change alters no answer or your check has a hole, and you need to know which.

Rerun the match, or try a new AI on it. Round one regrades any of the hundred answers with grade.py. Round two's free part reruns with checks.py and reproduces the planted-change numbers exactly. For round three, check_referee.py confirms the referee files match the sealed certificate, and judge_program.py judges any shipped program on the fifty certified days per version, printing the first failing day and rule. To test another AI, run your own teams on the brief and change requests in rounds/round3/brief-v3/ under the same clock, then judge what they ship. The rounds need macOS and a virtual environment with two packages, as rounds/README.md explains.

Limits of the library

No AI was run on the library, and it measures nothing about any AI. "Agrees" means no disagreement on the inputs tried; it isn't a proof. Complexity statements are reasoned rather than measured. And a check only sees mistakes its generator can produce. In cheapest-flights-within-k-stops, plain random graphs let the gut answer survive 2,998 inputs. The runner flagged it, and that generator now builds the input that exposes it. You can add your own case with the same five files, and python3 -m unittest discover -s tests tells you whether it follows the rules.

A practice plan

I'd train for three parts of the job: deciding what right means, building more than one check, and saying when it's enough. When something breaks, a person still answers for it and has to say how they knew. I call this work AI Reliability Engineering.

The AI goes as deep as your questions go. If you can't say what a right answer is, you can't ask for one, and you can't tell whether you got one.

So I'd stop collecting answers and practise differently. Take one case a day and solve it by hand, without AI, until its shape makes sense. Run the check and hunt both bugs. Then give the same problem to an AI and try to break its answer with the slow check. Keep the small failing case when you find one.

After a few weeks of that, choose something real you understand, such as a clinic's appointments or a family shop, and build one small thing with agents. Write down what right means before building it: inputs, outputs, the rules that must never break, and when it's done. In an interview, show the bug your check caught as well as the program you made. If a company still interviews with puzzles, practise them enough to get through.

If you run a team that ships agent-written code

These come from one project, so take them as my view. Four questions would have told me the most here, and I'd ask them of any team.

Who wrote the acceptance check, and did the builders see it? Here the referee was hidden, so nobody could write code to fit it. Each team also wrote more test lines than elevator lines. Even so, one manager kept a passing version open for two hours.

How do you know the check catches anything? Plant bugs and count. My referee missed two of seventeen in reference code and three of nine in an earlier team's code. I know those numbers only because I went looking for them. The earlier referee caught five of twenty.

What is the stop rule? A team sat on passing code from minute twenty-three to the deadline and never saw the rest of the job. The code was fine; nobody had a rule for calling it done.

What exactly does a pass cover? Here it was fifty sampled days, which is less than "it works". Whoever signs off should be able to say exactly what was tested.

Anyone can ask three more of an AI's answer or a person's, without reading code. How do you know? What would break it? Did someone check it who didn't build it?

Every test, run and puzzle is at github.com/qualixar/dsa-vs-ai. The film walks through the three rounds and the fire on screen: youtu.be/fFENRZjIdto.