I Tested REA With and Without It. What AI Reverse Engineering Actually Changes.

Varun Pratap Bhardwaj · AI Reliability Engineering · 20 October 2026

Watch the full experiment: Software Is Dead? I Tested the Viral REA Repo (With and Without It).

The rebuilt program passed every selected test with REA. Then the same model passed every selected test without it.

That result changed the question. Instead of asking whether a viral repository could make software disappear, I needed to ask what the repository contributed, what the model supplied, and who checked that the rebuilt program behaved correctly.

In the controlled painter comparison, both routes matched the original output on 20 selected seeds, byte for byte. The CLI-reported list-price cost estimates were $1.09 with REA and $1.25 without it. The difference was useful. It was also much smaller than the claim that one repository had made commercial software obsolete.

This is the written investigation behind the companion film. It develops the controls, the failure mechanisms and the limits of the results. The tests described here are case studies. They do not certify arbitrary applications, rank model providers, or establish a general cost saving.

The result that changed the question

We built an owned test program that accepts a number and produces a picture. Because we controlled the original, we could produce reference outputs without guessing. We made the task deliberately unfriendly to a model that reads only the decompiler's readable summary: important calculations were obscure or missing from that summary, but could be recovered by inspecting lower-level instructions and checking behaviour.

Before the experiment, the prediction was that inexpensive models would struggle. The actual first result challenged that prediction. Claude Haiku, using REA plus Ghidra, reconstructed the program and matched the reference output for all 20 test vectors, byte for byte. The recorded run took approximately 909 seconds, with a CLI-reported list-price estimate of $1.09. These are recorded results for this target, not an industry-wide benchmark.

To isolate the tool from the model, we repeated the target with the same model and ordinary command-line tools instead of REA. It matched 20/20 again, at approximately 979 seconds and $1.25. The recorded difference was about $0.16, or 12.7% of the no-REA list-price estimate, and 70 seconds of elapsed time. These are observed differences between the two runs, not a statistical estimate of the tool's effect. That pair of runs recorded lower estimated model cost and elapsed time with REA. They do not establish that REA determined whether the reconstruction was correct.

The hard painter comparison used 20 selected seed values. The score belongs to this one target and its scorer. It should not be combined with the simpler native and Electron tasks from the wider experiment programme.

This distinction matters for scientific honesty. A test against programs the researchers own offers a clean answer key, but it does not represent every application architecture. Timing can vary with machine load and model behaviour. A test with one target and limited repetitions can support a narrow conclusion: on this target, this model reached the same result both ways. It cannot establish a universal percentage improvement across every model, binary or organization.

What stayed fixed, what changed

The comparison held the owned painter target, the same model (Claude Haiku 5.5), and scoring method fixed. The REA arm could use the REA/Ghidra route. The control worked through ordinary inspection tools, including otool, lldb and Python. Both agents still had access to tools; “without REA” does not mean “without inspecting the executable.” It tests the necessity of this particular intermediary for the task.

The original executable supplied the reference behaviour. A seed was sent to the original and the reconstruction, and their output bytes were compared. A plausible description of the algorithm could not pass by itself. A picture that looked right could still fail the comparison.

The saved score files contain the selected inputs, including ordinary small integers, larger values and unsigned boundary cases. That selection is stronger than showing one attractive output, but it is still finite. Untested seeds, alternative builds and other platform behaviours remain outside the result. Byte equality on selected outputs is not proof that the source, internal algorithm or all possible behaviour was recovered.

The local timing files record 909 seconds for Haiku with REA and 979 seconds for the control. Their unrounded list-price model-cost records are $1.08979358 and $1.24886181. The CLI labels their cost basis as list price. They are usage-derived estimates, not proof of the cash amount billed to an account. They do not include the value of my preparation time, equipment, subscriptions, maintenance or every prerequisite. Calling this “the cost of replacing software” would be wrong.

There is one recorded arm for each condition in this comparison. We did not run enough independent repetitions to estimate variation, randomise the order, or calculate a reliable average speed improvement. Model behaviour, caching and local load can affect a run. The justified conclusion is narrow: this model could reconstruct this target through either route, with these observed resource totals.

A stronger follow-up would hide several newly generated targets from the agents, precommit the scorer, run multiple trials in random order and reserve inputs the investigator never shows during reconstruction. It would also separate hands-on setup time from model runtime. Those controls would let us ask how often a layer helps, rather than whether it helped once.

REA and PhotoCraft answer different questions

Two stories surged in the same week. REA attracted attention for connecting AI agents to reverse-engineering tools. Separately, the ArtCraft family included projects such as PhotoCraft, advertised as clean-room reimplementations of familiar creative applications, written in Rust with help from coding agents. Related applications amplified that conversation.

The social conclusion almost wrote itself: REA copied Adobe; therefore any closed product can now be cloned in an afternoon. The conclusion skips the method. PhotoCraft describes a clean-room approach based on public specifications, file formats and observable behaviour. That is a different task from taking a compiled native executable through Ghidra. ArtCraft's early alternatives are interesting engineering work; they are not proof that commercial products with mature feature sets, performance, accessibility, compatibility, distribution and support have already been replaced.

The PhotoCraft README describes its own work as a clean-room implementation based on public specifications and observed behaviour. That is the project's account of its method, not an independent certification of the development process or a legal conclusion. The repository does not establish that REA built PhotoCraft.

Evidence standard: a GitHub star count measures attention. A screenshot measures appearance. A passing test measures behaviour for that test. A business must satisfy many tests that never appear in the screenshot.

Follow the evidence through the tools

In the repository examined for the film, REA acts as a bridge between a coding agent and several kinds of inspection equipment. In technical terms, it exposes a tool interface through MCP, the Model Context Protocol. In plain English: instead of manually collecting output from multiple utilities and pasting it into a chat, an AI agent can request the relevant evidence through one connected workflow.

The workflow is roughly eight steps. You ask a question about an application. An agent decides inspection is needed and invokes REA. REA identifies the target format. It chooses a compatible engine: for a native binary, perhaps Ghidra, Hopper or IDA; for an Android application, tools such as JADX or apktool; for a JavaScript/Electron package, static readers may be enough. An adapter communicates with that engine. Evidence returns as functions, byte sequences, strings, structure, references and uncertainty notes. REA delivers those findings to the AI. Then the AI model interprets them, writes new code, tests it and iterates.

That last step is not a minor detail. It is where much of the design work lives. REA did not suddenly invent decompilers, and its documentation does not promise a magical one-click recovery of an original source tree. Its contribution is workflow coordination, useful readers, and—especially interesting to reliability engineers—an explicit distinction between what was observed, derived, inferred or remains unknown.

The evidence labels deserve attention. An artifact can record that Ghidra reported a function. That is an observation of Ghidra's output, not proof that Ghidra inferred the original source correctly. Calling those two forms of confidence by different names is valuable because AI can otherwise convert uncertain intermediate results into confident final prose.

The tool's own playbook provides a good reminder: if you already have an ordinary source repository, analyze the source rather than reaching reflexively for a reverse-engineering pipeline. Use the simplest evidence path that can answer the question.

Why readable pseudocode can still be wrong

To understand why reverse engineering is hard, open Compiler Explorer. Write a tiny C function and observe the machine instructions emitted by an optimizing compiler. The transformation is not a simple translation into another human-readable language. Optimization can inline calls, restructure loops, fold constants, eliminate unused work, reorder operations and replace an expensive operation with a faster sequence.

In our own example, a straightforward division by one hundred became a multiply by a large constant followed by a shift. The result was designed to behave the same on the relevant inputs, but the implementation no longer looked like the source statement. A compiled release also typically loses comments and many meaningful variable names. A decompiler must reconstruct plausible structure from lower-level operations. It cannot restore information that was never preserved.

A decompiler's output, often called pseudocode, is therefore a working hypothesis about a program rather than the source code itself. Ghidra, Hopper and IDA can expose functions, strings, control flow and readable approximations. Their readings may differ, particularly when compilers have optimized aggressively. A careful reverse engineer checks the raw instructions, exercises the original program and compares its outputs with the reconstruction.

This is why the answer key becomes the most important phrase in this entire story. Without a reference behaviour, a convincing-looking rebuild can be wrong. Without carefully selected edge cases, an implementation can pass a handful of examples and still fail in production.

The painter made the missing information visible

In the owned painter target, the hardness notes record switch cases whose bodies appeared empty in the decompiler output. The arithmetic still existed in the machine instructions at the jump-table destinations. Another pseudocode call appeared to have no arguments, while the caller stored the actual values and weights in the outgoing stack area.

A reconstruction that faithfully copies that summary can reproduce its omissions. The missing step is to treat suspiciously empty branches and unexplained function calls as questions. Inspect the instructions, recover the values being passed, then ask whether the revised implementation reproduces the original output.

Floating-point arithmetic supplied another boundary. Changing operation order, using double precision where the original used single precision, or rounding where the original truncated can alter output pixels. A visual comparison might miss the reason; a byte comparison exposes the disagreement. The test is useful because it can reject an implementation that a confident explanation makes sound correct.

OBS tests a component, not a whole application

The broader experiments put the same kind of reconstruction pressure on models from multiple AI companies. Gemini Flash and Muse Wonder achieved exact outcomes in recorded arms; one inexpensive OpenAI model did not complete a usable build. The Muse arm was interrupted and resumed and must not be treated as a directly comparable stopwatch race. The film uses this contrast to show that tool access alone does not ensure disciplined investigation.

Then we moved from a small owned painter to real software: the noise-gate component of OBS Studio, which adjusts microphone output when the signal falls below configured thresholds. OBS is open source, so we could still create reference checks, while inspecting a compiled plugin to see whether the models could reconstruct its behaviour without simply reading source.

This target exposes a trap. A direct implementation of Ghidra's readable pseudocode achieved only 2/20 on our grader because relevant detector operations were absent from that readable summary. The right response was not to insist that pseudocode must be correct. It was to return to the actual instructions, locate the missing behaviour and build a runnable checker.

The no-REA call check used string evidence from the saved binary conversation database rather than a parsed call trace; REA remained available. This supports a bounded no-call finding, not a claim that the tool was technically inaccessible.

The standout test was the stronger model's decision to extract the original routine's machine code, execute it beside the rewritten function and compare a very large audio fixture. Its report states that 73,296,339 samples matched bit for bit in that comparison. That larger count is the run's reported additional check; it is separate from the common 20-vector grader and is not a count of independent real-world deployments. That is a powerful demonstration of a useful workflow: build the test oracle as part of the investigation. It does not prove every input or configuration of OBS is covered. The public source also creates a possible recall confound: models may know parts of a famous codebase from training. We explicitly preserve that limitation.

It is tempting to describe this as cheap versus expensive AI, but the tested behaviour makes a more precise point. One model found relevant functions and constants, yet failed to build its output; another checked assumptions and executable behaviour. A high test score is not a substitute for reviewing the method, but the method explains a lot more than the mere availability of a connector.

Compare the intermediary with a simpler route

REA is useful in part because it packages work that existed before it. GhidraMCP is another project connecting AI to Ghidra. IDA Pro MCP takes a related approach for IDA. Decompiler Explorer lets you compare output from different decompilers. DeepWiki is relevant when the source is already available and you need architectural orientation instead of binary analysis.

That history is not an argument against REA. A good interface, tool routing, reproducible evidence and clear documentation can make difficult capabilities accessible to many more developers. But it changes the claim being tested. The breakthrough is less “reverse engineering appeared yesterday” and more “AI agents can now drive old engineering instruments through approachable workflows.”

The small benchmark published by leanrea illustrates another reason to separate convenience from necessity. On its own four-task, three-repetition comparison, leanrea reported 74/75 at $2.17, REA 75/75 at $3.36, and a plain-tool arm 67/75. The authors openly list conditions that limit comparison: one model, one machine, a particular tool setup, and no native disassembler installation for some tasks. This is their benchmark, not ours; it suggests there may be cases where a short skill and ordinary commands outperform a larger wrapper.

REA's own showcased work should be read at the scope of the demonstrated component. A function reconstruction or an application trace can be useful evidence without establishing that a complete product has been replaced.

A small adapter can be the right ambition

Not every use of AI must begin with an intimidating binary. In another demonstration, I used an AI coding agent to read my own video-editing project files, extract a simple timeline and export an interchange format that DaVinci Resolve could import. The saved demonstration shows a simple one-clip timeline imported into Resolve. The accompanying narration reports a second two-clip case; I treat that as the demonstration account, rather than a separately rechecked result here. Text overlays and stickers were not validated.

A small adapter around data you already possess can solve a useful problem. You often do not need REA, and you certainly do not need to recreate a whole application. You need to understand your file format, specify the desired output and check a real round trip.

If you try a similar project, work on copies, preserve the original files, avoid credentials in prompts, and verify that the target editor opened the exported result correctly. Do not claim a general migration tool exists because two test cases passed. State the successful case and the untested boundary together.

The payment run: count effects, not requests

Personal workflows and payment systems have different consequences. We built a mock payment API to demonstrate why operating a service requires knowledge that a happy-path prompt may omit. The beginner request was simple: loop over orders and POST each charge. The generated pattern omitted a timeout and a reliable idempotency strategy. When a slow request led to a hang and the script was restarted, duplicate charges appeared in the mock records.

What the saved charge events establish

The intended batch contained 1,000 order IDs. Counting the saved beginner charge-event log gives 1,182 events across 400 distinct order IDs. Every one of those 400 IDs appears more than once; the maximum is three charge events for one ID.

That means the beginner run did not successfully process all 1,000 intended orders. There are 782 events after the first event for each distinct ID, calculated as 1,182 minus 400. The other 600 intended IDs have no charge event in this saved log. These are counts within a mock service, not statements about real customers being charged or real money being lost.

The experienced run contains 1,000 events across 1,000 distinct order IDs, with no repeated IDs in the saved log. It demonstrates the intended behaviour under the failures exercised in this local test. It does not prove the design handles every payment failure.

The ten-order demo and one-lakh-order production scenario in the narration are illustrative. The measured batch underlying this table is the 1,000-order mock test. A viewer should not infer that a 100,000-order production experiment took place.

An idempotency key is a stable identifier that lets a payment service recognize that a retried request represents the same intended operation. It only protects against duplicate charges if the payment provider or server actually enforces the key correctly; merely setting a request header cannot guarantee safety. Real payment integrations also need reconciliation, persistent state, retry budgets, observability, correct exception handling and provider-specific guarantees.

The right lesson is not “a clever prompt makes payments safe.” It is that someone with operational experience knew to ask about timeouts, retries, duplicate effects, charge reconciliation and proof before relying on the integration. A novice can learn these techniques with AI, but they still need to learn them. That is AI Reliability Engineering: using capable models while retaining explicit evidence, failure assumptions and acceptance tests.

The server has to keep the promise

If a request times out, the client may not know whether the server performed the charge before the connection failed. Retrying can be necessary and still be dangerous. The client needs a way to refer to the same intended operation, and the server needs to recognise that identity across repeated requests.

A header alone is insufficient. The receiving service must bind that key to the operation and preserve the relevant result with the effect it protects. Restarting the client must not silently create new operation identities. Reusing an identifier with a different amount needs a defined outcome. The integration must also define what happens when a response is lost after the server has committed its work.

Those are design questions to resolve with the actual provider's contract. After the run, reconcile intended orders against distinct completed effects, not only response counts. The beginner log makes the distinction concrete: a large event total concealed both repetition and an unfinished batch.

The experienced prompt did more than add words. It supplied failure assumptions and a safety requirement. An inexperienced developer can learn to ask those questions with AI. The important boundary is that the acceptance conditions must exist and be tested before the result is trusted.

What the interface leaves out

“Build your own software” can describe four entirely different ambitions. First, create a tool for yourself, using files you own. Second, extend a product you use or automate a tedious workflow. Third, polish that software for other people. Fourth, operate a business that serves customers and cannot afford to lose their data or money. A competent demo from step one says little about step four.

A company sells much more than source code. It sells onboarding, support, account recovery, migrations, reliability under load, security review, compliance processes, documentation, ecosystem integration and somebody being accountable when a service fails on a weekend. These are unglamorous parts of the backlog because they do not make good six-second clips.

The beehiiv backlog example in the film makes a useful business point: a familiar interface leaves delivery, abuse controls, billing, support and maintenance largely out of view. The argument here does not depend on a precise subscription price or a counted backlog. It depends on naming which obligations a replacement actually assumes.

Security research adds a real warning. In an October 2025 study, Escape reported examining more than 5,600 publicly accessible vibe-coded applications and finding over 2,000 high-impact vulnerabilities and 400 exposed secrets. Those are aggregate findings, not proof that exactly that many distinct apps failed, and not proof that every AI-built app is insecure. They show why a deployed product needs systematic review rather than confidence borrowed from a screenshot.

For a personal utility, saving a subscription or unlocking your own data may absolutely be worthwhile. For an email platform that routes millions of messages, or payments that represent other people's money, the subscription can be the cheapest part of the total cost of ownership.

A practical way to decide what to learn and build

The skill set is not obsolete; its emphasis changes. Start with execution fundamentals: what a compiler does, how memory and binaries work, and what local versus server-side code means. Then learn verification: write test oracles, design adversarial fixtures, compare outputs and record confidence. Learn to read unfamiliar code, because future projects may contain large quantities of generated code that still need maintainers. Build domain judgment: an answer can satisfy a prompt and still fail a user's real workflow. Finally, learn operational failure modes: timeouts, retries, idempotency, rate limits, monitoring, backups and recovery.

Here is a compact exercise plan for the next month:

1. Week one: observe compilation. Use Compiler Explorer to alter a small C program and compare optimized and non-optimized output. Write down what disappears.
2. Week two: inspect an owned binary. Compile a tiny program, remove source from the working directory, compare Ghidra's guess with what you know was written and confirm behaviour with tests.
3. Week three: run the same task two ways. One arm with an orchestration layer, one with direct tools. Hold the target and scorer constant. Record wall time, cost, errors and retries.
4. Week four: add a failure case. Introduce a timeout, duplicate message, missing field or partial network outage into a local mock service. Test how the generated code behaves before calling it complete.

For creators and product builders, there is also a business lesson. Choose the smallest problem you can define and verify. Do not start by promising a complete replacement for a mature product. Explain what your tool demonstrably does, what it does not do, how its outputs are validated and who will maintain it. That is a better path to trust and sustained adoption.

The next test matters more than the slogan

The painter comparison established that the REA route was useful and that the control route could also succeed. OBS showed why inspecting a component and testing its output is more informative than copying a decompiler summary. The payment run showed why counting completed effects matters after a script appears to work.

The strongest objection to a sweeping conclusion is also the right next experiment: these are selected cases. A broader set of hidden targets, independent repetitions and withheld test inputs could change the measured advantage of the tool. That would be a useful result, too.

Before you replace software, define the smallest behaviour you need, identify a reference you can test against, exercise failure and recovery, and decide who owns maintenance. Then compare the working replacement with what you already have. That is a decision you can defend with evidence.

REA lit the spark

Our saved 17 October tracker snapshot recorded 73 AI-assisted rebuild projects; 3 entries explicitly named REA. That is a dated catalogue observation, not the current directory total or proof the other authors never used it. The wider ownership movement is the point. Ledger Reborn's author reports matching three official sample reports to the cent; picoprint's author reports matching captured handshake traffic byte for byte. Those are self-reported, bounded checks, not independent product certifications.

Building is cheap; proving and running is still the job.

We build to own too: SuperLocalMemory, local-first agent memory (demo), and ModStage, which runs Codex, Grok, Muse and Gemini inside Claude Code (film).

---

Watch the full REA experiment on YouTube.

Varun Pratap Bhardwaj · varunpratap.com · Qualixar · @varunPbhardwaj