Retro reversing logo

Decompilers - A Deep Dive

Edit on Github | Updated: 25th October 2026

From Machine Code to Pseudo-C

A disassembler names the instructions in a ROM or executable. A decompiler goes further, grouping those instructions into variables, calls and control structures that resemble source code.

That C-like view can help trace a game’s rendering path or compare a function with surviving SDK headers. It is a reconstruction, though: the compiler may have erased names, merged statements or moved code out of its original function. The assembly remains the evidence when the pseudo-code looks suspicious.

This hub follows the steps between machine instructions and readable output, then connects them to the particular problems found in retro games and matching decompilation projects.

For the disassembly stage that decompilers depend on, see:


Glossary of Key Terms

If you are new to decompiler terminology, this quick glossary should help:

  • AST - Abstract Syntax Tree; a tree-shaped representation of high-level code structure used when emitting readable output.
  • CFG - Control Flow Graph; a graph of basic blocks connected by branches, calls, and returns.
  • IR - Intermediate Representation; an architecture-neutral instruction form used between lifting and code generation.
  • Lifting - Converting native machine instructions into a higher-level IR instead of leaving them as raw assembly mnemonics.
  • P-code - Ghidra’s name for its stack-based intermediate representation used by the decompiler engine 1.
  • SSA - Static Single Assignment form; an IR where each virtual variable is assigned exactly once, which simplifies data-flow reasoning 2.
  • Pseudo-code - Human-readable C-like output produced by a decompiler; useful for analysis but not guaranteed to recompile.

What is a Decompiler?

A decompiler is a program that takes compiled machine code and produces a higher-level representation such as pseudo-C, C#, or Java. It does not run the program. It analyzes the binary statically, similar to an advanced disassembler, then applies additional stages that recover structure and data flow.

Decompilers vs Disassemblers

Disassemblers and decompilers share the same starting point but stop at different levels of abstraction:

Disassembler output (assembly):

mov eax, [ebp-0x4]
add eax, 0x1
mov [ebp-0x4], eax
cmp eax, 0xA
jl 0x401020

Decompiler output (pseudo-C):

local_4 = local_4 + 1;
if (local_4 < 10) {
    goto LAB_401020;
}

The decompiler inferred a local variable, an addition, and a conditional branch. It may still use awkward names like local_4 or LAB_401020 when type and symbol information is missing.

Decompilers vs Decompilation Projects

The word “decompilation” is used in two related ways:

  • Interactive decompiler tools - Ghidra, Hex-Rays, Binary Ninja, and similar tools that produce on-demand pseudo-code for one function at a time.
  • Matching decompilation projects - Community efforts to rewrite an entire game in C or C++ so it compiles back to the original ROM or executable, such as the Super Mario 64 project.

Tool decompilers help you understand code quickly. Matching projects also need compiler research and repeated compile-and-compare work. The pipeline concepts in this post apply to both, but only the tool path is automatic.


Why Decompilation is Hard

Decompilation is the inverse of compilation, and compilation deliberately destroys information 3. A compiler removes variable names, merges statements, inlines functions, reorders instructions, and picks register and stack layouts that never appeared in the source file. A decompiler must guess all of that back from whatever patterns remain.

Lost Source-Level Intent

The following information is usually gone from a release binary:

  • Local and global variable names - Replaced by stack slots and addresses.
  • Type information - struct, enum, and pointer types collapse into raw memory accesses.
  • Comments and file boundaries - Stripped entirely.
  • High-level control structures - Compiled into jumps and labels.
  • Macro expansions and templates - Expanded and specialized before machine code exists.

Decompilers recover approximations using data-flow analysis, heuristics, and user annotations.

Optimizing Compilers

Modern and even many retro compilers apply optimizations that make recovery harder:

  • Instruction scheduling - Independent operations are reordered for speed.
  • Strength reduction - Multiplies become shifts or lea-style address math.
  • Inlining - Function boundaries disappear inside callers.
  • Tail-call optimization - A call plus return becomes a single jump.
  • Dead store elimination - Temporary values vanish before memory is touched.

A decompiler may merge what was two source statements or split what was one expression.

Custom or Hand-Tuned Code

Game code often breaks the patterns decompilers expect:

  • Hand-written assembly for inner loops, GPU command submission, or audio mixing.
  • Custom calling conventions in early console SDKs.
  • Self-modifying code on very old platforms.
  • Overlay or bank-switched memory where the same address means different code at different times.

In those cases, pseudo-code may be wrong until a human marks memory regions, defines structs, or fixes control flow manually.


The Decompilation Pipeline

Most production decompilers follow the same broad pipeline, even when internal names differ. Ghidra uses P-code lifting into SSA before emitting C 1. RetDec lifts to an LLVM-style IR 4. Hex-Rays builds an internal microcode layer before its AST-based C output 5. The stages below describe the common logic.


Step 1 - Load and Disassemble

Every decompiler starts with the same binary-loading and disassembly work described in the disassemblers post. It must:

  • Parse executable headers and segment layout.
  • Identify code vs data regions.
  • Decode instructions for the target ISA.
  • Recover function entry points from symbols, exports, or heuristics.

If disassembly is wrong at this stage, every later stage inherits the mistake. Interactive tools let you mark misclassified bytes as code or data before decompiling.


Step 2 - Build a Control Flow Graph

The decompiler groups instructions into basic blocks: straight-line sequences that only enter at the top and exit at the bottom through one branch or fall-through 2. Blocks become nodes; jumps, calls, and returns become edges in a CFG.

CFG recovery matters because assembly rarely labels if, while, or switch. It exposes:

  • Conditional branches - Candidate if and loop headers.
  • Back edges - Often indicate while, do/while, or for loops.
  • Indirect jumps - Common in jump tables generated for switch statements.

A simplified CFG for a function with a loop might look like this:

flowchart TD
  A[entry] --> B[init i = 0]
  B --> C{i < limit?}
  C -->|yes| D[body]
  D --> E[i++]
  E --> C
  C -->|no| F[return]

Step 3 - Lift to Intermediate Representation

Lifting converts architecture-specific instructions into an IR with explicit operations such as load, store, add, compare, and branch 3. This step decouples “what the CPU did” from “what the source probably meant.”

Example x86 assembly:

mov eax, [ebp-0x8]
add eax, ebx
mov [ebp-0x8], eax

Lifted IR (conceptual):

t0 = load stack[-8]
t1 = add t0, ebx
store stack[-8], t1

Once in IR, the same structure-recovery and type-inference passes can target MIPS, PowerPC, SH-4, or x86 without duplicating all logic per architecture.


Step 4 - Convert to SSA Form

Most modern decompilers convert the IR into SSA form 2. In SSA, each assigned value gets a unique name, and merge points use special phi nodes to select the correct incoming value.

Why SSA helps:

  • Data-flow questions become local - “Where does this variable get its value?” is explicit in the graph.
  • Dead code is easier to remove - Unused SSA names drop out cleanly.
  • Expression trees rebuild cleanly - Chains of add/shift/load operations map back to one source expression.

Without SSA, recovering expressions from register shuffling and reused stack slots is much harder.


Step 5 - Data-Flow and Type Analysis

After SSA construction, the decompiler analyzes how values move and combine:

  • Reaching definitions - Which assignments can reach a use site?
  • Use-def chains - Which consumers depend on a given definition?
  • Constant propagation - Is a value always the same at a point?
  • Pointer analysis - Does an add come from struct field access or array indexing?

From these facts the tool guesses types:

  • Repeated + 4 stride often means a 32-bit array or pointer increment.
  • Loads at fixed offsets from the same base often mean a struct field.
  • Compare against small integers may recover enum or boolean tests.
  • Calls to known library functions can propagate parameter types backward.

Type recovery is heuristic. Two pointers and one integer can look identical until you name a struct and apply it manually in Ghidra or IDA.


Step 6 - Structure Recovery

Structure recovery turns flat CFG graphs back into readable control flow 3. This is where goto-heavy IR becomes if, else, while, and switch.

Common transformations:

  • If-else discovery - Match two-way branches with shared merge points.
  • Loop identification - Detect back edges and induction variables such as i++.
  • Switch recovery - Recognize jump tables and build case labels from entry counts.
  • Early return flattening - Fold multiple exit blocks into return statements.
  • Goto removal - Replace unnecessary jumps with structured blocks when safe.

A toy example shows why this stage exists. Given goto-style IR:

if (x < 0) goto L_fail;
result = x * 2;
goto L_done;
L_fail:
result = 0;
L_done:
return result;

Structure recovery emits:

if (x < 0) {
    result = 0;
} else {
    result = x * 2;
}
return result;

Real binaries produce far messier graphs, especially with compiler-generated break/continue patterns and optimized min/max sequences.


Step 7 - Emit High-Level Pseudo-Code

The final stage walks the recovered structure and types to build an AST, then pretty-prints C-like pseudo-code 5. Names come from:

  • Debug symbols when present.
  • Import/export tables for API functions.
  • Demangled C++ names when RTTI survives.
  • Heuristic defaults such as local_8, param_1, and FUN_80001234.

Interactive decompilers refresh this view as you rename variables, define structs, and fix signatures. Your annotations flow back into later decompilation of the same function.


Managed Code vs Native Machine Code

Not all binaries decompile with the same difficulty. Managed runtimes preserve much more structure than a stripped native executable.

.NET, Java, and Similar IL Targets

Assemblies for .NET or JVM bytecode retain metadata: class names, method signatures, field types, and often debug symbols. Tools such as ILSpy or JADX can recover readable C# or Java because the type system survived compilation 6. PlayStation Mobile titles are a practical retro example: C# plus Mono often decompile almost completely.

Native Console and PC Executables

Cartridge ROMs, ELF files, and PE executables for consoles such as Nintendo 64, PlayStation, and Game Boy Advance usually ship without rich metadata. Native decompilers must infer everything from machine instructions and whatever symbols remain. That is the pipeline described above.

For a platform-specific walkthrough using Ghidra on C++, see:


Special Cases in Retro Game Reversing

Some ecosystems give decompilers extra hooks beyond generic native analysis.

What If Two Modules Occupy the Same Memory?

Some games load a piece of code only when it is needed, then replace it with another. The Doubutsu no Mori decompilation guide describes actor overlays that are loaded and unloaded, with relocation data to adjust addresses when an overlay enters RAM 7. A ROM offset, the overlay’s virtual address and its eventual RAM address are different parts of that story.

If two modules can occupy the same address range at different times, one flat memory map cannot describe both sets of bytes. Ghidra’s memory documentation provides overlay blocks for alternate contents of a physical region. Within an overlay, discovered code references prefer that overlay’s address space when a block exists at the destination offset 8. This can keep two functions with the same numeric address distinct, though cross-overlay references still need scrutiny.

The game’s loader and relocation records establish which bytes and addresses belong together. A plausible decompilation at the wrong address is still the wrong function.


Delphi Binaries

Delphi executables often retain class VMTs, RTTI, DFM form resources, and published method names. Delphi-aware tools can recover forms, buttons, and event handlers even when method bodies stay assembly-heavy. That is structural recovery backed by metadata, not pure instruction lifting.

Game Maker and Other Domain-Specific Runtimes

Early Game Maker EXE files embedded recognizable resource and script formats. Dedicated Game Maker decompilers target those formats directly instead of relying on a general-purpose C decompiler. When a runtime carries its own bytecode or project structure, domain-specific tools usually outperform generic ones.

Bytecode and Script VMs Inside Games

Many games embed a custom bytecode interpreter for AI, cutscenes, or UI logic. A general decompiler shows the VM dispatcher in native code, not the script language. Reverse engineers then build a separate script decompiler once they map the opcode table. The native pipeline still helps locate that interpreter.


Interactive Decompilers Used in Game RE

These tools embed the pipeline above behind a GUI and scripting API:

  • Ghidra - Free, multi-arch, collaborative; uses P-code and SSA internally 1.
  • Hex-Rays Decompiler - Commercial IDA Pro add-on; widely used for PC and some console binaries 5.
  • Binary Ninja - Modern UI with built-in MLIL/HLIL intermediate layers and strong scripting 9.
  • RetDec - Open retargetable decompiler useful for batch conversion and integration 4.
  • Reko - Open-source multi-arch decompiler aimed at interactive analysis 10.

For a broader tool list and related workflows, see:


RetDec

Cooper presents Peter Matula’s Hack.lu lightning talk introducing RetDec, an open-source machine-code decompiler. It offers a short comparison point for the interactive tools discussed here.


Limits of Automatic Decompilation

Perfect source recovery is impossible in the general case 3. Multiple different programs can compile to identical machine code, and optimizations erase the original spelling of the source.

Expect these limits in practice:

  • Wrong or generic types until you apply structs from headers or debug builds.
  • Missing inline functions merged into callers.
  • Flat or goto-heavy output on functions with unusual control flow.
  • Incorrect calling conventions on old or custom ABIs until manually fixed.
  • No comments, asserts, or preprocessor context.

Treat decompiler output as an editable hypothesis. Matching decompilation projects succeed because humans iteratively rename, split, and rewrite code until it assembles to the same bytes as the retail ROM 11.

For a catalog of game projects taking that approach, see:

Patterns for Matching Decompilation

Steinar H. Gunderson’s series shows how to rewrite C recovered from assembly when trying to match the original machine code. The posts cover these recurring patterns:

  • Anti-CSE - Restore expressions combined by common-subexpression elimination.
  • Do-while - Turn a backward goto into a loop.
  • Do-while-while - Fold an initial condition and a do-while into a while loop.
  • Range comparison - Recognise a bounded range test hidden in unsigned subtraction.
  • Nested if/goto - Combine nested branches into one condition.
  • If-add-else - Move a default assignment into an else branch or conditional expression.
  • If to condition - Replace branches that assign zero or one with a boolean expression.
  • Un-repetition of conditions - Remove a condition calculation repeated across branches.

Troubleshooting Matching Decompilation

Matching decompilation reconstructs C or C++ that reproduces the target machine code with a particular build configuration. The N64 reversing guide covers establishing a baseline build and replacing assembly one function at a time.

The archived decomp.wtf project includes short guides written for automated decompilation agents. Their useful manual counterpart is a disciplined experiment: identify the remaining difference, change one relevant detail, compile, and record the result. The following sections collect troubleshooting ideas from those guides rather than requiring their automation tools.

Classify the Difference Before Editing

The matching workflow guide recommends resolving instruction structure before operands, then register allocation 12. Use the actual disassembly alongside the score; a percentage alone does not explain the cause.

These symptoms suggest experiments, not unique diagnoses:

Difference What to investigate
Missing or extra instructions Branch structure, loop shape, casts, repeated expressions and calls
Wrong constant or memory offset Literal value, field offset, element size and signedness
Wrong relocated symbol Callee, global declaration or referenced data
Same operations but different registers Declaration order, temporaries and how long values remain live
Same operations in a different order Expression structure, statement order and instruction scheduling
Different stack offsets or frame size Local types, arrays, temporary structures and inlined helpers

Preserve the best candidate and note unsuccessful experiments. Before changing compiler settings, check the project’s known configuration and neighbouring matched functions. The objdiff guide also explains how symbol and relocation differences appear in comparisons 13.


Inspect the Surrounding Source

A function is compiled in the context of its source file and included headers. The CodeWarrior guide highlights inline helpers, local declarations and per-file pragmas as matching variables 14.

If the target contains a call but your build contains arithmetic in its place, inspect whether a helper became inline. If the operations match but registers differ, try changing declaration order or introducing a meaningful temporary before rewriting the algorithm. Reuse the project’s existing vector, matrix and SDK helpers when they describe the observed operation.

These are compiler- and version-dependent experiments. Record the compiler, flags and resulting diff rather than treating a register-allocation heuristic as a universal rule.


Match the File’s Data as Well as Its Functions

Matching each function does not establish that the rebuilt file supplies the correct data or links into the final image correctly. The project integration guide describes checking literal pools, section placement and data ownership after functions match 15.

Inspect these remaining differences:

  • Constant and string pools - Function order and literal use can affect their layout.
  • Storage placement - Check read-only versus writable data and file-scope versus function-local statics.
  • Missing definitions - An extern declaration does not supply a table that the source file must define.

Compare the rebuilt object’s data and relocations, then run the project’s full binary comparison. Do not infer a complete rebuild from a function similarity score.


Recover Candidate Source-File Boundaries

Groups of functions sharing constants or strings, related class methods, and module entry points can suggest original source-file boundaries 15. Treat a proposed split as a hypothesis: assign the associated data, rebuild, and check that existing matches and the final image remain intact. These clues do not recover an original filename by themselves.

The Ghidra tools guide also introduces CodeCut and DeepCut for investigating object-file boundaries.


Learn from Already-Matched Functions

The similarity guide suggests comparing instruction sequences to find repeated handlers, expanded inline helpers and template instances 16. An already-matched neighbour can reveal a useful source structure or compiler idiom.

Start from its structure, then verify every changed call, constant, offset and branch against the target. Similar instruction sequences do not establish identical behaviour. Record the relationship so that matching one member of a group helps with the others.


Repair a Machine-Generated Draft

The m2c draft guide explains placeholders that need manual interpretation 17. Treat generated C as a hypothesis and resolve the evidence behind each placeholder:

  • Unknown types and guessed prototypes - Compare memory accesses and other callers before supplying declarations.
  • Pointer-plus-offset expressions - Collect repeated offsets and test a structure layout against them.
  • Saved-register artifacts - Distinguish calling-convention preservation from source-level variables.
  • Unstructured branches - Trace the assembly paths before replacing them with loops or conditions.
  • Large nested expressions - Try meaningful temporaries when their value lifetimes better explain the target registers.

After changing shared types or prototypes, rebuild affected code and verify previously matched functions too 12.


Test the Matching Checker with Deliberately Wrong Candidates

A useful checker must reject incorrect reconstructions as well as recognise correct ones. The decomp.wtf data tests vary strings, floating-point constants and pointer-table entries to check that referenced data is included in the judgement 18.

Start with a known matching candidate and make one temporary change at a time:

Experiment Expected result
Change one character in a referenced string Rejected because the data differs
Change a referenced floating-point constant Rejected even if the instruction sequence still looks the same
Swap two different pointer-table entries Rejected because the indexed results differ
Restore the original candidate Accepted again, confirming the test did not leave a stale result

For example, change a used string from "ready" to "reedy" while keeping its length unchanged. Rebuild and run the same checker used to report progress. If it still reports a complete match, inspect whether the check covers only instructions, masks relocations, omits referenced data or used an old build.

The project-level judge tests separately exercise wrong constants, generated literal names and stale match records 19. Different symbol names are not themselves evidence of different bytes, while an identical-looking load can refer to different data. State whether a result verifies an isolated function, its referenced data, a linked object or the complete image. After restoring the candidate, run the full rebuild comparison; a function-level data check does not establish final file layout.


Use Permutation Searches as Candidate Generators

A permutation search compiles source variations and ranks their similarity to a target. decomp-permuter supports random changes and explicit alternatives, using a function source, target object and compiler command 20. Use it after checking the toolchain, signature and main control flow.

Keep a known baseline, limit the search to the unresolved region, and retain the exact compiler settings with each candidate. Review improvements in the disassembly before integrating them into the real source file.

The decomp.wtf permuter includes statement swapping, type changes and casts 21. Those operations are not automatically semantics-preserving. For example, swapping x = 1; and y = x; changes the value assigned to y when the previous value of x differs from one. Signedness and narrowing casts can likewise change comparisons or arithmetic.

Treat a lower diff score as a search hint. A final candidate still needs source review, the project’s complete matching checks and relevant behavioural checks. The AI-assisted workflow applies the same distinction to automated submissions.


AI-Assisted Matching Decompilation

Recent AI agents can use a decompiler’s output as the beginning of a compiler-backed matching workflow. The agent writes candidate C, runs the game’s original toolchain, examines assembly differences and repeats until the bytes match. This approach is relevant to Nintendo 64 and PlayStation preservation projects, but it does not make the original programmer’s source or intent recoverable.

For concrete game case studies, tools, code examples and a comparison of Claude Opus 5.5 benchmarks, see:


Getting Started

For an N64 matching project, the AI Assisted N64 Decompilation Tutorial walks through decompdecomp’s free local pass, agent setup, and compiler suggestions, with measured model results and costs.

If you want to experiment with decompilation on a retro binary:

  • Pick a small function with clear compiler-generated patterns first, such as math helpers or init code.
  • Load the ROM or executable in Ghidra, let analysis finish, then open the decompiler window on one function.
  • Compare disassembly and pseudo-code side by side until the mapping clicks.
  • Rename one variable and one function, define a struct for a repeated memory access pattern, and re-decompile.
  • Read the disassemblers post first if CFG or basic-block terminology is unfamiliar.

Compare each inferred type or branch with the instructions that produced it. A single corrected function signature can make its callers easier to read too.

To practise matching decompilation rather than only reading decompiler output, Decomp Academy offers graded exercises using real GameCube, Game Boy Advance and Nintendo 64 functions. You write C and compare the resulting machine code against the target, which makes compiler behaviour part of the lesson.

Starting a Dreamcast decompilation

🔧

dc-decomp-kickoff is a tool for starting a decompilation of a Sega Dreamcast game from your own disc. It gives you a repo that rebuilds the executable byte for byte from one assembly file per function, as the baseline that keeps matching while C replaces the assembly, with functions named by SDK signature where possible. A function is done when its C behaves like the original, proven in an SH-4 simulator (calls, memory writes and result on test cases); byte-identical C is the bonus tier. It can also carry verified C for shared library code (the NAOMI library, the C runtime) from a sibling decomp into a new one, checked again on the new game’s own bytes, and it builds a text map for translators. The repo contains no game code or data.


References

  1. National Security Agency, “Ghidra Software Reverse Engineering Framework,” https://ghidra-sre.org/. ↩ ↩2 ↩3

  2. Cytron, Ferrante, Rosen, Wegman, and Zadeck, “Efficiently Computing Static Single Assignment Form and the Control Dependence Graph,” ACM Transactions on Programming Languages and Systems, 1991, https://doi.org/10.1145/115372.115320. ↩ ↩2 ↩3

  3. Cifuentes, Reverse Compilation Techniques, PhD thesis, Queensland University of Technology, 1994, https://www.library.qut.edu.au/documents/10909/7092644/7092644.pdf. ↩ ↩2 ↩3 ↩4

  4. Avast, “RetDec: Retargetable Machine-Code Decompiler,” https://github.com/avast/retdec. ↩ ↩2

  5. Hex-Rays, “Hex-Rays Decompiler,” https://hex-rays.com/decompiler/. ↩ ↩2 ↩3

  6. ECMA International, “ECMA-335: Common Language Infrastructure (CLI),” https://www.ecma-international.org/publications-and-standards/standards/ecma-335/. ↩

  7. zeldaret/af - Working with Splat, “Actors and Overlays”. ↩

  8. Ghidra API - Memory, overlay block and code-reference behavior. ↩

  9. Vector 35, “Binary Ninja Documentation,” https://docs.binary.ninja/. ↩

  10. Reko Decompiler Project, https://github.com/uxmal/reko. ↩

  11. decomp.me community wiki, “Matching Decompilation,” https://decomp.me/. ↩

  12. decomp.wtf - The matching loop. ↩ ↩2

  13. decomp.wtf - Reading objdiff. ↩

  14. decomp.wtf - Matching Metrowerks CodeWarrior. ↩

  15. decomp.wtf - Working in a fork of a public decomp. ↩ ↩2

  16. decomp.wtf - Look-alike functions. ↩

  17. decomp.wtf - Reading the m2c draft. ↩

  18. decomp.wtf - Referenced-data matching tests. ↩

  19. decomp.wtf - Project-level matching judge tests. ↩

  20. decomp-permuter - Search modes and required inputs. ↩

  21. decomp.wtf - Source mutation and permutation search. ↩