Static Devirtualization of Tencent VM

Prerequisite Material

• https://k0mkc.hatenablog.com/entry/2026/07/29/042406
• https://arxiv.org/abs/2603.18355
• https://arxiv.org/abs/1909.01752

Legal Disclaimer

This work constitutes reverse engineering, decompilation, and de-virtualization performed solely for the purpose of achieving interoperability with ACE-protected software on Linux and Proton environments. Such activities are undertaken in good faith under applicable exceptions to copyright law, including those recognizing the right to reverse engineer software for interoperability purposes.

Any sharing has been limited exclusively to trusted third parties for independent technical validation. The original protected software remains subject to copyright and the protections of the Digital Millennium Copyright Act (DMCA) and other applicable laws.

Introduction

Over the course of several months we have noticed an increased interest in Tencent VM obfuscation. We have had complete static devirtualization of this VM obfuscation for quite some time now and have noticed others have achieved similar deobfuscation results. As AI continues to advance it has brought to light the gap between strong obfuscation and simple obfuscation. We have long held the belief that the classic style of “Virtual Machine” obfuscators are not strong to an attacker that has a flexible lifting and recompilation framework. Many companies have attempted to build inhouse obfuscation solutions, most of which are not very strong and are being deobfuscated by private entities. In this article we will explain in great detail the Tencent virtual machine obfuscation, how it supports CET and SEH, and how it is weak against guided symbolic evaluation.

In our last article, Static Devirtualization of Themida, a reader asked about coverage statistics and other versioning related questions. Although we do not have version information for Tencent VM, we will provide detailed information about the static devirtualization coverage. A third party (Daax from secret club) has been provided with these files and can independently confirm the cleanliness and coverage of the output. A later section in this article will cover that more formally.

Virtual Machine Architecture

The Tencent virtual machine obfuscation is structured so that the original function entry point jumps into the .tvm section. Stack space is allocated for the VM context and all GPRs and EFLAGs are saved into this context (first on the stack and then moved into the VM context). The VM is highly reminiscent of VMProtect in this manner, as VMProtect pushes all GPRs to the stack and then the first few VM handlers pop them from the stack and store them in the VM context area. Execution threads in and out of a shared virtual machine dispatch loop. Exits happen to execute calls and boxed instructions.

Boxed Instructions

The Tencent VM models only a subset of AMD64. Anything outside that subset is handled by what we refer to as a boxed instruction: the VM performs a context restoration, executes the original instruction natively. At that moment the machine state is indistinguishable from what the unvirtualized function would have produced, so the instruction executes with correct semantics without the VM having to model it at all. Once the instruction retires, the VM captures register state back into the VM context and resumes dispatch. More usefully, every boxed instruction is a point where the VM is forced to materialize real guest state, which makes it a reliable sink point. We rely on exactly this property when recovering the original stack frame size. It is also important to note that a way to execute native instructions in virtualized functions is a practical necessity. Consider for example instructions like CPUID which would be impossible to model inside the VM without actually executing the CPUID instruction. Not to mention other architectural instructions like RDMSR, WRMSR etc.

CET Compatibility

Entering the dispatcher with a CALL creates a problem on hardware with Control-flow Enforcement Technology. Under CET a CALL pushes the return address onto a shadow stack in addition to the data stack, and a RET pops it and compares the two. The VM’s calls into the dispatcher never return. Left alone, the shadow stack would grow monotonically for the lifetime of the virtualized function and eventually fault. Tencent VM handles this at runtime rather than at protection time. It executes RDSSPQ to read the current shadow stack pointer. RDSSPQ is encoded in the hint-NOP space, so on a processor or in a process without shadow stacks enabled it retires as a NOP and leaves its destination register untouched, zeroing the register beforehand and testing it afterward is therefore a feature check that is correct on old and new hardware alike and costs nothing on either.

When the check indicates CET is active, the VM issues INCSSPQ 2 (register containing 2), advancing the shadow stack pointer by two entries and discarding the two return addresses that no matching RET will ever consume.

SEH Compatibility

The VM is covered by a single large .pdata entry whose unwind info spans the entire VM range, including the VM entry, the dispatcher, and the handlers. That unwind info carries a language-specific exception handler for the VM. There is a never-executed function that has what we refer to as phantom unwind info, whose only purpose is to describe unwind operations. The VM entry reserves a 0x68 byte local area, of which a 0x48 byte block is used exclusively during unwinding and holds all 9 non-volatile registers. The phantom unwind info attached to this dummy function describes the saves of all non-volatile registers, and an address of that dummy function is kept in [RBP+0], one of the slots the VM entry reserves, for the entire lifetime of VM execution. The VM entry’s unwind info carries a UWOP_SET_FPREG with RBP as the frame register, so that unwinding is performed relative to RBP. RBP is saved at VM entry and from then on is never used as scratch during VM execution, so the unwinder can correctly unwind at any point. When an exception is raised inside the VM, the VM’s exception handler runs and copies every guest non-volatile register into the save area reserved at VM entry, and overwrites the unwind target slot located 8 bytes below the guest RSP with the guest RIP. The handler then returns ExceptionContinueSearch.

The unwinder, following [RBP+0], picks up the dummy function as the next entry, its .pdata is looked up, and through that phantom unwind info every guest non-volatile register the handler just staged is written back into the native register in the CONTEXT record it ultimately belongs to. Finally the unwinder reads the contents of the unwind target slot as the RIP and advances RSP by 8, so that CONTEXT->RIP holds the guest RIP and CONTEXT->RSP lands exactly on the guest RSP. The original function’s virtual unwind then continues as a guest state. Boxed instructions are given chained unwind info as needed because they run outside the VM.

Guided Symbolic Execution

Guided symbolic execution is the process of lifting native instructions (AMD64 in this article) to a higher level, SSA, intermediate representation which can be easily optimized and manipulated. The objective is to symbolically evaluate the entire virtualized function so that an IR function is created containing all of the semantics of the routine. In order to do this, the symbolic lifting loop needs guidance when indirect control flow is discovered. Classical virtual machine obfuscation uses bytecode to influence indirect control flow in the virtual machine (encoding which vm handlers to execute). As documented above, the register R11 contains the address of the virtual machine bytecode for the interpreter to execute using.

Guided symbolic execution uses obfuscation specific information to achieve results. In the case of Tencent VM, we want to symbolically inline the call to the VM dispatcher loop. A simple but effective heuristic is to follow calls with an int3 placed directly after them. For whatever reason, every single one of these calls in the symbolic evaluation path is a call into the VM dispatcher loop. Alternatively you could declare the VM dispatcher function for the symbolic evaluation engine as a valid call target to follow/inline.

In order to solve indirect control flow inside of the virtual machine, bytecode must be promoted to a constant within the SSA IR so that other optimizations can fold bytecode decryption operations away. This promotion of load operations must be carefully scoped such that original semantic load operations are not promoted to constants within the IR.

To prevent re-lifting (unrolling) of virtualized loops, we must track VIP so that if we have already lifted the next VIP/vm handler we will simply create a backedge to it. Tracking VIP is obfuscation specific, but for Tencent VM it is held inside of the VM context structure, the offset at which can be dynamically resolved using an algorithm that finds the last stored value into the VM context with a value being a pointer into the .tvm0 section (bytecode address will be inside of this). This heuristic works well and will automatically reveal VIP.

Virtualized Conditional Control Flow

Conditional control flow inside of the Tencent VM is done by expanding the flag comparison operations normally performed by native JCC operations into multiple VM handlers. When symbolic evaluation halts on indirect control flow with symbolic destination, it can either mean we are stopped at a virtualized JCC or our optimizations are incomplete. Virtualized JCC logic can be transformed back into a native JCC using pre-defined IR SSA DAG’s. If the current indirect control flow matches a pre-defined JCC DAG then a rewrite can be performed. Additionally during this JCC rewrite step, branch targets can be extracted from the IR as the DAG also implicitly defines where the branch destinations are in the IR.

; ── CF ── mask 0x1, idx 0
pattern vjcc.cf.ae.zero { body(0x1,   0x0)   ; %r = R{e}  %z } => { %r = R{ae} %cf }
pattern vjcc.cf.b .zero { body(0x1,   0x0)   ; %r = R{ne} %z } => { %r = R{b}  %cf }
pattern vjcc.cf.b .mask { body(0x1,   0x1)   ; %r = R{e}  %z } => { %r = R{b}  %cf }
pattern vjcc.cf.ae.mask { body(0x1,   0x1)   ; %r = R{ne} %z } => { %r = R{ae} %cf }

; ── PF ── mask 0x4, idx 1
pattern vjcc.pf.np.zero { body(0x4,   0x0)   ; %r = R{e}  %z } => { %r = R{np} %pf }
pattern vjcc.pf.p .zero { body(0x4,   0x0)   ; %r = R{ne} %z } => { %r = R{p}  %pf }
pattern vjcc.pf.p .mask { body(0x4,   0x4)   ; %r = R{e}  %z } => { %r = R{p}  %pf }
pattern vjcc.pf.np.mask { body(0x4,   0x4)   ; %r = R{ne} %z } => { %r = R{np} %pf }

; ── ZF ── mask 0x40, idx 3
pattern vjcc.zf.ne.zero { body(0x40,  0x0)   ; %r = R{e}  %z } => { %r = R{ne} %zf }
pattern vjcc.zf.e .zero { body(0x40,  0x0)   ; %r = R{ne} %z } => { %r = R{e}  %zf }
pattern vjcc.zf.e .mask { body(0x40,  0x40)  ; %r = R{e}  %z } => { %r = R{e}  %zf }
pattern vjcc.zf.ne.mask { body(0x40,  0x40)  ; %r = R{ne} %z } => { %r = R{ne} %zf }

; ── SF ── mask 0x80, idx 4
pattern vjcc.sf.ns.zero { body(0x80,  0x0)   ; %r = R{e}  %z } => { %r = R{ns} %sf }
pattern vjcc.sf.s .zero { body(0x80,  0x0)   ; %r = R{ne} %z } => { %r = R{s}  %sf }
pattern vjcc.sf.s .mask { body(0x80,  0x80)  ; %r = R{e}  %z } => { %r = R{s}  %sf }
pattern vjcc.sf.ns.mask { body(0x80,  0x80)  ; %r = R{ne} %z } => { %r = R{ns} %sf }

; ── OF ── mask 0x800, idx 5
pattern vjcc.of.no.zero { body(0x800, 0x0)   ; %r = R{e}  %z } => { %r = R{no} %of }
pattern vjcc.of.o .zero { body(0x800, 0x0)   ; %r = R{ne} %z } => { %r = R{o}  %of }
pattern vjcc.of.o .mask { body(0x800, 0x800) ; %r = R{e}  %z } => { %r = R{o}  %of }
pattern vjcc.of.no.mask { body(0x800, 0x800) ; %r = R{ne} %z } => { %r = R{no} %of }

template vjcc<FLAG, MASK, IDX, CC_SET, CC_CLEAR> {
  %rf = X86ReadFlags %f[0..5]
  %w  = launder %rf
  %m  = And %w, imm MASK
  %s  = Sub %m, imm SUB          where SUB ∈ { 0, MASK }
  %z  = X86Flag.ZF %s
  %r  = R{KIND} %z               where KIND ∈ { e, ne }
} => {
  %r  = R{ (SUB == MASK) ⊕ (KIND == ne) ? CC_SET : CC_CLEAR } %f[IDX]
}

instantiate vjcc<CF, 0x1,   0, b, ae>
instantiate vjcc<PF, 0x4,   1, p, np>
instantiate vjcc<ZF, 0x40,  3, e, ne>
instantiate vjcc<SF, 0x80,  4, s, ns>
instantiate vjcc<OF, 0x800, 5, o, no>

Simple MBA Identity Rule Reduction

Tencent VM uses trivial MBA identity rules recursively applied to generate larger MBA expressions. Below is an exhaustive list of Tencent MBA identity rules. You can simply define these in your inst-combine ruleset, running optimizations to a fixed point should fully reduce Tencent MBA.

(A|B) + (A&B)        = A + B
(A^B) + (A&B)        = A | B          [from ((B^A)+(B&A)) → x|y]
(A|B) - (A&B)        = A ^ B
((A|B)^A) + A        = A | B          [also A + ((A|B)^A)]
~((~A ^ ~B) | ~A)    = A & B          [De Morgan variant]
~(((A^B) & ~B) ^ ~A) = A & B
A - (A - (A&B))      = A & B
B - (((A&B)&B) ^ B)  = A & B

((c&A) ^ A) | A      = A              [+ all operand orderings]
(A & ((A^c) | c))    = A
(((A&c) ^ A) | A)    = A
(((c|A) ^ c) | A)    = A
(((A^B)|B) & A) + B  = A + B          [((A^B)|B)=A|B, (A|B)&A=A]
(A&B) + (A|B)        = A + B

~(A - c)             = -A + (c-1)     [reported as -A, const folded]
~( <A+c gadget> )    = -A + c

Lowering

Preparation must be done before lowering the SSA IR back to native. Again we concretized RSP to an arbitrary constant value when we started lifting so that we could reuse our existing optimizations to fold up RSP modifications. We also use concrete RSP values as a heuristic to determine if we are at a VMEXIT or not. You can keep RSP symbolic but this is just the approach we have taken for Themida, VMP, vxlang, Tencent VM, Denuvo, etc.

Additionally we must determine how big the original stack frame of the function was. For almost every function in AMD64 PE files you will not see dynamic stack allocations. It is very rare, and so, we can assume that at sink points in the program (calls, and boxed instructions) RSP will reveal the original functions’ stack frame size. I call this technique the “stack frame high water mark”.

Once we know the stack frames’ high water mark we can rebuild the functions prolog and epilog to properly represent the original functions stack frame. Special care must be taken if any spilling happens in our recompiled code. If there is any spilling, we must place the original functions stack frame after our stack frame. Any references to RSP at or above the return address will need to be adjusted for the new found size of our spill space. This will result in a truly proper devirtualized output for functions that do not make dynamic stack allocations.

Deobfuscation Coverage Statistics

Coverage is measured per driver as the fraction of virtualized functions successfully recompiled to native code. A virtualized function is identified by its entry trampoline (E9 padded with int3); after devirtualization the trampoline retargets from the .tvm0 bytecode interpreter to the recompiled .devirt code, while functions the tool could not recover still point into .tvm0

Across the four kernel drivers, 815 of 865 virtualized functions (94.2 %) were fully recovered to native code.

Limitations

Ranged for loops cause a major issue with our aggressive indirect control flow optimization approach. Take for example this for loop: for (int i = 0; i < 10; i++). When virtualized, the condition i < 10 will be turned into a VJCC, however i = 0 so our optimizations fold the VJCC to only a single branch destination, which is the loop back edge. Since we have already lifted the back edge we stop lifting and create an infinite loop. Relifting back edges may be required, symbolizing everything except VIP.

Conclusion

We have long held the belief that classical virtual machine obfuscation is not strong in the face of guided symbolic evaluation. We have demonstrated that simple compiler optimization passes (constant promotion, constant folding, and some trivial MBA reduction rules) are enough to simplify Tencent VM.

As AI assisted static deobfuscation continues to improve we will see more virtual machine based obfuscators fold away. In an upcoming article we will publish details on how we fully statically devirtualized Denuvo anti(tamper/cheat). One of the most attacked pieces of software second only to anticheats.

For our own product CodeDefender we have taken special care to directly hinder lifting based attacks and guided symbolic evaluation. This article showed that by analyzing hardened targets, obfuscation can easily identify and avoid common pitfalls that may otherwise be overlooked. If you are interested in our product or consulting, you can reach us at: [email protected].