This post shows how to reverse a niteCTF-22 Rust VM challenge.
There are two files vm (Rust Vitual Machine) and encrpyt (Program File containing instructions). Running the binary, it asks for a input, prints ‘\n’ and exits.
Analyzing Binary in IDA
Loading up the binary in IDA, there is a main function which reads the content of the encrypt file into an array mem, and calls run_loop. 
The run_loop calls process_opcode, and program runs till pc doesn’t become 0.
1
2
3
4
r-> register array (total 8 registers)
mem -> memory
cflag -> jump flag
pc -> program counter
The process_opcode function interprets instructions according to the op_code and returns new pc after executing the instruction. 
I like to remove all the panic and overflow conditions from the decompiled functions, it makes my eyes bleed. So here is what the process_opcode function looks like.
Reversing process_opcode the various OP codes can be summarised as:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
0x10 - 0x17 // 2 byte instruction
a b
load into register ---->
reg[a & 0xf] = b
eg: 0x11 0xa3 instruction -> reg[1] = 0xa3
0x18 - 0x1f // 3 byte instruction
a b c
reg[a & 0xf] = mem[(b<<8)+c]
0x20 - 0x27 // 3 byte instruction
a b c
load to memory
mem[c+(b<<8)] = register[a & 0xf]
0x80 // 3 byte instruction
a b c
print data from mem
prints from mem[(b<<8)+c] till it encounters a null byte
0x81 // 3 byte instruction
get input and store in mem[(b<<8)+c]
0x90 // 1 byte instruction
nop(no operation)
'0' // 2 byte instruction
'0' 0x(c)(d)
register[c] = register[d]
'1' // 2 byte instruction
'1' 0x(c)(d)
register[c] = ~(register[c] & register[d])
'2' // 2 byte instruction
'2' 0x(c)(d)
if register[c] ==register[d]
cflag = 1
'@' // 2 byte instruction
'@' 0x(c)(d)
reg[c] = reg[c] >> (mem[0xcd]& 0xF)
'A' // 2 byte instruction
'A' 0x(c)(d)
reg[c] = reg[c] << (mem[0xcd]& 0xF)
'P' // 3 byte instruction
'P' b c
if(cflag == 1)
cflag = 0
pc +=3
else:
pc = ((b<<8)+c)
Invalid op_code
print UNRECOGNIZED OPCODE: {opcode}! and exit
Analysing the encrypt file
Printing every instruction in a new line for better visiblity. All the instructions are here
1
2
3
4
5
6
7
8
9
10
17 00 // reg[7] = 0
16 01 // reg[6] = 1
81 b0 00 // reads input from std:io and stroes it into mem[((0xb0)<<8) + 0x00]
10 59 // reg[0] = 'Y' (0x59)
20 c0 00 // mem[((0xc0)<<8) + 0x00] = reg[0]
10 6f // reg[0] = 'o' (0x6f)
20 c0 01 // mem[((0xc0)<<8) + 0x01] = reg[0]
10 75 // so from mem[(0xc0)<<8)] onwards it stores 'You got it right!'
20 c0 02 // which gets printed when we enter the correct flag.(80 C0 00 last instruction in the file)
...
Further instructions loads a character from our input into in reg[0], some number in reg[1] , operates on it and checks whether it is equal or not to some hardcoded value.
1
2
3
4
5
6
7
8
9
10
...
18 b0 00 reg[0] = mem[(0xb0 << 8)+0x00]
11 a3 reg[1] = 0xa3
30 20 reg[2] = reg[0]
30 31 reg[3] = reg[1]
31 01 reg[0] = ~(reg[0] & reg[1])
30 10 reg[1] = reg[0]
31 20 reg[2] = ~(reg[2] & reg[0])
31 31 reg[3] = ~(reg[2] & reg[0])
...
1
2
3
4
...
32 12 if (reg[1]==reg[2]) cflag=1
50 00 00 if (cflag!=1) pc = 0 and the program exits
...
Basically there is a character-by-character comparison and the program terminates as soon as it encounters a wrong character, I used this vulnerability to solve this challenge.
I opened hex editor, edited the 50 00 00 instruction to invalid instruction 88 00 00, now one by one went on correcting the instruction back and forth, side by side brute forcing characters with a simple python script, so whenever I got a correct character program returns UNRECOGNIZED OPCODE and I know that previous entered character is correct.
1
2
3
4
5
6
7
8
9
10
11
12
from pwn import *
import string
flag = "nite{"
for i in string.printable:
p = process("./vm")
p.sendline(flag+i)
x = p.recv()
if len(x)>3:
print(flag+i)
break
And the flag is nite{n4nd_1s_un1v3rs4l_g4te}.