Basic division with negative numbers in Linux x86 Assembly and spanning over multiple registers

28 Mar 2013

So I'm doing the SLAE32 course by SecurityTube, it's quite interesting and in my honest opinion a very good bang for your buck. I normally don't make publicity, but I must say, for a 150 dollar course you get quite a lot in return. It even comes with the SecurityTube Gnu Debugger Expert to make sure you understand how the gdb works (since it's used a lot throughout the course). One of the assignments in the course was to discover how division and multiplication works. I was really interested in division as it poses some caveats to watch our for.

The source code for our example:

global _start           
section .text
    mov edx,0x00000001
    mov eax,0x00000000
    mov ecx,0x00000002
    div ecx
    mov edx,0x00000000
    mov eax,0x7FFFFFFF
    mov ecx,0x00000002
    idiv ecx
    ;divide with negative number
    mov edx,0
    mov eax,0x0000000A
    ;move -2 in hex into ecx
    mov ecx,-0x00000002
    idiv ecx
    ; exit program
    mov eax, 1
    mov ebx, 10
    int 0x80

My GDB hook stop is:

(gdb) define hook-stop
Type commands for definition of "hook-stop".
End with a line saying just "end".
 print/x $eax
 print/x $edx
 print/x $ecx
 print $eflags
 disassemble $eip+10
(gdb) break _start

div: Unsigned Division

Let's look at unsigned division first. Unsigned means the registers we will use can only contain positive values. This means the numbers can range from 0 to 4 294 967 295 Have a look:

    mov edx,0x00000001   ;move 1 into edx
    mov eax,0x00000000   ;move 0 into eax
    mov ecx,0x00000002   ;move 2 into ecx
    div ecx              ;divide EDX:EAX by the value in ecx

The number we have is spanned over EDX:EAX giving us 0x10000000 or 4 294 967 296 \( 2^{32} \) (this doesn't fit a 32-bit register anymore, so we span it over EDX:EAX). What we want to do is divide this by 2, so \( \frac{4 294 967 296}{2}=2 147 483 648 \) . Let's see what's happening in the registers with GDB after we execute the div:

$17 = 0x80000000 ;EAX
$18 = 0x0        ;EDX
$19 = 0x2        ;ECX
As we can see EAX contains 0x80000000, which is indeed 2 147 483 648 decimal (unsigned).

idiv: Signed Division

Now idiv is an instruction which looks at signed divisions. Meaning we also need to take negative numbers into account. This means our numbers are limited from -2 147 483 648 (0x80000000) up to 2 147 483 647 ( 0x7FFFFFFF). If you try using bigger numbers (which I did during my quest to understand division in asm) you will run into exceptions. We will now do \(\frac{2 147 483 647}{2}\) which yields 1073741823.5, but since we aren't with floats will result in 1073741823 and a rest of 1. Let's have a look at the source code:

    mov edx,0x00000000   ;move 0 into EDX
    mov eax,0x7FFFFFFF   ;move 2 147 483 647 into EAX
    mov ecx,0x00000002   ;move 2 in ECX
    idiv ecx             ;Signed division by 2


$33 = 0x3fffffff ;EAX
$34 = 0x1        ;EDX
$35 = 0x2        :ECX
As we can see we have 0x3FFFFFFF in EAX (1 073 741 823) and a rest of 1 in EDX. Let's continue and divide with negative 2:

    ;divide with negative number
    mov edx,0           ;move 0 into EDX
    mov eax,0x0000000A  ;move 10 into EAX
    mov ecx,-0x00000002 ;move -2 in hex into ECX
    idiv ecx

You can also write 0xFFFFFFFE instead of -2. Let's see what happens in GDB:

$49 = 0xfffffffb ;EAX
$50 = 0x0        ;EDX
$51 = 0xfffffffe ;ECX
We have -5 in EAX, no rest in EDX and -2 in ECX. That's it, part of the SLAE32 is exploring yourself, so I hope you found this interesting and that this clarifies some further questions you might be having about dividing in ASM with signed integers. If you have questions, remarks or constructive criticism, feel free to leave a comment below or email me (my email is on the about page).