[Failure] I suck at reverse engineering

Hey /prog/, here's my problem. I have a dll with a couple of functions that I'm currently attempting to figure out what they do, but the fact that I seem to be completely lost at this, plus the fact that Windows crap is alien to me and all the tutorials/FAQs/etc I find in Google are all about ``cracking r3gg3d warez appz'' isn't helping. All I have is that dll and an executable that calls functions from that dll. I obviously already have the names of the dll's functions and how many arguments they take for all but one function (OllyDbg's ``Call DLL Export'' was able to guess from the RETNs, but even I could have done that).

A few basic questions:
- Opening the executable directly in OllyDbg doesn't work, I have to launch the program and then attach to the process. Not really a problem, but is this some sort of lame anti-debugging feature or something else?

- In Windows, arguments are passed on the registers, on the stack or what? I'm guessing registers.

- In the main executable, how can I search for calls to the DLL? I can set breakpoints in the DLL's functions, but that doesn't let me see where in the original exe they were called.

- How can I figure out the types a function is expecting/returns (and the number of arguments when there's no RETN at the end)? I realize the answer to this is ``Go read the disassembly, you dumbass,'' but are there any basic pointers that might make this easier?

Basically, any input related to disassembly and reverse engineering that isn't "how 2 insert NOPs 4 n00bz" or "how2make a keygen if upay me enouhg ;)" is highly appreciated.

>>1
In Windows, arguments are passed on the registers, on the stack or what? I'm guessing registers.
You assume there are enough registers to store all the NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL arguments, right? I don't believe it.

Go read the disassembly, you dumbass

>>2
HABEEB IT

Win32 function calls could be optimized by using REP STOSD or something like that for filling the stack with 1 KB worth of zeroes prior to CALL.

>>1

1) Most likely the application is calling LoadLibraryA/W on the DLL, to actually bring it into the process space. This isn't an anti-debug feature, it's what normally happens... Try set a breakpoint on LoadLibrary and inspect the arguments in Olly, to see if it's loading your DLL. Speaking about arguments...

2) Arguments are passed on the stack, from right to left. This is true for the "stdcall" calling convention, which all of the Win32 API uses, as well as what is typically (almost always) used for exported functions in dlls.
However, just be aware that there is a calling convention known as "fastcall" whereby the first two arguments are passed in ECX and EDX, with the remaining args passed on the stack.

3) If the function you're reversing has a "normal" stack frame (push ebp; mov ebp,esp; sub esp, XXX), then the function accesses the arguments via [ebp+8], [ebp+C], [ebp+(4N+4)] where N is the number of arguments. Just be aware that this is assuming that the stack frame is not destroyed by the compiler, and that the stdcall convention is being used.

>>6
Just a quick example of arg passing:
foo(1,2,3)
Upon entering foo, the stack will look like:

Return Address   <---- bottom of stack, pointed at by ESP
1
2
3                <----- higher address

And the code:

push 3
push 2
push 1
call [foo]

>>1
Just by the way, you'll find a wealth of information if you search for tutorials on x86 programming on Windows. Most of the things you'll learn you can directly apply to RE.

Seriously, 3 seconds in google. Fuck off OP.
http://www.programmersheaven.com/2/Calling-conventions#stdcall

>>6
1) I don't get it, shouldn't LoadLibrary be able to find the dll when ran through Olly as well? Also, I feel like an idiot, but what's the equivalent of gdb's break function_name? I can only seem to place breakpoints on the instruction I currently have selected.

>>6,7
Ah, yes, I was mixing C function calls with syscall arguments (on the registers for Linux, on the stack for BSD and OS X), in retrospect, that was a dumb question, but I never really dealt with assembly much. In my case, it looks like the compiler felt like treating ebp like a regular register, because it seems to access everything via [esp+n]. I'm also assuming OllyDgb displays instructions in destination, source format, otherwise I'm seeing would make even less sense.

Thanks for the actual helpful feedback.

First let me give you a pointer to http://tuts4you.com/ , which is IMO the best site with reverse engineering tuts, tools and a forum. Also highly recommended: Lena151's video tutorials for beginners ( http://tuts4you.com/download.php?list.17 ). Anyway, on to the questions:

Opening the executable directly in OllyDbg doesn't work, I have to launch the program and then attach to the process. Not really a problem, but is this some sort of lame anti-debugging feature or something else?
Hard to say without more info, but if(IsDebuggerPresent())exit(1); is indeed a possibility. For a quick fix try using an anti-antidebug Olly plugin, Olly Advanced does the basics and fixes some Olly bugs.
(More hardcore are the Phantom and Poison plugins, but you don't need them yet)
Also, don't try to run the .dll and make sure the power is on

In Windows, arguments are passed on the registers, on the stack or what? I'm guessing registers.
Calls to system libraries (kernel32 etc) are stdcall. C compilers use cdecl or stdcall, unless the optimizer goes crazy and all bets are off. Fastcall is pretty much only used in Delphi.

In the main executable, how can I search for calls to the DLL? I can set breakpoints in the DLL's functions, but that doesn't let me see where in the original exe they were called.
Right click in Olly -> Search all intermodular calls
When you hit a breakpoint you can see where it's called from in the stack, the 'K' button gives a nice table.

How can I figure out the types a function is expecting/returns (and the number of arguments when there's no RETN at the end)? I realize the answer to this is ``Go read the disassembly, you dumbass,'' but are there any basic pointers that might make this easier?
Pretty much the only way is to see how it is called or how the arguments are used in the function. Setting a breakpoint and looking at the stack might give you some idea too.

what's the equivalent of gdb's break function_name
ctrl+g (goto), F2 (set breakpoint). Also there are some command bar plugins that let you type "bp function_name".

the compiler felt like treating ebp like a regular register
You'll have to know the value esp each time to figure out what it references or use the Interactive Disassembler (IDA) and let it do that for you.

instructions in destination, source format
Only GNU uses AT&T for x86, everyone else uses Intel syntax.

Back to /gorp/ please.

Get a copy of IDA Pro. You'll find it much more useful than Ollydbg for this sort of thing.

>>13
0/10
Because having four rows of useless toolbars, and 15 different windows which you'll never use, AND having the most socially awkward interface ever designed is useful.

Listen, I might just ask you to kindly GTFO, if you don't mind.

>>14
socially awkward interface
Wat the hell does that mean?

>>15
It means the interface has glasses and acne.

>>13 is DQN.

useful for very many things Several AI   techniques provided as   a core API   or used by   use at all   and even if   you know no   other popular LAMP?

>>14
Spoken by an idiot who has obviously never used it.