Reverse engineering

I want to do some reverse engineering but my knowledge is rather limited. I'm working on Windows.

To be precise, I want to reverse-engineer a protocol used by a flash application, but I can't just sniff packets as the connection is encrypted.

So I thought that perhaps I could "hot patch" the functions that encrypt/decrypt the data sent/received so that they would save the data in a file, which I could then analyze.
But I'm not sure what to do now. I know how to use OllyDBG to attach to the process I'd want to infiltrate and check the modules it imports and the functions imported/exported by the modules, but I don't know how to (easily) check which of those functions are called, and then I would have to change the code of those functions, which I've never done before, and I don't even know if I should prepare a second DLL to replace the original one or simply change the functions directly in the memory of the process I'm targeting.
Or maybe I'm overthinking all this and there's no need to mess around with WinAPI stuff and I could exploit the fact that the protocol is used by a flash application.

you stink

>>2 cant be cose u are behind the internet so u can smell me what u say now smart ass?

*can´t

you still stink

Reverse engineering is hard and takes some practice (actually it's easy, but time consuming).

Maybe these tutorials will help you:
http://tuts4you.com/download.php?list.17

its hard but easy

>>6
Yeah, I realize that. I'm just wondering if you guys know anything that would help me, as I only want to reverse-engineer a particular part of a particular type of application (for now, at least).
But I'll skim (at the very least) through those tutorials, thanks. I've had the link for a while but never really read them.

This thread is absolutely crazy!

Learn to statically reverse engineer the code. Get a good disassembler, like IDA Pro. Once you're familiar with the target, you may debug it in a safe environment that you control.
It's possible to try to do all analysis live with only debuggers and various API monitors/hookers, but you will likely miss a lot of detail. My personal preference is to mix the two approaches. If I know exactly what I want to get out of something, I may skip the static part, but otherwise it's not wise to do so.

flash application
one word, SWF DECOMPILER, thread over.

>>6,10
Using native tools for this is as productive as trying to debug a Java application by tracing through the JVM.

>>11
I haven't noticed that he was talking about Flash. There are some specialized Flash disassemblers/assemblers/deobfuscators and even emulators/JITs, besides Sourcetec's decompiler (which doesn't always generate valid code). Unfortunately I don't think I know any debuggers, in which case you will have to resort to native debugging: I've done it multiple times, especially when digging undocumented Flash stuff or when hooking certain Flash function calls - why mess with complicated permissions when you can just grab everything you need from Flash's DLL memory (the only problem is that it's written in SEPPLES and it's filled with indirection making it a pain to understand, but give it a few hours and you'll get familiar with the code).

>>11,12
SWF DECOMPILER
Yup, tried flasm before and managed to clean up the source and get about 2k lines which I'd need to analyze, but the code was still obfuscated and I think that all the network stuff was placed in sections that were decompiled as unknown actions with unknown parameters.
Tried the Sourcetec (Sothink) decompiler now, but while it's generally more convenient and divides the flash into actions and shit, the decompiled code is full of ULs (which I guess stands for "unknown label").

hooking certain Flash function calls
This is what I believe would be the best in my case; any tips on how to do that, or links to useful resources?

I'm mainly interested in the functions encrypting/decrypting data (for SSL), which I'd guess would be imported from crypt32.dll, but it seems that npswf32.dll doesn't import enough of them to matter (or at least I can't see them by looking at the names defined in the DLL). Perhaps the relevant functions are imported dynamically?