Reverse engineering

I want to do some reverse engineering but my knowledge is rather limited. I'm working on Windows.

To be precise, I want to reverse-engineer a protocol used by a flash application, but I can't just sniff packets as the connection is encrypted.

So I thought that perhaps I could "hot patch" the functions that encrypt/decrypt the data sent/received so that they would save the data in a file, which I could then analyze.
But I'm not sure what to do now. I know how to use OllyDBG to attach to the process I'd want to infiltrate and check the modules it imports and the functions imported/exported by the modules, but I don't know how to (easily) check which of those functions are called, and then I would have to change the code of those functions, which I've never done before, and I don't even know if I should prepare a second DLL to replace the original one or simply change the functions directly in the memory of the process I'm targeting.
Or maybe I'm overthinking all this and there's no need to mess around with WinAPI stuff and I could exploit the fact that the protocol is used by a flash application.

Reverse engineering is hard and takes some practice (actually it's easy, but time consuming).

Maybe these tutorials will help you:
http://tuts4you.com/download.php?list.17

Learn to statically reverse engineer the code. Get a good disassembler, like IDA Pro. Once you're familiar with the target, you may debug it in a safe environment that you control.
It's possible to try to do all analysis live with only debuggers and various API monitors/hookers, but you will likely miss a lot of detail. My personal preference is to mix the two approaches. If I know exactly what I want to get out of something, I may skip the static part, but otherwise it's not wise to do so.