Down with OPP: Other Programs' Pixels

I'm trying to figure out how I can read the graphics output from other programs for the sake of creating (meta) visualizations.

I think the key to this is finding out where the framebuffers are, or somehow accessing them. If I can start drawing bitmaps of framebuffer data from one other (abritrary) program, I'm satisfied.

The thing is, I'm a little over my head. I've never worked with low-level languages before, but the idea of reading other programs' memory seems to necessitate one's use, if only for the sake of bypassing restrictions from a compiler. But I wouldn't know how to start looking for framebuffers in memory. (Come to think of it, I've never looked for anything in memory before, at least memory I didn't allocate myself.) There is an assembly language for GPUs by OpenGL.

I've also run across Cg and GLSL in my research, which look awesome for making shaders, but I don't know if they'll help me do this specific task.

Any ideas?

but the idea of reading other programs' memory seems to necessitate one's use, if only for the sake of bypassing restrictions from a compiler.
Not really, you can read other program's memory using documented system APIs on Win32 (ReadProcessMemory/WriteProcessMemory/VirtualAllocEx/VirtualProtectEx...), and on *nix you can do it via ptrace or sometimes by reading some path in /proc/. You will need corresponding priviledges in either OS (Administrator or SYSTEM or just the SeDebug priviledge on Win32, and I think you need root on *nix). So far, this only requires having access to a FFI in a high-level language, or the ability to link or low or generate low-level code.

In a weird OS which doesn't let you do such things, you would require having to write your own driver which runs in kernel mode to do this.