Down with OPP: Other Programs' Pixels

I'm trying to figure out how I can read the graphics output from other programs for the sake of creating (meta) visualizations.

I think the key to this is finding out where the framebuffers are, or somehow accessing them. If I can start drawing bitmaps of framebuffer data from one other (abritrary) program, I'm satisfied.

The thing is, I'm a little over my head. I've never worked with low-level languages before, but the idea of reading other programs' memory seems to necessitate one's use, if only for the sake of bypassing restrictions from a compiler. But I wouldn't know how to start looking for framebuffers in memory. (Come to think of it, I've never looked for anything in memory before, at least memory I didn't allocate myself.) There is an assembly language for GPUs by OpenGL.

I've also run across Cg and GLSL in my research, which look awesome for making shaders, but I don't know if they'll help me do this specific task.

Any ideas?

but the idea of reading other programs' memory seems to necessitate one's use, if only for the sake of bypassing restrictions from a compiler.
Not really, you can read other program's memory using documented system APIs on Win32 (ReadProcessMemory/WriteProcessMemory/VirtualAllocEx/VirtualProtectEx...), and on *nix you can do it via ptrace or sometimes by reading some path in /proc/. You will need corresponding priviledges in either OS (Administrator or SYSTEM or just the SeDebug priviledge on Win32, and I think you need root on *nix). So far, this only requires having access to a FFI in a high-level language, or the ability to link or low or generate low-level code.

In a weird OS which doesn't let you do such things, you would require having to write your own driver which runs in kernel mode to do this.

i don't know much about graphics, infact, i know nothing.
but if i was in your situation i would write a kernel module that hooks the device and modifies any data before it hits the real driver.
if you've never done any kernel programming, then lol, good luck.

>>2
priviledge

Figures you're a Win32 programmer.

>>4
You figured it out from use of word privilege? You are an idiot.

>>5
i lol'd

>>6
You were amused by he's statement? Your a gay.

>>2
Of course, that's right. At first I was hesitant to consider this route because I wasn't sure if it would give me access to the "raw" bitmap info I needed, as in a typical framebuffer, but opening specific processes might actually be more useful to me.

>>3
This is clever and in line with the idea of the project. Actually writing or modifying a device driver would be ideal, but of course I can't write a fucking video card driver. I've never done kernel programming, and for the sake of the project I'd like to do this on a windows machine. I'm sure there's a concept of a LKM in Windows but I can't seem to find anything like it; everything I find that's related redirects me to information about programming device drivers.


I'm also wondering about DLL hooking. Maybe I could get something useful out of hooking a process from ddraw or opengl, or maybe gdi, although I don't have much experience with that, either.  Thoughts?

>>8
On Windows a LKM is a device driver. It's probably a misnomer, but that's how it is. The driver is loaded in kernel mode and you can do anything you want with it, but I'm not sure if it's the best idea for your problem.

Hooking ddraw/d3d/opengl/gdi/... is actually more simple and clean/safe than doing it via a driver. There are also some documented ways of grabbing output from other applications, but you will have to provide more information about what kind of application. You won't be able to capture d3d/ogl output via gdi functions.

>>9
I see. Thanks for clarifying.

As to what I would actually like to be grabbing: ideally I would be able to have access to just about everything the computer has been doing to create the graphics it's displaying, all buffers and states.

As a result, I don't have a particular program or type of program in mind as a "target". I'm planning for this program to be run under a typical desktop user environment, which means I probably won't be dealing with many 3D processes -- although those would probably provide the most possibilities for interesting visuals. To that effect, it would be good if I could get something out of the 3D acceleration for the desktop environment from the newer versions of Windows.

>>10
The methods for capturing GDI/D3D/OGL are all different, but mostly well known. Here's an open source Fraps clone which can capture from any of them: http://taksi.sourceforge.net/

>>11
Very helpful. Thanks man.

I've done this before, but in a different way.  Basically, I have a web cam looking at my screen, and just save a picture every 5 seconds.  It's a lot easier, and uses less memory (because you are off-loading it to the camera), so it is more efficient.