Reduce size of compiled binary

Hey /prog/, I'm making a remote control tool that works over Tor  because reasons.
But Tor itself weights 3Mio and my final executable is over 1Mio upx'd.
I'm compiling in plain C with Visual'12, have activated all the size optimisation but it's not enough. The client must be able to transfer itself over the network very fast and fit even on small embeded systems.

So do you know of any methods to reduce size further, or any packer better than UPX ? I don't care if it makes my code really slow, but size is critical.

Start by figuring out what consumes space.

Yes, this always works for me. If you are using linux, add these arguments to the gcc:

--fvomit-framepointer --gmark-of-the-beast

Recompile the source and when its ready, transfer the compiled binary to a floppy disk. Take the floppy and place it on the floor in the center of your room. Wait for the next full moon, and then surround the floppy with ten lit candles, tracing out a circle of radius 666mm. Once the candles are going, go out into the night and slaughter a goat. Bring the corpse back to your room, and empty its blood unto the floppy drive. Carefully smear the blood to form a pentagram that links the even index candles in the circle. Quickly, before the blood dries, collect a bundle of dry brush and grasp it firmly in your hand. On the stroke of midnight, touch the brush to a candle in the circle to ignite it and hold the brush over the blood-soaked floppy, and begin chanting:

``Eater of redundancy, preserver of entropy, come and feast, feast tonight...''

The goat blood will begin to smoke and melt partially into the floor. The smoke will rise and circle within the room, turning grey, dark green, and then bright orange. At this point, raise your hands into the air, still holding the burning brush, and yell:

``COME FEAST, EATER OF REDUNDANCY. COME AND FEAST TONIGHT!''

Very suddenly, all lights will be extinguished save for the candles in the circle and the smoke will envelope the room. Two piercing red eyes will open on the ceiling and paralyze you. You must quickly fight the paralysis and hold the beheaded goat up to the demon as on offering BEFORE THE MOUTH APPEARS, or you will be devoured instead. His powerful tongue will penetrate the goat. You'll feel it riping the body apart from within, pulling the raw muscles from the bones and sucking the stomach dry. After the initial frenzy, let the demon pull the goat up to its jaws, and after he is satisfied say the words:

``Grand eater of redundancy, I present thee with data. I humbly request you eat the binary, and preserve the data''

The candles and goat blood soaked floppy will begin to hover in the air and tumble slowly in a circle. The evil master's eyes pierce into the floppy, burning its magnetic tape. Wait for the process to complete.

This next step is very important. After eating the redundant data, the red eyes will close in on you. Hold up the severed goat head and the demon (he isn't very smart) will take the goat's body instead of yours. The goat head will become animated and disgruntled, so take it outside and leave it in the street. The birds will pick it over and eventually release the demon so he may return home.

Once the chaos has subsided, pick up the brunt floppy and recover the remaining data. This will form the compressed image that you can release as your final product. The only downside is your customers will have to repeat the ritual to get the demon to vomit the original data for them.

>>1
1. disable inline function
2. disable templates (use something like type-tags)

>>2
I need to static link with LibEvent, ZLib, OpenSSL, some libc and the default Tor implementation (that I run in a thread).
Tor is 3Mio alone, I managed to get down to 1Mio but it's still way too much.

>>3
Now where do I find a goat ?

>>4
Is this even possible with MSVC, or do I need GCC ?
I've already disabled exceptions, i'll try this.
Porting everything to work with GCC is a pain, though. And visual is a pretty good IDE.

Try manually stripping out the fluff, and if that doesn't help, rewrite it in Asm.

Do NOT link with libc statically. Use /MD and link with MSVCRT.DLL. Use CyaSSL instead of OpenSSL, and rip out LibEvent - replace with WaitForMultipleObjects(). Replace zlib with http://code.google.com/p/miniz/ . If you're willing to assume that your client already has zlib/OpenSSL DLLs somewhere on the system, just replace them with code to search for and load them instead.

The optimised switches I use to merge sections/trim BSS/shrink the stub/reduce alignment/etc are only good if the binary is in the <64k range, as they only save a few K at most.

>>6
I can't afford Asm sadly, I don't have the time to rewrite everything, gcc|msvc will have to do.
I need to run on small embedded systems, often x86 linux, so I can't link with msvcrt or use windows-specific apis like WaitFor*Object.
I'll look at CyaSSL and miniz, but "stripping out the fluff" implies rewriting tor and netcode to use light libs, possibly bringing tons of new security issues.

>>3
I have bad experience with --fomit-frame-pointer, gcc used to break my OS during stack switchs, had to use dirty tricks.
Is it worth the 4Bytes/function saved ?

Mio? Is that like 10⁶ IO operations?

>>7
WTF? AFAIK MSVC only outputs PEs, but you're targeting Linux?

--fomit-frame-pointer isn't going to reduce size in the case where you have many live variables on the stack; esp+x is 1 byte larger than ebp+x per access. Try -mpush-args, it stops it from emitting stupid "move to stack" and forces it to use push, like Intel intended.

You can try forcing the compiler to not align everything, my usual method is to compile to Asm only and strip out all the .align directives, then assemble the result. (16-byte(!!) function alignment is absolutely retarded for all but the most speed-critical code on a very small subset of CPUs, most of which are probably not in widespread usage today.)

People crying about "DLL hell" on Windows have it easy. *nix is worse, although in this case I'd bet that most Linux systems are going to have the libs you need already (including libc...)

>>9
WTF? AFAIK MSVC only outputs PEs, but you're targeting Linux?
It's complicated.
I'm used to my Win build environnment with Visual studio.
The Tor dependencies (LibEvent,ZLib,OpenSSL) are compiled with gcc/mingw and linked static.
All the code that is NOT using syscalls directly is built with MSVC, it's the majority of the code I write. Instead this code is calling portable wrapper functions.
The OS-dependant code (for netcode, file IO,..) is built separately with gcc/mingw as a statically-linked lib, it's just a wrapper that calls system APIs with #ifdefs.
Then I plan on cross-compiling the OS-dependant libs and linking with the MSVC code. Plan on, because I'm still testing the code on Win.
Thus I can write most of the code in Visual, and still cross-compile.

>>9
.dso is nearly the same concept as Windows' DLL. Plus, as we both know, he can embed the the libraries in the ELFormat.

>>10
If we knew your target was Windows, I could have offered you this:
http://portableapps.com/development
In particular the AppCompactor.

If you have a Windows build environnment, you will need to a compiler that can do more than PE, so you will need to get MinGW or Cygwin, to build in ELF.

Cudder is correct, that if you want to scale the size of a client to something smaller than 1MB, you will need to rewrite it in ASM since there's bound lots of unnecessary instructions. Assuming you have experience in ASM, You can just take out much of the fluff in the code. I assume it has lots of it since it uses zlib instead of xz. Actually is the only reason zlib is used to compress the program itself, or does it need it for something else in the tor specification?:
https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=tor-spec.txt

glancing over it, it does not seem so, so you can get away with only libev and CyaSSL, and miniz if you need to compress the program files and/or itself. I assume you are making your application semi-Posix compliant, right⸮

Here are all the specifications:
https://www.torproject.org/docs/documentation.html.en#DesignDoc

Warning about windows:
http://support.microsoft.com/kb/149902

>>11
I'm targeting regular desktop windows, linux and embedded linuxoid systems.
I'm not sure what I would use AppCompactor for. I'm already using UPX when possible, I tried other packers but they all give worse results.

I do have experience in ASM. I made a freaking operating system. Thingy.
But rewriting 3Mo worth of opcodes by hand, and without breaking Tor's anonymity and security, seems a bit unrealistic.

I'll probably have to rewrite tor to use cyassl/miniz anyway, and drop the support for relays and bridges since I only need client functionnality. But I'll do it in plain C.

Also gcc has interesting flags, I'd like to switch to mingw-gcc only instead of msvc, but I can't find a suitable IDE.
Code::blocks is shit with a broken autocompletion and weird bugs, Eclipse is slow and not made for C/C++. There is also QtCreator but it's not made for plain C.

because reasons
Visual'12

Back to /g/, _``please_''!

>>13
¶1 It is a specific standard we use to make applications small, and it uses a UPX bin compressor:
http://portableapps.com/apps/utilities/portableapps.com_appcompactor
That's why you need our specifications, since we make them even smaller with other calls. And again, this is only for Windows. But it is fine you are using UPX.

¶2 You are welcome to output the C code to be able to decipher some of the OPcode:
http://panthema.net/2013/0124-GCC-Output-Assembler-Code/
http://www.delorie.com/djgpp/v2faq/faq8_20.html
http://24alpha.wordpress.com/2007/12/18/how-to-get-gcc-to-interleave-assembly-output-with-original-source-code/

And then make a repository for other to take a look. I am interested in the project, as you can tell.

¶3 That's fine. It shouldn't be that hard, since the call the SSL wrapper functions separate as modules:
https://gitweb.torproject.org/tor.git/blob/refs/heads/master:/src/common/crypto.c
https://gitweb.torproject.org/tor.git/blob/refs/heads/master:/src/common/crypto.h

  27 #include <openssl/err.h>
  28 #include <openssl/rsa.h>
  29 #include <openssl/pem.h>
  30 #include <openssl/evp.h>
  31 #include <openssl/engine.h>
  32 #include <openssl/rand.h>
  33 #include <openssl/opensslv.h>
  34 #include <openssl/bn.h>
  35 #include <openssl/dh.h>
  36 #include <openssl/conf.h>
  37 #include <openssl/hmac.h>
and libev for the network events:
https://gitweb.torproject.org/tor.git/blob/refs/heads/master:/src/common/compat_libevent.c
https://gitweb.torproject.org/tor.git/blob/refs/heads/master:/src/common/compat_libevent.h

¶4 IDE? Why do you need one? Is gvim, emacs, heck even notepadd++, with gdb and Valgrind not enough? If you are mentally handicapped for one, use:
http://www.bloodshed.net/dev/devcpp.html
http://wxdsgn.sourceforge.net/

The only one I used (thanks to stupid professor) was Geany. You are welcome to use that. There are more:
http://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments#C.2FC.2B.2B

>>1
Hey /prog/, I'm making a remote control tool that works over Tor  because reasons.
Do you like to have sex standing a hammock?
All the code that is NOT using syscalls directly is built with MSVC, it's the majority of the code I write. Instead this code is calling portable wrapper functions.
The OS-dependant code (for netcode, file IO,..) is built separately with gcc/mingw as a statically-linked lib, it's just a wrapper that calls system APIs with #ifdefs.
... ok, wrong question. Do you code that standing in a hammock?

OP, can you use?:
https://www.cypherpunk.at/onioncat/
https://www.cypherpunk.at/ocat/download/

>>15
>output the C code to be able to decipher some of the OPcode
And why would I want to do this ?
Trying to maintain 3Mo of compiler-generated, uncommented, assembly would be batshit crazy.
As I said I'm going to modify the tor implementation, but I'll make it in plain C.

I'm going to look into porting to smaller libs later. First thing is removing the useless functionnalities like bridge/relay, hidden service and the fuckton of runtime options I don't need.

>IDE? Why do you need one?
Well there's a reason why people use them. Having stuff like autocompletion, IntelliSense,"Jump to definition/declaration", or just the formating of compiler output saves a lot of time.
I'll try Geany, if not I'll probably try C::B again.

>And then make a repository for other to take a look. I am interested in the project, as you can tell.
Well, it's a remote control tool, (probably) no one's going to decompile it and study the assembly. But giving the source code is just calling for someone to exploit it. I might want to trust security by obscurity this time.

>>17
I'm kindof doing the same thing, but I need less functionalities, for example I don't want hidden services on the clients. Onioncat depends on Tor, so it's bigger than what I already have.
I'm just running a custom Tor implementation in a thread and sending data through the SOCKS proxy, I don't really need this.
Thanks for the suggestion anyway.

No compiler flags are going to make an appreciable difference, and suggesting going directly to assembler is true /prog/ level of stupid. Remove all unneeded functionality from your libraries, remove libraries in favour of OS-provided functionality altogether. Give up your dreams of cross-platform compatibility. Refactor your application into an OS-independent core library and implement the OS-dependent functionality in an OS-dependent way rather than saddling yourself with abstraction layers and compatibility shims.

>>18
¶1 Just saying it can be done. Is not like you are reading MenuetOS.

¶2 k

¶3 Because gvim, emacs, and Notepad++ with gbd and Valgrind do not offer that... Ok.

¶4 Ok, you are officially insane. An IPsec anycast protocol is better suited for anything remote controlled than tor: "Let me see if I can find my one grain of rice in this miller machine after I dropped it, and make an cereal with it only"

>>19
Then:
https://trac.torproject.org/projects/tor/wiki/doc/SupportPrograms

>>20

>No compiler flags are going to make an appreciable difference
I need every Kio I can get. Off course there's a limit, but MSVC with no optimisation/fast compilation makes a 4.7Mio exe. With size optimisations it gets down to 2618Kio, and with UPX ultra-brute down to 894Kio.

>>Remove all unneeded functionality from your libraries
I'm doing this right now. But this is dangerous waters. Just removing unreferenced functions is the job of the linker. But removing a functionnality is not so easy when you have hundreds of interdependant files. Thankfully Tor is kinda-clean c. But it's still a pain.

>> remove libraries in favour of OS-provided functionality
I'm targetting embedded systems too. With embedded meaning "We removed all the OS-provided functionnality we could" it's a bit tricky.

>Refactor your application into an OS-independent core library ... and OS-dependent functionality in an OS-dependent way
>rather than ... abstraction layers
Please explain the difference between an OS abstraction layer and separating OS-independant and OS-dependant code. I'm not sure what you're trying to say here. I already separate the OS-independant code -- the one I compiled with msvc -- into a lib.

>>21
>gvim, emacs, and Notepad++ with gbd and Valgrind
That's 6 different tools. And I'm counting emacs as only one tool here.
That's why people made IDE. A consistent, ready to use, gui to the underlying stuff. Of course I could bake a linux build environnment with a full-custom emacs. But that's not what I'm trying to do.

>anycast
>ipsec
Seriously ? If it was this simple, do you really think people like dread pirate roberts would bother with Tor ?
I need both encryption and anonymity of all the clients and servers. The Tor routing model is exactly what I want. Routing my encrypted traffic through a circuit of randomly picked anons. No one knows what I'm sending, and no one can know the source and the hidden service's rendezvous point at the same time.

>>22
Instead of having an application sit on top of a compatibility layer you have a native application calling into a library.

>>22
Instead of having a /g/tard sit on top of your thick head you have to heed their calling back to Reddit

DA BEST GAIM DEEZIGN IS MY SUBREDT

CAUSE DEY B-LONG 2 A TOTAL PLAYA

DONT B PLAYA HATIN U

>>9
People crying about "DLL hell" on Windows have it easy. *nix is worse
Counterpoint: *nix at least has the cultural expectation that libraries be versioned using sonames. Compare this to Windows, where DLL name collisions are common... you might tend to see more missing dependencies on *nix, but that's much more tractable than the conflicting dependencies Windows' lack of convention encourages.

>>23
Okay.

I've started removing useless crap from the code. I'm <500ko atm.
Thanks.