Factory classes

Can somebody explain what are factory classes good for? As in, "why all this shit"? I have the impression factory classes are yet another suit in the Emperor's wardrobe. To me, they always looked like some bored programmer who needs to keep his job and hates the KISS principle decided to make our lives worse.

Then you're forced to do a lot of object juggling, which is inefficient; rely even more on the garbage collector, which may be inefficient and may fragment and/or waste your memory
Wrong, as I explained this is why factories exist; to prevent instantiating new objects unneccessarily.

have to create and/or use even more classes and methods, which affects your productivity; and makes your code even more insane and hard to maintain.
True. Performance hacks always make you pay for speeding the program by slowing down the programmer. Rarely worth it these days. Compilers should be responsible for any and all optimization, making the programmer do it by hand is Bad and Wrong.