Ok, so I can get neural nets to converge to solve a training set perfectly, but when I put other values, it always gives weird as fuck answers. In other words, my neural nets don't generalize (which is the point of AI), even if the training set is fuckhuge. How do I make them generalize? Some site on the internet told me to make the net smaller, but that just causes it to not even be able to solve the training set and become stuck on a local (nonzero) minimum.
I don't see how neural nets are any better than just making a program that creates random programs and changes them slightly until the result is appropriate, they don't generalize jack shit, it's the same as just returning
if(input==challenge1)return answer1;
if(input==challenge2)return answer2;
etc.
Yes, clearly when ancient, well-tested algorithms don't behave as documented, the problem is with the algorithms and not with your implementation.
Name:
Anonymous2012-08-30 16:44
>>2
Oops, I indeed made a mistake (I added some values to correct the weights, plus I didn't realize it's the activated, weighted neurons that summed and not their weights.)
Anyway, now I can get them to generalize (somewhat) but it takes insanely long to converge. If I make the training set size large enough for it to generalize, the error seems to decrease exponentially when training (means it takes way too long) and I can see it getting closer and closer to the solution, but it's too slow! If I decrease the training set size, it converges quick as fuck but doesn't generalize. I could be still doing something wrong, but I just want to ask does good generalization mean slow convergence?
You said "perfectly." Errors in the code aside, it might be that you're overtraining your net, so that it always gets the right answer for the training set but is actually crazy. What you want to do is split your "training" set 80/10/10, where 80% is used for actual training, 10 is used to see how well the trained data is working, and 10 is used to see the "real" performance of your net. Because the first 10 is of course going to affect the training as well. Got it? There was an awesome free online course on Machine Learning that covered this from Stanford, taught by a guy with more communication skills than I have. Anyway, google "overtraining" and you'll find something. I think I need to google "overuse of quotes" now.
Name:
Anonymous2012-08-30 22:18
one word, the forced over parameterization of the model. sample overfitted
In most ANNs, backprop training retrains everything, so you'll have to teach it all skills at once in a sense. There's a few techniques to get past that (though my favourite is: fuck backprop.) If you want to stick with backprop you'll have to build the net in such a way that the training sets are accumulated and enforced in the network.
Also what >>9 said about overtraining. Don't do that. If your results are all coming up 1.0 you might as well just write them down because that's how useful your net is now.