Time travel is impossible

If causality can't be broken then time travel is impossible.

Wouldn't time travel break the conservation of energy?I mean,when you went to the past for example,you would pop out of nowhere,and you would carry all of your energy to the past.