What IF: time travel

People say you cant go back in time because if you can then we would see space travelers traveling back in time to visit us. Well... what if.. WHAT IF: we didn't travel back to the past because we didn't want to change the past which would later change the future. So in the future the scientists mke an agreement that they'll never travel into the past because they do not want to disturb anything.

Time travel is really interesting since it can create many paradoxes. Mathematically unable to be proven however is the ability for physical matter to disappear from the universe. Since the universe is a constant, at all times, being able to remove and add matter to it via time travel would mean that all our scientific and mathematic truths would have to be reevaluated.