Re: The arrow of Time
Ok I am understating your assumption more here. Lets take your current position as the the fixed point. Everything that is constant (or known) in terms of space/distance is your local frame of reference. That means that whole solar system is your local frame of reference because we knew exactly where Mars was, or Jupiter was on 18th of Jan 1964 with respect to us. So if we knew where it was, then we can time travel to it, as long as we are not leaving our local frame of reference.
But that means that time is a very local phenomena, and it must flow at different speeds in different parts of this universe (which in fact is true, and pretty much what Special Theory of Relativity talks about), and we can only time travel within our own local region. What I am trying to say that from our point of view, time slows down at the edge of a Black hole, so we can never time travel in past to that edge (at least not from our current location).
I have no idea if I am making sense or further confusing myself.