貴方は？could be impolite, but it's grammatically correct and, if the listener is a stranger and doesn't look older, it's not that rude. If you want to be polite, you can use the polite version of the original phrase, e.g.; if the original question was 東京に行ったことはありますか, then you can say, 貴方は行かれたことがありますか？
Things get complicated when you don't know the listener but are positive that you should avoid 貴方 because he/she looks older or you can tell he's a superior for some reason. Of course, you should be polite so you use the polite form of the original question, but the problem is how to refer the second person.
We (I'm Japanese) take advantage of every kind of clue we can get. For example, if you can guess his title, occupation or the like, you can use it to refer to the second person. If you're certain he's a teacher or professor because you're talking on campus and he doesn't look a student, then "先生は行かれたことがありますか？" If you're a shop keeper and you're talking with a customer, you can say, "お客様は..."
There are many other ways to avoid 貴方. If the second person is obviously a married woman, you may say 奥様. If the second person is with a small kid, then he or she is probably a parent so you may use お父様 or お母様. When they're elderly persons and you're relatively young, you may use おじいちゃん or おばあちゃん.
Another typical technique is indirect questions or rewording if you will. If you're asked if you have been to Tokyo and you want to know if he went there, then you might say, いえ、ありませんが、行ってみたいと思っています。お勧めの観光場所はありますか, which means "No, but I'd like to. Do you know some spot I shouldn't miss?" That way you'll probably know if he has. If you don't want to answer his question, then just replying by お勧めの観光場所はありますか works.
That said, the simplest way is asking his or her name right after you answer their question. And then ask the second person the same question.