This cat won't leave me alone

So I fed this cat one day wandering around my apartment and he followed me right to my door.  I won't let him in but he stayed out there for a while.  He was gone in the morning but when I returned from work, he was back in front of my door again.

Any ideas how to get rid of it without scaring it?