Taylor Swift's swanky New York City apartment building had an unwanted visitor Friday -- cops arrested a guy who'd traveled from out of town, desperately trying to find her. Law enforcement sources tell TMZ ... Patrick Nissen, a 28-year-old from…