James Corden has retracted the apology he made to an NYC restaurant owner earlier this week ... 'cause he's now saying he did nothing wrong. 'The Late, Late Show' host sat down with the New York Times Thursday and called allegations "silly" ... he…