"What I didn't expect was that people would be so conflicted around it," the actor tells EW of fan admiration for his ...