When you said that Manu should keep his mouth shut. Those three cases (which are much more extreme, or to use your word, inflamatory, than "I'm not a believer") resulted in protecting speech even though it would provoke a backlash. Why would the court do such a thing? because the "bedrock principle underlying the First Amendment is that the expression of an idea should not be prohibited because society finds it offensive or disagreeable." If what he says provokes a backlash -- our law says good because that response to "unpopular views" will result in a debate -- like what's going on here.
That's the point that I think you're missing -- that the "intolerable" view that promotes a "backlash" is really an invitation to a public dialogue - i.e. a good thing. What's bad is when a subject gets silenced completely.