There's more than that, IMO, and unfortunately, a lot has to do with him. Things like the Brent Barry incident was very unspurlike, whether you think it was a player move or not, then coming out and saying the Spurs were good, but no longer be contending, basically throwing Tim and Manu under the bus... stopping a bottle with his eye on some new york nightclub... taunting Miami about Green then trying to seize the spotlight... it's not all fair, Tim always has been the franchise and a quiet guy, Manu has that latino connection, but Tony never had that kind of leeway to pull off that kinda stuff... and frankly, the legend of Speedy Claxton predates 2004, tbh...
That's not to say fans don't appreciate his contributions to the franchise. He'll get his jersey retired and inducted in the HoF, deservedly so.