How many great players in NBA history played consistently great on both sides of the ball? Just think of many of the greats over the past 30 years.
Magic
Bird
Kareem
Erving
Barkley
Malone
Stockton
Wilkins
Jordan
Pippen
Olajuwon
Miller
Iverson
Garnett
Kidd
Thomas
Ewing
Robinson
Duncan
Shaq
Dirk
Pierce
Nash
Kobe
Lebron
Wade
The only ones on that list who were known for consistently playing well on both ends is MJ, Pippen, Stockton (overrated defender), Olajuwon, Robinson, Garnett, Kidd, Lebron, and Duncan. That's 9 out of 26 transcendent stars over the past 30 years. Now many of the others showed an ability to step up defensively and be at least decent when needed, but if we are honest, that is basically every single player that plays basketball. Most of the ones who don't play consistent defense, its usually because they are A) saving their energy for offense and end of game defense, B) lazy and don't really care, or C) at a disadvantage athletically.
And just how many players in the NBA right now, are leaders of their team, and also happen to be the teams top defender along with top scoring threat? The only one that comes to my mind is Lebron.
So I think you can overlook his defense a bit. Give him an elite defender like Duncan to play alongside, and perhaps his defense improves? It's not particularly easy to have a strong defensive frontcourt when you play alongside Paul Millsap.