I think the disconnect here is that he doesn't really play as a PF. On offense, he's strictly a perimeter guy. He plays basically as a tall guard. Among perimeter players, shooting 47%/35% is pedestrian at best. There's nothing 'special' about a PF that plays like a guard and makes average percentages compared to any other guard. Defensively, he's the weakest of your 'bigs', period. He can't guard any physical big and his shot blocking is strictly limited to weak-side help.
You're conflating two completely different things here: man defense is the ability to defend your own guy. Help defense is the ability to go help when one of your teammates defense brakes down. His man defense is poor by any standards. He doesn't have the build to play the physical bigs in the league. His shot-blocking in face up situations is also mediocre. Excellent defenders like Chandler, Duncan, fat Gasol, they all can face up and block shots, anchor the defense, and tackle the physical bigs. His help defense and weak side shot blocking is great, but its obviously reliant on guys like Perkins and Sefolosha softening the attacking player. He certainly has great timing and athleticism, and he takes pride on doing that kind of work, so he should be commended for that.