Exactly. This is a huge factor, probably the largest (no pun intended). In fact, in this sense, the roster is upside-down. It's small at every position except center and at center they have a huge lumbering man when that's really not what you want anymore. A correctly constructed roster would have taller players at nearly every position and a lighter, more fluid player at center.
Even if the team has a few isolated good defenders on the roster, the Blazers are poorly constructed overall to defend and, in addition, they also have awful defenders who have to be key reserves (because they have no real talent beyond them). It's not Stotts' fault for playing Melo and Kanter--the roster forces him to play them significant minutes. You can complain about playing them together, sure, but it's not clear whether it's worse to play them together or to separate their minutes and essentially ensure one of them is on the court at all times sabotaging the defense.
I don't think Stotts is a great coach, but I also don't think Popovich or Thibodeau would be getting significantly better defense out of this roster. When it comes to defense, it's broken at the construction level.