CJ is abused all the time defensively, and his length and size is one reason. IIRC, his wingspan is about 1.5 inches less than Dame's
I mentioned Toronto but they aren't "far smaller"; Lowry, Powell, and Van Vleet are all very physical players for their size, and size is part of the equation. The Kings have 6'4 Heild, and 6'6 Bogdanovic has replaced Heild lately as a starter. Utah runs 6'7 Ingles at SG a lot of the time. None of the Clippers top-5 lineups have Beverly and Williams, and of course, Beverly is a very good defender so it's not quite the same thing. They also have 6'5 Shamet, and Harkless spends 41% of his minutes at SG if bbref is to be believed
the main thing though is that Dame and CJ have the same defensive deficiencies (as well as some offensive redundancies). Dame/Matthews was a better pairing on both ends of the floor because they didn't have so many overlapping weaknesses and weren't as redundant on offense.