More than anything, I think Larry did the job he was hired to do. He got the team on a much more level financial footing.
But when it came to having Allen's confidence when it came to basketball-related decisions, it seems that he was being tuned out. The hire of Olshey seems clear evidence of that.
Not to mention, the whole Rose Quarter/Memorial Coliseum redevelopment called "Jump Town" - an example of the non-basketball stuff that Miller was probably expected to have a stronger hand in - is going nowhere fast.
Miller wasn't a bad president, but I think he could sense he'd be working against a headwind, having a GM he didn't favor on the basketball side, and little momentum toward progress on the non-basketball side.
He might've also learned that Nike is just an easier place to work than One Center Court, especially when there's only one clique, and you appear to be on the outside of it (same thing that seemed to have doomed Pritchard, Cho, etc.).