It depends what you expect him to be. If you think he's going to be an all-star and see some of the fundamental issues in his game which haven't measurably improved, then absolutely be concerned. If you think he should be a good player who might become all-star level, then it's still probably two years too early to push the panic button.
Being focused on the plus-minus is a mistake. Like any statistic, it can be misleading. Shaedon's game is such that it can really look bad in plus-minus, especially with, as you noted, him playing with a Blazer bench right now that is comprised of G-Leaguers and 21-year-olds. Also, he's coming in off the bench, but he's playing starter minutes. He's not playing against other team's second and third stringers most of the night.
If you want to look at stats and analysis that are more indicative, look at his turnover percentage and usage rate, look at his free throw attempts and look at his 3-point shooting percentage and his percentage of shots that come from 3. His biggest problems, in my view, are that he settles for bad shots and he passes up good shots to create more difficult shots. He thinks he's a better 3-point shooter than he is. There are reasons he gets so many open 3s and it isn't just because he added a nice step-back J.
Also, when he misses, he misses in bunches. When his shot is off, he needs to get to the line and get himself together, which, he has done a good job at doing.
I'm a little concerned with the number of shots he's had blocked this year, but I'm encouraged that since coming back from injury he's been attacking the rim with vicious intent. He needs to be more like Sidy in that regard and it will help him get to the line more. Also, his FT shooting and form indicates he should be a better 3-point shooter than he is. I've said before I see things in his game that only make sense to me as a player that either doesn't play much when he's off the clock or a player that doesn't have a really good training regimen.
When I was in college, one of my classmates was in the gym at 6 p.m. every day shooting 1,000 shots on his own from spots and off moves he'd get in a real game. He did things to simulate passing coming from angles where he'd receive them in a game. He got drafted and played a couple of years with the Mavs before having a long career in Europe.
Shaedon's problems to me are fundamental and could be fixed just with some hard work and probably proper mentorship. His handle is poor for a guard. I've written this a lot. He is too loose and imprecise with his handle dribbling in traffic in the halfcourt. He needs to be able to change up his high and low dribbles and angles of attack. Otherwise, he's always going to bet the ball knocked away. And, since I think his best shot is the midrange, an inability to handle in traffic is a detriment.
That could be solved by playing with a firm objective and in competitive situations in the offseason. I watch his game and I get the feeling he doesn't do that to the extent really good players do. That's when you develop your consistency and your feel for the game, what shots are good and the space you need for a particular move or shot.
Defensively, still a lot of work to do. He looks fine some nights and some possessions. Other times, like on offense, he doesn't look mentally or emotionally engaged. He's a good enough defender when he wants to be, but, like the other question marks in his game, a lot of the time it doesn't look like he really wants to be. He looks satisfied with being a good player and not one particularly driven to take advantage of his immense innate skills to become great.
TL;DR It's too early to worry yet, but Shaedon needs more passion for the game and to work out more effectively in the offseason.