NEW ORLEANS: For three years now, Byron Scott has quietly coached a roster lacking much legitimate NBA talent. The Cavaliers have fielded a team of D-League players many nights, yet never once has the coach publicly complained about the lack of weapons provided.

This was the plan all along, to tear the roster down to the roots and slowly rebuild it through the draft. Scott was expected to nurture the youngsters through the overhaul and stick around long enough to enjoy any success that followed.

Only now, as the Cavs stagger to the finish of another forgettable season, it’s clear various figures within the organization are wondering if Scott is the right man to lead this franchise to prominence.

This team has been ravaged with injuries again this season, but the concerns aren’t related to the roster. They are core beliefs and in-game coaching strategies ranging from play calls to substitution patterns, his refusal to call timeouts at crucial junctures and an overall lack of defense.

They have culminated in the Cavs squandering countless leads this season, including the two biggest collapses in franchise history.

The defense hasn’t improved in three seasons, irritating everyone within an organization that spent years enjoying one of the league’s best defensive minds in Mike Brown.

After a home loss to the Denver Nuggets in early February, when the defense again faltered and a three-game winning streak was washed away, Cavs owner Dan Gilbert wrote on his Twitter account, “We have made good progress recently, but when the Cleveland Cavaliers arrive back to the top tier of the NBA we will be a DEFENSIVE 1st team.”

It was a public proclamation directed to everyone with an office at Quicken Loans Arena — this type of defensive performance will not be tolerated.

Players whisper

Asked directly last week if he has received any indication he could be fired at the end of the season, Scott said, “No.” But as these crushing losses have mounted, players are privately questioning Scott’s guidance. To characterize the Cavs’ locker room as divided or turning on their coach would be inaccurate and melodramatic, but the whispers are growing louder.

“We’re exhausted,” said one player, who spoke on condition of anonymity due to the sensitive nature of the topic. “It goes back to training camp. He killed us in camp. We don’t have any legs left.

“Our shootarounds are an hour and 45 minutes. They’re not physically demanding, but we’re still on our feet all that time. We had a two-hour and 15-minute practice the other day and an hour and a half of that was a hard scrimmage. At this point in the season, that’s crazy.”

Teams hold shootarounds the morning of games. They are intended to serve as a walk-through or strategy session and are typically scheduled to last an hour.

Another Cavs player, who has been around the league a few years, dismissed the notion Scott is too hard on them.

“That’s [expletive]. If we were going to the playoffs, that’d be one thing,” he said. “We’ll have six months to rest. Shut up and play.”

Without knowing players have started complaining, Scott defended his practice habits prior to Friday’s loss to the Philadelphia 76ers. In an anonymous poll of NBA players last April, Scott finished third among coaches players would least like to play for, ranking behind Stan Van Gundy and Scott Skiles. Van Gundy was fired at the end of last season after the Orlando Magic imploded around Dwight Howard and Skiles resigned from the Milwaukee Bucks earlier this season. Cavs training camps, termed “Camp Scott,” are considered among the most grueling camps in the league.

“We’ve had so many injuries that I’ve had to tone down [practices],” Scott said Friday. “But I’m not going to scale back what I do practice-wise because I really do believe it prepares guys to play in this league and play on a consistent basis.”

Against timeouts

Yet even the player who defended Scott’s practice habits agreed he has questioned the coach’s in-game strategies, including his refusal at various points to use timeouts.

As the Miami Heat came roaring back from a 27-point deficit two weeks ago, Scott let his floundering team play on while LeBron James and Shane Battier fired 3-pointer after 3-pointer.

Instead of using a timeout to ice the Heat’s momentum and give his players a mental break, Scott watched the lead vanish. The Cavs led by 18 with 2:51 left in the third quarter and the Heat had the game tied within the first 92 seconds of the fourth. Scott never used a timeout and ended up taking one home with him.

Last week against the Boston Celtics, the Cavs blew a 14-point lead in the game’s final eight minutes. With his team again falling stagnant on offense, finishing the night shooting 1-of-9 with no assists and three turnovers, Scott refused to call a timeout. He ended the game with three left over, although he conceded the next day he should’ve taken one prior to the Cavs’ final possession.

Reached Saturday night by phone, Scott declined to respond to any complaints raised against him.

Tristan Thompson has flourished under Scott and this coaching staff this season, even entering into the conversation for the league’s most improved player despite struggling terribly in the two most recent games. Yet the Cavs rarely run a play designed for him on the low post.

“He’s doing a great job with that push shot he’s developed,” said one Cavs player, who spoke under the condition of anonymity. “And no one will run a play for him.”

Tyler Zeller played well in the first half against both the Celtics and Heat as the Cavs established leads, yet played less than two minutes of the fourth quarter against the Heat and didn’t play at all during the fourth quarter against the Celtics.

Zeller has rarely played in fourth quarters lately as Scott has cited matchup issues and opponents going small. He said following the loss to the Celtics he doesn’t believe Zeller creates a mismatch in the Cavs’ favor because he’s not yet a strong scorer in the post.

When Kyrie Irving was healthy and available, Scott rarely ran a play in end-game situations. He’d simply put the ball in Irving’s hands and tell him to go make a play while the other four players spread the floor around the baseline.

Rather than bring Irving off one or two screens or run some sort of motion play, Scott said he wanted it this way so Irving could see the entire defense in front of him and be able to react to any oncoming double teams. Sometimes it worked brilliantly, other times it did not.

Scott’s use of Irving this season has also raised questions. He typically plays Irving the entire first and third quarters and the second half of the second and fourth quarters. The only time that changes is if Irving falls into foul trouble or if the game is slipping away early in the fourth quarter, in which case he brings Irving back sooner.

“What other star in this league doesn’t play until the final five minutes of the fourth quarter?” one member of the organization asked recently.

Defense struggles

It’s at times difficult to decipher the Cavs’ defensive system. Scott talks about core principles, such as never leaving shooters open in the corners, yet it continues to happen. The Cavs have struggled for three seasons to defend the pick-and-roll, opponents’ top players are often left wide open and the rim always seems to be left exposed any time the Cavs start switching defensively.

Part of those issues can be excused to the team’s youth and inexperience, but not all of it. The Cavs’ defensive numbers are regressing. Opponents shot .475 against them in Scott’s first year, improved that slightly to .467 last season before watching it swell to .477 this season.

In the past three seasons prior to this year, 20 teams have allowed opponents to make at least 47 percent of their shots. Of those 20, only one has made the playoffs. The New York Knicks were the sixth seed in the East in 2011 after opponents shot .472 against them — and the Knicks were swept out of the first round by the Boston Celtics.

There are 11 games left and next season is the final year on Scott’s contract. But all the whispers, the grumbling, the late-game collapses and horrific defense have made it clear: The grand plan might be changing. He has been here for the darkest days, but it’s no longer a certainty Byron Scott will be the coach who leads the Cavs back to prominence.

Jason Lloyd can be reached at Read the Cavs blog at Follow him on Twitter Follow ABJ sports on Facebook at