Wow - you're right; the ref encouraged BU to have another look. From ESPN.com:
After that series, Brewer was briefly evaluated by the Baylor training staff, but he returned for another offensive series. After that possession, head referee Mike Defee asked coach Matt Rhule to have his training staff again examine Brewer.
"I didn't know anything," Rhule said after the game. "Mike Defee came over and said 'Coach, can you please look at Charlie? He just seems a little bit out of it.' Now, I had been talking to Charlie on the sidelines [before] and he didn't seem out of it to me, but I have a lot of respect for Mike Defee. And he said that, so I called over Matt [Kuehl, team trainer] and said 'Hey, they're saying on the field that they were concerned about Charlie.'"
During the second sideline examination, Rhule said the doctor "looked at him and ruled him out." Baylor then sent Brewer to the locker room for further evaluation.
At halftime, Rhule told ESPN's Holly Rowe they decided to keep Brewer out to "err on the side of caution."