I keep hearing this and I want to point out where it started. Twitter, based on a TV station doing lip reading. Their tweet said it looks like the fan said "go back to ____". A college basketball twitter feed then made it "Go back to A-----". Which the TV station said they didn't see. There was nothing concrete about any of this. The TV station even noted it was based on amateur lip reading they were doing.
Also, if this lip reading is from immediately before the shove, it's not what irked Smart because that would be after Smart already quickly shot up off the ground and approached the fan.
The go back to Africa example seems to be the perfect example of how nobody has any idea what was actually said with the exception of a bunch of lip reading that anyone who watches the film is capable of doing. I believe it looks like Smart is complaining to the ref while he's explaining to Ford the call "He called me a (camera blocked). Racist!(while pointing toward Orr)".
But it's all total speculation. The only reason I saw the go back to Africa thing is because I was watching the information get more and more elaborated on the trending topics on Twitter. I can almost guarantee that story is entirely speculation and not the basis of what caused the confrontation.