Very hard to schedule games during the conference season in any sport. Takes away options to make any NCAA tournament in a sport where a conference tournament gets an auto invite.
Remember when the ultimate bachelor, Warren Beatty decided to get married because of all the risks of not?
It's kind of the same thing with college athletics. Even Notre Dame, the ultimate independent, has formed an alliance with a conference out of their geographic area, because there are more options when you're in a conference. I'd love for Texas to go independent, because it's a very very hard thing to do. Navy was an independent for a long time. Now they're in a conference.
And I believe it would hurt recruiting. Parents of many recruits want some idea of where road games are going to be. That goes away when you go independent.
THe grass on the other side that is greener is usually a spray paint job.