An old thread that covers some interesting angles on why not to use an inboard brake system:
Are inboard brakes outlawed ? - Forum - F1technical.net
It would probably be easier, given sufficient budget to do either, to do a more-advanced brake system that's still in the wheel but works well at a smaller diameter. Exotic disc/pad materials, much better airflow/cooling, etc? I bet even engineering a multi-disc system (as in 2x discs in parallel inside each wheel with 4 pad surfaces) would be easier than engineering inboard brakes.