Libertarianism is based on the principle of non-aggression. A libertarian state would establish non-aggression among it's citizens. It would protect the life, liberty and property of each citizen from violation by others, including the state itself. And, it would do no more. It would not regulate consensual behavior in any way.
Hence, people who follow sharia voluntarily will be able to follow all the regulations of sharia of their own free will and to the fullest extent. They will have complete freedom of association and can form their own exclusive communities, if they so wish.
It does also mean, however, that if they interpret any regulation of sharia in a manner that violates the principle of non-aggression then that would not be allowed. For example, no violent punishments for blasphemy, apostasy or heresy can be imposed.
Muslim libertarians believe that such an Islamic community will be more truly Islamic than a community where sharia regulations are enforced by the state.
"There is no compulsion in religion" Quran 2:256