It’s always the “haram police” who are spreading such things, however there is no legal rule which tells us to do so. It’s just like with reading the Quran, some will tell you that you have to do wudhu before reciting the Quran, however it’s not necessary. You can do it If you want, but it’s not prohibited for us to recite the Quran without doing wudhu.
So yeah, If you want to be extra careful about it, go ahead.