Good point.....Many non-muslims argue that we go on a pilgrimage to Mecca and worship the Kaba'h by circumabulating it and kissing the black stone i.e Hajr-e-aswad, so it is not less than idol worship. But let me clear it.
Kaaba is the Qibla i.e. the direction Muslims face during their prayers. It is important to note that though Muslims face the Kaaba during prayers, they do not worship the Kaaba. Muslims worship and bow to none but Allah(SAT))
It’s best to check with Hindus why they bow before the idols before assuming their intentions are any less noble than yours.