We have ruled India for 1000 years

And I would contend that some random Afghan whose ancestors stayed put in their village in Afghanistan and never had anything to do with India has far less of claim in this situation than the Punjab/Bhopal/UP/Bihar/whatever-based descendants of Afghani immigrants to the Subcontinent.