Just go , chit chat , eat, come back you do not have to do pooja paat , you participation is not going to add to their emaan on their religion or attestation of their religion, neither will it weaken your emaan.
I agree with Mirch. Go eat, chat whatever come home. I have been offered 'blessed' bombay mix from the local hindu temple.... just because I ate blessed bombay mix from a sadu doesnt mean that I am now a Hindu or committed sin. Islam allows us and enourages us to live in peace with other religions. We expect them to respect our belief and join our celebrations, why can they not do the same? There are many Indian Muslim who happily enjoy holi, simply because its fun.
In my opinion :Its better to avoid such gatherings.
You are Not going to gain anything if you attend it & you are Not going to lose anything if you Do not attending it.
If the family is Hindu, then whatever you decide be cordial. They may not understand a reufsal, or if you go they may not understand trepidation and participating in certain rituals.
If it's a Muslim family, did they at least invite you over for Eid?