first of all the world isn't going to end if you wear the wrong bag lol chill out.
I think bigger bags are great in the day time, but when you go to other people's homes if you have the time to change to a smaller bag it's nice because we've all been at a gathering and had tons of womens giant bags to step over on the floors and such. Wristlets and clutches are great but also work in the daytime. If i know i'm going to a crowded restaurant or going to a concert in the park i'd just rather have a smaller bag just so that i'm not smackig people with it as i walk by.
also if you do want to bring a bigger bag to an evening occasion just go for a darker color due to it being an evening thing.