It means, you dont like the girls only on two occasions
1- When it's raining 2- When its not raining.