Let me correct you that permission was not given to marry 4 wives in Islam but it is better to say that Islam put the max number of wives to be restricted up to 4. Islam put the max limit upto 4 because before there was no restriction and men were allowed to marry n numbers of women. So each man was having hundreds of wives. Remember Prophet Sulaiman AS son of Prophet Dawoud AS.
On the other hand Islam did not put any limit for slave girls, you can have n numbers of slave girls to fulfill your sexual desire.