In earliest sources ahlulbayt are always banu abdul muttalib , the blood relations of Prophet.
But later as Alid and Abbassid factions fell apart after the successful revolution against ummayyads each claimed itself to be ahlulbayt alone. And since 12er school is concerned with imams of Alid descent they only include the Five as ahlulbayt and no one else.
You are not confused as u selectively read history take what u like and leave the rest.Quite an easy position to take i must say !