Sorry I misread your post.. :o
I was replying in the context of “egg donation”.. what you mentioned is different from the OPs question and it’s permissible. more info here in light of Quran and Sunnah.
Again, like you said so long as the egg/sperm is from the married couple, there’s no harm. If a third party is involved that’s where there’s a clear line.