The TV antenna signals, channels 2 - 69, are in the 54 - 860 MHz range, and the dish signals are usually in the range of 950 -1650 (I think) MHZ. The ability of one device to respond to frequencies outside of these ranges should not affect your system at all, so it looks like your setup should be okay with the original parts.

It is to be expected that the antenna input of the multiswitch does not pass DC. DC at specific voltages is required from the satellite receivers up to the dish, so if the antenna signal is mixed with those signals, it can't mix any DC. That's why the preamp's power supply has to go upstream from the multiswitch.

Check the signal coming out of the multiswitch to see if both halves of it are okay by NOT using a diplexer and:
a) connecting it to the satellite input, checking for all transponders, then
b) connecting to the UHF/VHF input and checking for digital TV stations.

If all is well with that cable when connected independently to each half, then the problem is in the diplexer.

The BMS 5-8 has its own power supply, right?

By the way, I have seen lots of such problems and this is a new one on me, so I am trying to gather more information.