Not weird, a perfectly normal function on Harmonys.
Use the Mute command, or the series of commands to take you to the TV's menu option to turn off the internal speakers, or - if you really can't use those - the series of Volume commands.
Either way include the command or series of commands as Additional When Activity Starts.
You will probably want to add the command or the reverse series When Activity Ends.