You control this by setting up your TV so it knows how to select inputs properly, then you setup the activity by specifying the input that you need on the TV.

My guess is (assuming you have the correct input selected for the TV for that activity) that you told it you have a discrete input selection but gave it the toggle command. When you enter the activity the harmony thinks the input command needed for the TV is a discrete command and resends it (just to be safe), but it's really a toggle which changes the input. If the harmony thought it was a toggle it wouldn't resend it. Adjust the input setting for the TV and that should fix it.