Can you confirm that if you are in one of your activities that use the TV and you switch everything off with the off button, then select a *different* activity that includes the TV with a *different input* that everything works OK? [Note it is important that you turn everything off and not just switch activities in this experiment]

If the above isn't working correctly, then it will most likely be (as jlet suggested) that the problem is the TV "Power On" delay. This would explain why switching from an audio activity to a TV activity exhibits this problem .... as it involves turning the TV on; it isn't the turning off of the TV that could be the problem but the turning on.

If the above does work then I'm a bit mystified as to what the problem could be .... I think you might need to explain what options/commands you used to fix your input selection problem that you initially had. Maybe this would shed some light on the problem.