When you set up the activity you tell it which device you use to control the volume and that's what should initially be mapped to the volume buttons in Watch TV activity. Check the button customization for Watch TV activity and you can set the volume buttons to control whichever device you want. Each activity can be different.