The obvious would be one of the REAL remote solutions that are out there. RTI, URC, either one would work. Depending on the size of the job, you could even put multiple RF receivers in and utilize your existing IR hard wires. The IR reception in the rooms would be irrelevant.

With the right control processor, you could even control the Matrix switch via RS232 if the IR commands of the matrix switch are more of a problem than you think...