As you described the mind-numbing possibilities and possible complications of this system, I was reminded of an extremely simple system that used to run statewide in California. Like throughout the house you're talking about, but STATE WIDE!

The members (rooms in your system) were used auto parts yards, some ten to twenty of them, all live (rhymes with"alive") all the time to monitor and to call into. Each location had a push-button mic for announcing into the pool, a speaker for monitoring the pool, and of course interconnection with everyone else.

Apply this concept to your site. Anyone could feed audio into the pool, and amps at all locations would play the instantly mixed audio of all feeds. Experience with the system would teach the users what the volume levels should be for proper operation.

Before proper operation we have how to construct the pool. Here I envision feed amps of some low impedance output, high enough in impedance to keep from attenuating the pool signal too much. Everything from this point on is experimentation to create a system where signal can feed into a pool and that same signal is of a level appropriate to listen to the pool signal.

You may also have to learn how to eliminate local sidetone....