Sensitivity to delay varies between individuals and can be enhanced with training (experience). The annoyance factor also depends on the material. Intimate percussive music would suffer more than playback of a recording of a live event.

In a college party we inserted a deliberate tape delay of a few hundred milliseconds between two rooms and no one seemed to notice. Perhaps it was the "brownies" that some were consuming, but that much delay would have driven me crazy if it had been permanent.

Carefully managed latency can be constructive, as in Ernie's church above. Recently, I set up a live outdoor sound reinforcement system where the stage was 70 feet from the audience. We couldn't wire to the stage. I deliberately picked a wireless relay that included a 70ms latency. Alignment was perfect. Without the latency there would have been a discomforting collision between the acoustically delayed direct sound from the stage and the nearly instant arrival of the amplified sound.

The 70ms latency was a lucky coincidence in this case. Otherwise I would have used a signal processor. Modern signal processors include delays, equalization, compression, etc, and make all of this very easy. In earlier times, at one "box" per effect, including all of this processing would be an expensive, difficult to manage mess.