I have a Denon processor, which appears to share its serial protocol and hardware with at least the 5308 and 4308. I can't speak for certain regarding the 3808 since I haven't seen its documentation, but I'd be kind of surprised if it were different from the others.
The Denon is one of the few components I have where I did NOT need to make any kind of adapter for the Nevo serial module. It has a female connector where the Rx and Tx pins are already reversed (from the serial module's pin layout), which is what you need. Pin5 should be ground on both, so you defintely don't want to re-wire that to Pin 1.
My guess is that your commands may not be correct. It's very important for every parameter to be configured correctly. Double-check all those, here's what my Denon uses: Baud = 9600 Data bits = 8 Parity = None Stop Bits = 1 Flow Control = None Format = ASCII Universal delimiter = \r
Be sure to include the universal delimiter. (Put it into the delimiter box, no need to write it after each individual command.) Also, double-check your port assignment in the NC-50 and confirm that it's working. (Should see green lights flash on that port when you send a commend to the 3808.)
So, for example, the Power On command PWON will actually look like PWON/r to the receiver.
If there's still trouble with it, maybe you could link the protocol document and we could go from there.