google sticky remote and you will find this is a fairly common problem. It seems some had success with 70% isopropyl alcohol. But, it also looks like in some cases the rubber texture/satin finish is a coating and the alcohol may remove it too.
Civil War reenactment is LARPing for people with no imagination.