If the CD player has a door and a tray that slides out, it may be possible to damage the door and tray. If it's a more modern unit where only the disc slides out, damage is unlikely unless the phone holder's fingers reach far enough back to engage the rollers.The optics will be centered on the disc hub and usually slide side to side. I doubt that the phone contraption will reach back far enough to risk damaging the optics.

I don't play CD's in the car. I bought the car used and never tested the CD player. There a jukebox in the trunk that annoys me and at some point I'll remove it.