I've been using XD with a wacom tablet (Intuos 5) for several weeks and I have the same problem as with other design software: the pen input is too sensitive to movement!
Often times I click with the pen to select an object and I accidentally move it at the same time (a few pixels) which is really annoying and sometimes I don't even notice it if I've very zoomed out.
In Paint Tool SAI, which ironic is not even a design tool, there's an intentional lag snap when dragging stuff so you don't move things accidentally. Maybe something like this could be useful in design software as well specially in XD where precision is key!