When an object or group is selected in the layers panel, but positioned underneath lots of objects, you should be able to hold a key (such as Cmd or Control) so that you can drag it around without XD selecting a higher layer that is above it. Sketch allows you to hold a key so that the selected item stays selected.

Yes you can use the arrow keys to nudge something and nudge it all the way to an empty space on your artboard, but that can take forever if you want to move your object way down the page. This is really slowing down my workflow. In the link below, you'll see this in action.

https://www.loom.com/share/bae198ec89734708b88ee2d6203b4968?from_recorder=1

It would be really useful to be able to select a layer in the Layers palette (or by clicking in the artboard), then move that selected object from within the artboard by dragging it (as opposed to the top most object being selected). The object on the selected layer should always remain selected so long as you drag from within its bounding box. This could be enabled by holding down a key, but I think an option to make it the default behaviour makes sense (as it is in Figma).