Use Command key to quickly select object
Pressing the Command key (on Mac), regardless of what tool is currently active, should give you the selection (arrow) cursor so you can quickly select and move objects. This feature is available in Illustrator, and it would make designing more efficient in Xd. Thanks.
Jorge Pinon commented
The only key (I know of) to switch to the select tool is V, which can't be used when you're editing text. So you have to fully switch to the select tool, move the text box, and then click back into text editing. Slow process.