Right now, when you open up the color picker, the "HEX" box is highlighted. I get it, it's so you can manually type a value or copy the existing value easily.

However, once you have selected a new color from a saved swatch, I think the automatic highlighting of the hex should be removed. That way, you could close the color menu by pressing "V" and returning to the regular "move" tool. Right now, I constantly type "V" to try and close the picker, but I end up typing "V"s in the hex box instead.

Honestly, if there was an option in "Preferences" that allowed us to close the color picker automatically after you clicked on one of the saved swatches, I'd most likely use it. Most of my color changes are pretty confident once I have my saved swatches.