Selecting the artboard in layers panel when it's already selected should not zoom to fit
Sometimes my artboard collapses out of nowhere so I have to double click the artboard name to open it back up to see the layers again. But when I do that, XD zooms me out to see the entire artboard. So I have to zoom back in to find the section I was working on again.
If the artboard you're opening is the one that's already in view, don't perform any zooming actions. (I understand that this feature is to help you find artboards when you need to, but it doesn't help when you're already on it)