When hovering and clicking, Inspect selects the parent group by default. To select a nested group or layer, double-click the layer within that group. To access the most deeply nested layer, hold the Command key (Control for Windows) while hovering and clicking.
Selecting a layer on the artboard will also select that layer in the left-hand layers panel, and vice versa.
To deselect a layer, click the canvas outside the artboard or press the Escape key.