The metal ones stay tight. The plastic ones get wallowed out over time. Swapping out requires a partial disassembly of the trigger group.
I've found that removing/replacing the flat detent spring by itself makes the job easier.
Remove grips, making sure the grip safety doesn't get away from you.
Remove flat detent spring by sliding up and out (sometimes a small flat-bladed screwdriver will help in getting the spring out of the detent notch). This should give you enough play in the selector lever to get the button slid down and off, and back on.
Replace selector detent spring. Make sure grip safety is in place and fully depressed, reinstall grips.