Assuming this system is here to stay, here is a way it improve it.
Make me press the glove button, facing the color I want. Running into them makes it very hard to actually get the outfit I want. If you change it to a glove press then it will be easy and effortless to go in and quickly change my colors, instead of having to play minefield with the colors I don't want.