If you leave the gall there it will just ruin all of the MT tool shanks ... actually already has caused some damage to all of them and for absolute certain you won't get perfect alignment with that gall ring there. Morse taper reamers are hand tools and not meant to be chucked up into any power tool unless you know for certain that you have one that is. There are two different Morse taper hand reamers that you will find ... one is for removing large burs and the other is for light touch up work. Generally, you will only need the the latter. The purpose of the reamer is to only remove any material that is proud of the surface. Any grooves aren't a problem.
To use a MT hand reamer, first give it a good coating of lightweight machine oil (3-in-1 is good) and then while slowly twisting it clockwise gently insert it into the socket until it engages the burs. Give it about a half turn and then remove it and clean off all of the swarf. Also use a paper towel to clean out the spindle socket. Re-oil it and repeat until all of the bad material has been removed. There is no need to force the tool into the spindle just far enough to make solid contact. When satisfied that all of the burs have been removed, check with a flashlight. You will still see a groove, but no bur. You could stop at this point, but I like to polish the socket using a
SpinLMate tool.
The reason that the drill chuck is loose as Odie indicated is there is usually a a flat at the back end of those adapters which fits into a keyed slot on drill press quills. This keeps the tapered shank from slipping and also there is a slot in the drill press quill quill where a knockout tool is inserted to remove the chuck and MT shank. Some of the new woodworking drill presses do not have this feature, but instead just use a chuck that screws onto a threaded shank. I have used my bench grinder to remove this flat "key". I don't know what the deal might be with your buffing MT shank. Mine must be an older design that uses a threaded bolt that goes into a drill chuck.