Beagle
Ambassador of Slippin' and a Slidin'
One can of this will last a lifetime, and it's nonabrasive.View attachment 53796

This is still going from the 1970s
One can of this will last a lifetime, and it's nonabrasive.View attachment 53796

Ramo, anyone can do anything, with the right mindset & equipment.
If the inlay is completely loose, I'd remove it, scrape the bottom of the inlay & the cavity is sits in, and reset it with superglue, or even clear epoxy if there are gaps to be filled.
If it just seems like it's slightly loose, you may be able to gently lift the loose edge & work some slow set superglue into the void and then hit it with an activator, like "Glue Boost.
Fine Wood Glues, Fills and Matched Repairs | GluBoost
Unlock the potential of your woodworking projects with GluBoost's leading Fine Wood Glues, Fills & Matched Repairs. Click to explore our Glues, Tints, Fills and our GluDry Accelerator. Available Nationwide.gluboost.com
Then after all is set up, if the inlay is smooth & you can feel nothing, your done.
If the inlay protrudes, even slightly, it'll need to be scraped flush with the board using a single edge razor blade.
To clarify I only use the sandpaper during a fret level to get out the file scratches after crowning.I never use any oil or wire wool near my guitars. I use 320 and 400 to level the frets, it far too coarse for polishing them. My fretboards never get dirty.