It's the Fletcher-Munson curve - at higher volume levels we perceive treble & bass as being more prominent. Louder often seems harsher.
That's the reason behind the Loudness button on old stereos, to boost bass & treble for low-level listening.
Another possible contributing factor is beaminess in speakers. At high volumes we're often further out in front of our amps, where we hear more of the treble that shoots straight forward rather than spreading out the way lower frequencies do. This is why mic'd sound to tape or PA often sounds quite different from the way we normally hear our own amp.
It can be pretty unsettling. One hack that can help a lot is to dial in your tones at full gig volume, while standing at least eight or ten feet away out in front - or with the amp up at ear level, or at the very least tilted up aiming at your ears.
That way you'll get a much more realistic idea of what your amp really sounds like to the microphone and the audience.
I always used to put my cabinet up on its roadcase if I couldn't stand at least ten feet away.
Of course, the breakup of some amps is different when they're driven hard. Some really are literally harsher in tone.
But IME other factors have been to blame in most cases.