I think a lot of what it takes to do something musical, whether singing or playing, is so different from things like public speaking or acting.
Of course, with acting there is the challenge to know your lines and to act convincingly. That is a sizable hurdle, indeed.
However, music has some more exacting requirements. First, you have to deliver a note on pitch. If singing, you're not singing in your natural voice. You have to hit a specific pitch or you'll be sharp or flat. If playing, you have to the same thing - hit the correct note or notes. Then there is timing. In speaking or acting, timing the delivery is important, but there isn't the same type of unforgiving requirement that you have in music to play or sing a specific note or chord at a specific time.
Of course, different parts are more challenging than others. Slower parts with a less obvious beat are actually more difficult to get right.
So, since there is more to music that is fundamentally less natural and requires more precision, I think it can make one feel more anxious.