Musical notes have a characteristic volume “envelope”—attack, decay, sustain, and release. The loudest part of the note is near the start, when some initial disturbance causes something to vibrate. The decay of the vibration is at least partly caused by the second law of thermodynamics: the energy of the vibration is being lost to the environment in the form of heat and sound, and the amplitude decreases with the energy.

If you play it in reverse, it sounds like the vibration is being fed energy from an external source and then abruptly cut off.