The first blackmailers were Scottish landlords who exploited farmers by making them pay rent in livestock or services if they couldn't pay in cash. The goods they had to hand over were usually worth more than the rent owed, and the landlords didn't make change.
Around the same time, local chieftains started going after the same farmers with the kind of scheme the mafia usually refers to as "selling insurance." They made an offer the farmers couldn't refuse: pay a fee for protection. If the farmers didn't pay, then the chieftains would unfortunately be unable to prevent ruffians from destroying crops and sacking property. The Scottish farmers called both nefarious deals "black" because they associated that color
with evil, and because both payments were made in goods rather than silver coins (called "white money"). As for the "mail" part, it doesn't refer to the postal system. That "mail" comes from the German word for "pouch." The "mail" in blackmail is related to the Old Norse word for "payment" or "agreement."
This explanation originally appeared in the "25 Most Important Questions in the History of the Universe" issue of mental_floss magazine.