At 5 feet 4 inches, Madison was America’s shortest commander-in-chief—but he left behind a towering legacy. To honor his 264th birthday, we’ve dug up some lesser-known details about this “Father of the U.S. Constitution” and the colorful life he led. Did you know...

1. That He Was the Oldest of Twelve Children And Prone To Sickness?

Madison wasn’t the healthiest kid—or adult for that matter. A frequent victim of nasty stomach aches, he’d also weather attacks “somewhat resembling epilepsy” throughout his life.

2. That Madison Fought—And Won—an Oratory Duel With Patrick Henry?

This was no small feat. Throughout their home state of Virginia, Henry was renowned as a public speaking heavyweight who wooed crowds wherever he went. Thomas Jefferson declared that the booming debater spoke “as Homer wrote.” Madison, in contrast, had a quiet voice and little talent for theatrics.

In 1788, a ratifying convention was held in Richmond to determine if Virginia would grant her approval to the Constitution Madison had helped engineer. Henry staunchly opposed this document, and loudly decried it for leaning “towards monarchy.” But though several people in attendance complained of Madison’s mumbling, he offered concise, well-articulated rebuttals to every argument. When the dust settled, his slow and steady approach paid off big-time: Virginia’s representatives voted 89 to 79 in favor of ratification.

3. That He Competed With James Monroe for a Seat in the Newly-Minted House of Representatives?

In 1789, both Monroe and Madison sought the job, with Madison decisively winning the election. Throughout their campaign, the two got along amicably and, every so often, would accompany each other en route to debates.

4. That He Once Thought America Should Rent Portugal’s Navy?

Congressman Madison suggested that the American government ought to guard its oceanic interests by hiring the Portuguese navy for anti-pirate protection instead of constructing one of her own.

5. That Aaron Burr Introduced Him to His Wife, Dolley, in 1794?

Alexander Hamilton’s future dueling opponent knew Madison through congress (where he also worked at the time) and had gotten acquainted with Dolley by virtue of her mother’s boarding house, which Burr frequently visited.

6. That He Fathered No Biological Children but Helped Raise a Stepson?

Dolley’s first husband and oldest child had both died the year before she married Madison. The statesman immediately took an active role in caring for her only surviving son, two-year-old John Payne Todd.

7. That Madison Loved Ice Cream?

Before freezers came along, ice cream was one labor-intensive treat, and during their tenure in the White House, the Madisons served several varieties of this chilly dessert at official functions. Apparently, the First Lady’s favorite flavor was oyster (don’t knock it ‘till you’ve tried it).

8. That He Employed a Number of Secret Codes?

Madison—like any good politician—was terrified by the idea that someone might intercept one of his private letters. Along with Jefferson and many mutual allies, Madison used complicated encryptions when relaying delicate info. “Having now the use of my cypher,” he informed Jefferson in 1784 after mastering a new system, “I can write without restraint.”

9. That When the British Burned Down His White House, They Also Ate His Dinner?

On August 24, 1814, an eerie emptiness fell upon the Presidential mansion. As British forces advanced towards Washington, Madison’s home and office was completely abandoned mere hours before it was besieged. After breaking in, the Brits set the place ablaze before discovering and gobbling up a beautifully-cooked meal that had been intended for the President. Apple, cider, and wine complemented the menu.

10. That Both of His Vice Presidents Died in Office?

George Clinton kicked the bucket in 1812. His short-lived replacement was former Massachusetts governor Elbridge Gerry, who’d redistricted that state to tip the political scales in his favor, a process we now call “Gerrymandering.” Gerry also died in office, and is the only signer of the Declaration of Independence to be buried in Washington, DC.

11. That He Had a Naughty Sense of Humor?

During his days as Secretary of State, Madison once had to cover the costs of a visiting Tunisian diplomat named Sidi Soliman Mellimelli. This honored guest was accompanied by several concubines whose expenses Madison lightheartedly justified to President Jefferson as “appropriations for foreign intercourse.”

12. That He Once Proposed a Brutal Hangover Experiment?

At an 1804 party thrown in Jefferson’s honor, journalist Samuel Harrison Smith spent part of his evening drinking with Secretary Madison, who voiced his desire to find out how many glasses of champagne would be necessary to trigger “a headache the next day.” It’s unknown if anyone present ever actually tried this experiment.

13. That He Helped Amend Virginia’s State Constitution at the Ripe Old Age of 79?

This event—which consisted of a 96-man assemblage—became his last public appearance as a politician.

14. That He Played a Big Role in Establishing the University of Virginia?

Among UVA’s original trustees, Madison later served as its second rector (“chairman”) from 1826 to 1836 and bequeathed most of his personal library to the school.

15. That He Was the Last Surviving Signer of the U.S. Constitution?

For months, the dying ex-President remained bed-ridden. On June 28, 1836, Madison found he couldn’t swallow his breakfast. “What is the matter, Uncle James?” asked his beloved niece, Nelly Willis. “Nothing but a change of mind, my dear,” he replied before died.

16. That His Face Appeared on a $5000 Bill?

The government stopped printing these in 1945.