Metaphorically speaking, to open a can of worms is to examine or attempt to solve some problem, only to inadvertently complicate it and create even more trouble. Literally speaking, opening a can of worms, as most fishermen can attest, can also mean more trouble than you bargained for. No surprise, then, that the phrase was inspired by real live creepy crawlies.
While the exact origin and first usage isn’t clear, various dictionaries and historians of slang and idioms agree that the phrase was born in the U.S. in the 1950s or earlier and references an actual conveyance for Oligochaetes. In the days before plastic and styrofoam containers were ubiquitous, bait shops often sold earthworms and other live bait to fishermen in metal cans with handles and lids.
The great thing about live bait is that it's alive, so it wriggles on the hook and tempts fish with its movement. The bad thing about live bait is also that it's alive, and leaving the lid of the container loose or off is a great way to lose your bait. Given the opportunity to exit, worms will often either escape or just generally make it difficult to get them all back in the can and replace the lid. Once you've opened a can of them, you've got a problem on your hands.