On September 3, 1814, Francis Scott Key and John Stuart Skinner set sail from Baltimore aboard the ship HMS Minden, flying a flag of truce on a mission approved by President James Madison. Their objective was to secure the exchange of prisoners, one of whom was Dr. William Beanes, the elderly and popular town physician of Upper Marlboro and a friend of Key’s who had been captured in his home. Beanes was accused of aiding the arrest of British soldiers. Key and Skinner boarded the British flagship HMS Tonnant on September 7 and spoke with Major General Robert Ross and Vice Admiral Alexander Cochrane over dinner while the two officers discussed war plans. At first, Ross and Cochrane refused to release Beanes, but relented after Key and Skinner showed them letters written by wounded British prisoners praising Beanes and other Americans for their kind treatment. Because Key and Skinner had heard details of the plans for the attack on Baltimore, they were held captive until after the battle, first aboard HMS Surprise and later back on HMS Minden. After the bombardment, certain British gunboats attempted to slip past the fort and effect a landing in a cove to the west of it, but they were turned away by fire from nearby Fort Covington, the city's last line of defense.
During the rainy night, Key had witnessed the bombardment and observed that the fort’s smaller "storm flag" continued to fly, but once the shell and Congreve rocket barrage had stopped, he would not know how the battle had turned out until dawn. By then, the storm flag had been lowered and the larger flag had been raised.
When Dr. Beanes heard that the British were withdrawing from Baltimore, he told the 3 or 4 British soldiers quartered at his home that there were no hard feelings, and that they should share a brandy before they left. And another. And a few more. Pretty soon the British soldiers were drunk, and Beanes sent a servant to get the county Sheriff, who came and arrested the soldiers. When the British commander heard about it he was mighty pissed off, and sent a party ashore to arrest Beanes. The doctor was taken to a prison ship, which was thought to be no place for a gentleman, and so the effort on his behalf. The anthem was written that night onboard HMS Minden, tied up to HMS Tonnant.
Congreve was a British engineer who had fought in India, and had seen rockets used there. They were totally inaccurate, but they were new and loud and bright, and scared the bejeebers out of the colonists.