James Madison Bell was born in Gallipolis, Ohio. His parents' identities are unknown. At age sixteen, in 1842, he moved to Cincinnati. While there, in 1848, he married Louisiana Sanderlin (or Sanderline), with whom he had several children, and also learned the plastering trade from his brother-in-law George Knight. Bell worked as a plasterer during the day and attended Cincinnati High School for Colored People at night. Founded in 1844 by Reverend Hiram S. Gilmore, the school had a connection to Oberlin College and was said to have given impetus to the sentiment found in Uncle Tom's Cabin and the cause of human freedom. Through his studies Bell was thoroughly indoctrinated into the principles of radical Abolitionism.
In 1854 Bell moved his family to Chatham, Ontario, Canada where he felt he would be more free under the authority of the British government ...