Adultery - Yes

 After it was discovered that Joseph had an affair with his 16 year old housemaid, members of the church wanted to replace Joseph Smith with David Whitmer as leader of the church.

CHRONOLOGY OF AFFAIRS


Fanny Alger is Joseph's first known plural wife, whom he came to know in Kirtland during early 1833 when she, at the age of 16, stayed at his home as a housemaid. Described as "a varry nice & Comly young woman," according to Benjamin Johnson, Fanny lived with the Smith family from 1833 to 1836. Martin Harris, one of the Three Witnesses to the Book of Mormon, recalled that the prophet's "servant girl" claimed he had made "improper proposals to her, which created quite a talk amongst the people." Mormon Fanny Brewer similarly reported "much excitement against the Prophet[involving] an unlawful intercourse between himself and a young orphan girl residing in his family and under his protection." Former Mormon apostle William McLellin later wrote that Emma Smith substantiated the Smith-Alger affair. According to McLellin, Emma was searching for her husband and Alger one evening when through a crack in the barn door she saw "him and Fanny in the barn together alone" on the hay mow. McLellin, in a letter to one of Smith's sons, added that the ensuing confrontation between Emma and her husband grew so heated that Rigdon, Frederick G. Williams, and Oliver Cowdery had to mediate the situation. After Emma related what she had witnessed, Smith, according to McLellin, "confessed humbly, and begged forgiveness. Emma and all forgave him." While Oliver Cowdery may have forgiven his cousin Joseph Smith, he did not forget the incident. Three years later, when provoked by the prophet, Cowdery countered by calling the Fanny Alger episode "a dirty, nasty, filthy affair." Chauncey Webb recounts Emma's later discovery of the relationship: Emma was furious, and drove the girl, who was unable to conceal the consequences of her celestial relation with the prophet, out of her house.  At least one account indicates that Fanny became pregnant. Chauncey G. Webb, Smith's grammar teacher, later reported that when the pregnancy became evident, Emma Smith drove Fanny from her home (Wyl 1886, 57). Webb's daughter, Ann Eliza Webb Young, a divorced wife of Brigham Young, remembered that Fanny was taken into the Webb home on a temporary basis (Young 1876, 66-67). Fanny stayed with relatives in nearby Mayfield until about the time Joseph fled Kirtland for Missouri. Fanny left Kirtland in September 1836 with her family. Though she married non-Mormon Solomon Custer on 16 November 1836 and was living in Dublin City, Indiana, far from Kirtland, her name still raised eyebrows. Fanny Brewer, a Mormon visitor to Kirtland in 1837, observed "much excitement against the Prophet [involving] an unlawful intercourse between himself and a young orphan girl residing in his family and under his protection"

-- Sep 24, 1834 -- Article on Marriage read at General Assembly stating that "...one man should have one wife, except for death."

-- Aug 17, 1835 -- Oliver Cowdery and Sidney Rigdon presented a revised set of "inspired documents" before a special Conference of the Church, which approved the collection for printing. Section 101, placed almost at the end of the book, gave instructions on purely monogamous marriage. The text stated that the "Church of Christ has been reproached with the crime of fornication, and polygamy..." Joseph Smith was out of town when the Conference accepted the collection as being the Doctrine and Covenants of the Church. As no known allegations of Mormon polygamy had been published at this time, it appears that the "reproach" came from within the Church itself.

-- Oct 17, 1835 -- Joseph Smith's journal entry states: "Called my family together arranged my domestick concerns and dismissed my boarders." This may refer to his first plural wife, sixteen-year-old Fanny Alger, with whom his relationship has been causing rumors and dissension among Joseph's inner circle, leaving his home to live apart from him.

-- Nov 24, 1835 --  While illegally performing a marriage for Newel Knight and Lydia Goldthwaite (still legally married to another man), Smith publicly announces his view of theocratic ethics: "I have done it by the authority of the holy Priesthood and the Gentile law has no power to call me to an account for it."

-- late Aug 1836 -- In Joseph Smith's absence some members call for David Whitmer to assume leadership of the Church. A similar situation happened during Smith's trip to Canada in July of 1837. Accounts of these two events are difficult to sort one from another. The earlier attempt to unseat Smith may possibly have been related to his affair with Miss Fanny Alger.

-- late Aug 1836 -- Joseph Smith asked Levi Hancock to take Fanny Alger to Missouri. She and her family left the following month, and after a lengthy stop-over in Indiana (for unclear reasons, perhaps her pregnancy), reached Missouri a year later.

-- Nov 16, 1836 -- Fanny Alger (Smith) marries Solomon Custer in Wayne county, Indiana.

-- Apr 28, 1837 -- The presidents of seventies meet and declare for publication that "we will have no fellowship whatever with any Elder belonging to the quorum of the Seventies who is guilty of polygamy or any offense of the kind."

-- Aug. 1837 -- David W Patten: Joseph Smith Jr. "slapped him in the face & kicked him out of the yard" Aug. 1837 when David W Patten asked if Fanny Alger was Joseph Smith Jr. 's plural wife

-- early Nov 1837 -- Oliver Cowdery confronted Joseph Smith over the matter of Smith's having carried on sexual relations with Miss Fanny Alger in Kirtland. Smith called in witnesses and shook hands with Cowdery, their both having agreed "to drop every past thing."

-- Nov 6, 1837 -- Oliver Cowdery: Allowed by leading quorums 6 Nov. 1837 to resolve difficulty with Joseph Smith Jr. over polygamy without involving the church council

-- Jan 13, 1838 -- Luke Johnson arrests Joseph Sr. for performing a marriage without proper authority but also helps him escape. Joseph Sr. hides at Oliver Snow's for two or three weeks.

-- (Sun) Jan 21, 1838 -- Oliver Cowdery confronts Smith with charge of adultery with Fanny Alger. (Perhaps by letter, as Smith had not yet arrived in Missouri)

-- Jan 21, 1838 -- Oliver Cowdery: "A dirty, nasty, filthy affair of his and Fanny Alger's was talked over in which I strictly declared that I had never deviated from the truth."  Oliver's letterbook

-- 1838 July -- The Church's Elders' Journal said "... on account of the unfaithfulness of Oliver Cowdery ... and [his] opposition to our beloved brother Joseph Smith, jr ... [he has] been excluded from fellowship." It printed a statement from the husband of William Morgan's widow saying Cowdery discussed the "crime (adultery/polygamy) alledged against" Joseph Smith, Jr. in her house. It also said Mormons did not have more than one wife and that Joseph Smith was a money-digger. (Joseph Smith may have married Lucinda in June 1838, evidence for this is not solid)

 Sources: 

Source: http://mormon-church-history.blogspot.com/2009/10/joseph-smith-and-marriage-in-1830s.html. Also: http://wivesofjosephsmith.org.