The Motown songwriters Brian Holland, Lamont Dozier and Eddie Holland (Holland-Dozier-Holland) wrote this song. It was inspired by the trademark line of Jackie Gleason, "How sweet it is!" as it was said on The Jackie Gleason Show. Gleason was one of the biggest TV stars of the time.
Lamont Dozier said the lyric was "wishful thinking." He was "sitting at the piano, fantasizing about how good it would be if the woman I had eyes for was madly in love with me. But the truth is she never gave me a second look. So I created this fantasy world where I was the object of all her affections, a bit of fiction that eventually came true due to my persistence and showering her with affection."
This was the second of Gaye's 13 Top-10 hits. His first was 1963's "Pride and Joy," which peaked at #10.
The Holland-Dozier-Holland team often started songs as ballads, crafting a heartbreaking story and then setting it to upbeat music. This song took a different approach, which they sometimes used: letting the music dictate the song's direction. The result is one of their few hit songs that is utterly joyful in both music and lyrics.
In a Songfacts interview with Lamont Dozier, he explained: "The drum feeling is the shuffle and it's a 12-bar shuffle blues, basically. But, finding the chords, I remember Harvey Fuqua of the Moonglows who had came to Motown, he mentioned that song. He said, 'Man, those chords that you played were really interesting because it's the blues but not your run-of-the-mill blues. The chords you have are very intricate.' I said, 'Well, I just play what I feel.'
That's the way that particular song came out. Those chords gave me the feeling of that melody."
This became a #5 hit for James Taylor when he covered it in 1975. It was also recorded by Jr. Walker and the All-Stars, whose version hit #15 in the US in 1966.
Eddie Holland came up with the title and hook for this song. The rest of it took a while to put together, and by the time Gaye recorded it, he had to read from lyric sheets because he had not had time to memorize them.
Although it's not featured on the film's official soundtrack, "How Sweet It Is (To Be Loved By You)" was performed by the school choir in the 1999 teen comedy, American Pie. The performance takes cues from James Taylor's 1975 cover, which featured his then-wife Carly Simon on backing vocals and earned him a #5 hit on the Billboard Hot 100 chart.
In 2001, ESPN took some heat in the press when they used this song as background music in a Father's Day tribute featuring athletes and their fathers or kids. Gaye had been shot by his father in 1984.