Picking songs by using a survey

Many careers ago I used to work for a radio station. One of the things I learned while I worked there is how they pick the songs they play—and more specifically how long to keep a song in rotation and how frequently people want to hear that song. Radio stations use surveys to do this. That one time you called in to win a t-shirt, you gave them your contact information in return. So later in the year, someone would call you from an “independent” consultant group (I put that in quotes, because it really never is independent). They would play clips of songs over the phone and ask you to rate them based on a scale they gave you. Nowadays, radio stations use online surveys based on the same concept of playing audio clips and asking you to rate them.

The goal of a radio station programmer and the goal of a worship leader are vastly different. One purely wants to entertain you while the other wants to use music and songs to help the church express their love for God. But even though the goals are different, I’ve found this tool works well in both cases.

I generally try and do two surveys a year. Each survey contains 18-20 songs. At the beginning of the survey, I try to set the tone that this isn’t a popularity contest. How you pick songs for a worship service and how you pick songs for your Spotify workout playlist are very different. A lot of picking songs has to do with context (what are the other songs in the set, what is the sermon about, etc.) But, at the same time, I want to recognize the fact that we know certain songs just resonate with our hearts more than others. And the only way to really find out is to ask people. If you frame the questions in such a way that it’s not about picking your favorites but rather trying to think back to how God has personality used that song over the past few months, you can get some good feedback.

Here’s a PDF file of what the first few questions of the online survey looks like.

With each song clip, I ask people to pick from one of five choices. They are:

  1. I really enjoy singing this song
  2. It’s not my favorite, but it’s okay
  3. I used to enjoy singing this song, but I’m ready for something else
  4. I’ve never really enjoyed singing this song
  5. I’m not familiar with this song

The first two options tell me that the song works (although if too many people say it’s just okay, I probably should lean more on songs that score higher on the “really enjoy” option). If a song scores high with option 3—that they’re ready to move on—I look into putting that song on the shelf. I know the song works, but we’ve burned people out on it. We put it away for awhile, but I know I can bring it back later when it fits—that it’s a viable option in the future after we give it a break. If a lot people tell me they’ve never enjoyed a song, then it’s probably time to scrap it for something else. The fifth option just gives people an out if they’ve never heard the song.

Here’s a PDF file of the final report I received. When analyzing the data, I try to look at it as a whole. If a song scores low, it may not be just about the song itself. It may be speaking about your overall philosophy of how you build your set lists. The first time I did one of these surveys, songs like “Let It Be Known” by Worship Central and “Alive” by Hillsong Young & Free scored very low. But by looking at all the data as a whole (including the comments), what I discovered was my philosophy was off. I was building sets that had too much “let’s get excited about Jesus” and not enough introspective love songs to God. I was also trying to be too trendy and using too many new songs, when our people wanted more of what they were already familiar with (like “How Great Is Our God” by Chris Tomlin and “God of Wonders” by City on a Hill). Once I fixed the philosophy, I tested the song “Alive” again and found it scored pretty high.

Also, this doesn’t replace your other means of deciding which songs to include in your sets—it’s in addition to. For example, I noticed Kari Jobe’s “Forever” didn’t test as high as some of the other songs. However, the last time our band did the song (a few weeks ago), you could tell that people all over the room were connecting with God through the song. Context is everything.

Think about how you use the songs in your sets as well. The first song in our sets is typically designed more to set the tone and put energy into the room. We don’t necessarily expect everyone to sing along (and quite frankly we have tons of people still walking into the auditorium during the first song), so these songs are more performance songs that might have a catchy chorus to sing along with. Some examples would be “In Sync” by Hillsong Young & Free and “Glory In The Highest” by Fellowship Creative. I don’t expect these songs to test very high compared to a song like “Lord, I Need You” by Matt Maher. But I am using the survey to compare opening songs against each other, to see which ones we should lean on more (or we should drop).

One of the most important questions you can ask on any survey is “What else do you want to share with us?” It can open a whole new door of information on how the worship ministry is succeeding (or not succeeding) in their goal. Get ready for some brutal comments when you ask people’s opinion. You have to have thick skin when you attempt something like this. The very first comment I received on our very first survey was this:

“Changing worship leaders didn’t improve the worship.”

Music is such a subjective thing. So you’re not going to hit the mark every time with everybody. But try not to take the individual comments personally and instead look at the overall message. Pay attention to the trends, the things people say over and over.

I’ve found surveys to be a great tool to help keep a pulse on how our worship ministry is best serving the church. It’s not the only tool, but the feedback I get from an online survey has been very helpful in shaping how our church worships God through music.