A/B Testing for Nonprofit Organizations

  1. A/B Test your Fundraising Appeals -- Are you getting the most out of DonorSnap?
  2. Free DonorSnap Training
  3. Not Using DonorSnap Yet?

A/B Testing Appeals -- Are you getting the most out of DonorSnap?

You've probably heard the words "A/B Testing" from a consultant, fundraising article, or your executive director... but what exactly is it and how do you do it?! In DonorSnap, you can use a combination of tools to help you easily test the effectiveness of your fundraising asks.

A/B testing for nonprofit organizations

What is an A/B test?
An A/B test is creating two versions (or more) of the same letter, email, script, etc. that are delivered to randomly selected segments of one group. By tracking which version each person receives, you can compare the success or conversion rates of each version to determine which one was more effective.

Why should I test my fundraising appeals?
When you know which version performed better, you can use those techniques again next time to increase giving, email open rates, event signups, and any other action that you want your supporters to take. Usually an A/B test will contain only one or a few differences so that it's more clear which change caused the effectiveness to skyrocket.

What should I test?
There is no limit to number of ways you can test your fundraising appeals. Start by identifying your goal. Common goals might attempt to increase one of these:

  • Number of individual donations
  • Total amount given
  • Average gift amount
  • Email open rate
  • Number of volunteer signups

Your organization is unique, so don't feel limited by those ideas. Once you know what you want the outcome to be, you need to decide which parts of your appeal you should change. Here's a short list of ideas to help get you thinking:

  • Letter layout or formatting
  • Specific dollar amount asked for
  • Pictures or no pictures
  • Email subject lines
  • Insider language or common language
  • Call scripts
  • Individual stories or a general overview
  • Letter content
  • Long stories or short stories

The important thing to remember is that every organization is different. What may work positively for one organization could be negative for another. Start with something simple or talk with your consultant for tips on getting started.

Set up your Audience Segments in DonorSnap

Next comes the setup and administration. This can often times be the difficult part, but these tips should help make it simple.

1. Create your tracking fields. You will need a new user defined drop down field and a choice for each version you plan to use. (View instructions on the help page.) Create this field on the Contact tab and give it a title specific to this test. For the sake of this article, I'll just call it TestVersion. Once you have finished the test and analysis, you can inactivate or delete this field so that it doesn't clutter your screens in the future.

Tip: Be specific in your naming so that it is easy to understand which version is which. Instead of using generic terms like "Version A" and "Version B" use names like "Ask 500" and "Ask 1000," "Short Letter" and "Long Letter," or "Support Headline" and "Save Headline."

2. Create the entire list. Use DataMiner Platinum to create a list of all the contacts you want to include in your fundraising appeal. (Watch DataMiner Platinum trainings.) Include the ContactIPK field and the new TestVersion field that you created for the test in your Field Export Selection tab as the first two fields. Export the list to Excel.

3. Segment your list in Excel. You currently have a list of all the contacts you want to test, but they haven't been assigned to a version yet. In order for the test to be statistically valid, you need the two groups to be randomly selected from the same audience. Here's the scary part... you'll need to use Excel to randomize your list, but I promise it's easy.

In an empty column title it Random Number. In the cells below for each line enter =RAND() which will create a random number between 0 and 1. When you type =RAND() into the first data row, click on the cell to select it. Drag the square in the lower right corner of the cell's outline and drag it down to the bottom of your data to copy the cell quickly down the column.

Sort your entire spreadsheet using that column to give you a random order. In the TestVersion column, type one version name for the top half, and in the bottom half type the other. The random numbers will automatically change each time you edit a cell in your spreadsheet, but you can ignore that. After the spreadsheet is sorted once, you have a random selection.

4. Import your spreadsheet back into DonorSnap. Save your spreadsheet and open up DonorSnap. Go to Maintenance > Excel Import. Upload your spreadsheet and choose the Contact Update option in step one. In the field mapping step, you only need to match the TestVersion field in your spreadsheet with the new user defined field in DonorSnap. To avoid importing any accidental changes, remove the mapping between all other fields by dragging the field from the box on the left into the box on the right. Follow the steps and import your changes. (Watch the Contact Update training for help.)

5. Create a list for each test segment. Go back to DataMiner Platinum and create a new batch. Using the Query Builder or the filters in the Review tab, select only one of the versions. Create another batch to select the other version. Now you have your separate lists that can be used in DataMiner Batch Letters, Constant Contact, Contribution Statement Emails, or export to Excel.

Analyze Your Results

After you have waited long enough to conclude your test, DataMiner Platinum will help create the reports needed to analyze the results of your test. Since the TestVersion field is on the contact tab, it can be included in any DataMiner Platinum report. For example, let's say my goal was to increase the number of individual donations from this appeal. Here's how I would do that:

As I received the donations in response to my A/B test, I made sure to label them in a way that set them apart from other types of donations. Let's say I used the Appeal field and labeled it as "Year End Letter 2015." Next I'll create a new DataMiner Platinum batch with the following settings:

  • Data Export Selection: Donation and then Detailed Information at the bottom
  • Field Export Selection: I'll include all the important fields including TestVersion, DonationAmount, DonationDate, DonationAppeal and some general contact details.
  • Query Builder: I'll skip this tab and just use the filters in the next one.
  • Review Data and Export: Add a filter to the DonationAppeal column of "Year End Letter 2015."

The above settings would give me a report of all the donations that were made in response to my test. From here I could export it to Excel for more in-depth analysis. Another option would be to use the TestVersion field to filter out the different versions in the Review tab. I could look at the total number of donations for each version by switching between Ask 500 and Ask 1000 in the filter for that field.

Finally, I need to know if the test is "statistically significant" which means "Was my test large enough, and are the results different enough for me to know that it is not just random chance?" There are many tools out there that you can use for this. Here is a simple one worth trying: abtestcalculator.com. I'll enter the total number of people that received each version along with the total number of donations that each audience made. In most situations, you will want the calculator to report that there is a 95% confidence level that the test is significant.

A/B Test Results for Fundraising Appeal

Once you know which version performed better, you should use that change going forward... or use that version as the base for your next test!

Free DonorSnap Training

DonorSnap is now offering a free training almost every week. The free trainings take place on Wednesday afternoons and the topics range from Getting Started with DonorSnap to advanced features like DataMiner Platinum and Online Forms. The next 4 trainings can be viewed at any time.

Find the link to the training schedule under the Help menu from within DonorSnap:

Upcoming Trainings:

  • December 9th: DataMiner Platinum Intermediate
  • December 16th: Getting Started with DonorSnap
  • December 23rd: No training this week
  • December 30th: No training this week
  • January 6th: Contribution Statements

Space is limited so Register Now.

Want to Watch a Past Training?

Each of our weekly trainings are recorded and placed on the DonorSnap Help Site. These trainings are generally available a few hours after the training ends.

See all of the other past trainings on the help site under Training Webinars.

Not Using DonorSnap Yet?

Sign up for one of our free webinar demonstrations to see the software in action.
Ready to be set up with your own database? Get started today!