Leveraging GovDelivery for
MedlinePlus Email Campaigns and
Outreach
Tyler Moses
National Library of Medicine Associate Fellow
Date of Submission: August 24, 2017
Project Sponsors: Katherine Masterton and Katie Chan
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 2
Acknowledgements
This project would not have been possible without the support received from
the Project Team, Katie Chan, Kate Masterton, Fedora Braverman, Stacy Coppess,
and Ekaterina Eby. Also, thanks are given to those who assisted in the hallway
usability test, Alla Vysokolova, for creating the email layouts, and Joanna Widzer for
her advice and assistance in conducting the test itself. The Associate Fellow thanks
those who took the time out of their schedule to be interviewed and taught her
about email marketing at their institutions as well as email marketing best
practices. Special thanks are given to the Associate Fellow’s mentors, Lisa Boyd and
Wanda Whitney, for supporting her throughout the project. Additionally, the
Associate Fellow wants to thank Dianne Babski, Joyce Backus, Jerry Sheehan, and
Dr. Patricia Flatley Brennan for their support of the Associate Fellowship Program.
Lastly, the Associate Fellow wants to thank Kathel Dunn, the Coordinator of the
Associate Fellowship Program, and the 2016-2017 Associate Fellows, Megan Fratta,
Kendra Godwin, and Candace Norton, for providing encouragement and creative
ideas on how to make a better email.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 3
Table of Contents
Acknowledgements ........................................................................................ 2
Abstract ........................................................................................................ 4
Background ................................................................................................ 4
Methods ..................................................................................................... 4
Results ...................................................................................................... 4
Conclusions ................................................................................................ 4
Introduction .................................................................................................. 5
Methodology/Procedures ................................................................................. 6
Email Marketing Education ........................................................................... 6
Email Bulletin Creation and Dissemination ...................................................... 7
Hallway Usability Testing .............................................................................. 8
Results ......................................................................................................... 8
Outcomes from Email Bulletins ...................................................................... 8
Results from the Hallway Usability Testing .................................................... 10
Discussion ................................................................................................... 11
Information Acquired from Email Marketing Resources ................................... 11
Best Practices for Email .............................................................................. 11
Limitations ............................................................................................... 12
Recommendations ........................................................................................ 13
Bibliography ................................................................................................ 17
Appendix A: A Table of the Individuals Interviewed Regarding Email Marketing
Strategies ................................................................................................... 19
Appendix B: Hallway Testing Script and Notes ................................................. 20
Appendix C: Two Panel Newsletter Layout ....................................................... 26
Appendix D: Five Panel Newsletter Layout ....................................................... 27
Appendix E: Customized Email with a Blue Banner Layout ................................. 28
Appendix F: Customized Email without Blue Banner Layout ............................... 29
Appendix G: Comparison of HubSpot, Granicus, and MailChimp ......................... 30
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 4
Abstract
Background
As a health information website, MedlinePlus provides information on an array of
health topics, such as diabetes and aspirin, to the public. Between MedlinePlus and
its sister site, MedlinePlus en Español, there are 83,700,000 email subscriptions
that receive email updates on their health topics of interest. Though a large volume
of emails is sent to subscribers, the links within are seldom clicked on. The Project
Sponsors wanted to identify potential email marketing strategies to engage more
subscribers with MedlinePlus emails.
Methods
Through learning opportunities, such as webinars and interviews with staff from
federal agencies that participate in email marketing, the Associate Fellow applied
the knowledge acquired to create customized emails that were sent to several
health topic subscriber groups; the data from GovDelivery and Google Analytics
were utilized to identify which elements of the emails attracted subscribers. This
information was compiled to design two types of email layouts (newsletter and
customized email) that were used in a hallway usability test to acquire real time
feedback on email elements the participants liked.
Results
Data from these tests illustrated that subscribers prefer emails with large font, to
feature images relevant to them, and have colorful designs. Subscribers are more
likely to open emails with statement subject headings. In addition, subscribers are
more likely to engage with emails if there is a clear call to action and if there are
noticeable links that lead to new information. Furthermore, subscribers feel that the
customized emails should contain less information while newsletters should feature
a variety of new stories.
Conclusions
The elements that have been identified from these experiments should be
incorporated into future emails to better engage subscribers. More tests should be
done to further identify other elements (i.e. time of day sent, day of the week, etc.)
that could help make MedlinePlus email marketing more effective.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 5
Introduction
MedlinePlus is a consumer health information website produced by the
National Library of Medicine (NLM). The site provides a range of health information
on medical topics and conditions (e.g. tanning and osteoporosis) as well as drugs
and supplements (e.g. aspirin and cod liver oil). MedlinePlus users can subscribe to
receive email notifications when an update on the website pertaining to their health
topic of interest occurs. To date, there are 930,000 subscribers between both the
English and the Spanish versions of MedlinePlus; among those subscribers, there
are 83,700,000 email subscriptions for both MedlinePlus and MedlinePlus en
Español. The emails are delivered to subscribers through GovDelivery, an electronic
communication system provided by Granicus that assists government institutions in
reaching the public through various mediums such as email or text messages
(Granicus, 2017b). For MedlinePlus, these emails are typically automatically
generated and contain a hyperlinked text with an image and/or a brief description.
Despite the large number of subscribers and the frequency in which the emails are
sent, a small percentage of these emails are opened and few of the links within the
emails are clicked on.
The goal of the Spring 2017 project, “Leveraging GovDelivery for MedlinePlus
email campaigns and outreach” was to see whether customizing the emails
increases the number of subscribers who open the emails as well as the volume of
online traffic to MedlinePlus from subscribers clicking on the links in the email. The
objectives to achieve such a goal include:
• Becoming familiar with the GovDelivery email system and assessing how it
is currently used by MedlinePlus staff as well as the different features that
have not been optimized;
• Exploring GovDelivery analytics and identifying opportunities for potential
growth;
• Proposing different approaches for the enhanced emails, and approaches
for metrics;
• Creating and sending emails with new formats and content and;
• Analyzing effectiveness of various techniques, using data collected by
GovDelivery but also by utilizing URL parameters to look at the Google
Analytics data, segmenting users who visited the site from GovDelivery.
Another objective that was later included was to construct a process that would
lead to the project being replicated upon the conclusion of the Associate Fellow’s
participation.
The project was conducted by Katherine (Kate) Masterton (project sponsor),
Katie Chan (project sponsor and GovDelivery troubleshooter), and Tyler Moses
(National Library of Medicine Associate Fellow) with Fedora Braverman (Spanish
translation expert), Stacy Coppess (MedlinePlus social media and marketing
consultant), and Ekaterina (Kate) Eby (digital analytics consultant) serving as the
main resource people.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 6
Much of the research literature highlights the benefits of health professionals
using email to connect patients with their medical records to make patients more
informed as well as engaged in their medical care. However, emails also serve as
an excellent method to connect patients with quality consumer health information
(Atherton, Huckvale, & Car, 2010). Email enables patients to access information on
their own time and save the information for future reference; in addition, sending
health information directly to patients decreases the amount of time they spend
looking for credible health information (Atherton, Huckvale, & Car, 2010).
In marketing, emails with lackluster designs often deter people from clicking on
the links within the body of the content (Rowe & Burridge, 2010). Some techniques
that are believed to make emails more engaging are (Rowe & Burridge, 2010):
• Using color and icons to:
o Direct subscribers to navigation tools and
o Make them visually appealing.
• Creating interesting original content with a strong call to action (a message
that compels people to do something).
Though these strategies are more applicable for emails that advertise products,
these same design tools can be applied to MedlinePlus’ emails to increase
engagement among subscribers by persuading them to click on the links within the
emails.
Methodology/Procedures
The project was divided in to four phases:
1. Email marketing education
2. Email bulletin creation and dissemination
3. Testing variables
4. Hallway usability study
Email Marketing Education
Throughout March 2017, the Associate Fellow conducted research on email
marketing and interviewed eleven individuals from both NLM and other institutions,
such as the National Institute on Aging and the Food and Drug Administration to
gauge their experience with GovDelivery, email marketing, and what facets within
an email are engaging to them personally as well as their subscribers. See
Appendix A for a list of individuals whom were interviewed.
Along with the interviews, the Associate Fellow attended the Granicus Digital
Communications Summit in Washington D.C. on April 13, 2017. The conference
focused on the diverse features the platform offers to engage users as well as what
tools can be utilized to determine how well an institution is connecting with its
users. The Associate Fellow also took online email marketing and Google Analytics
courses on Lynda.com as well as read diverse email marketing pamphlets produced
by Granicus.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 7
Email Bulletin Creation and Dissemination
Prior to creating any email bulletins, the Associate Fellow met with the
Project Team to design a process on how the emails would be created and which
websites could be used as resources; to familiarize herself with the features in
GovDelivery, the Associate Fellow looked at the “MedlinePlus GovD Home” pages
from the NLM wiki page on the Confluence platform. The deadlines for the work
process were documented on pages of a calendar from March 2017 through July
2017. The Associate Fellow was granted access to both the Spanish and English
GovDelivery accounts to compose and send out emails; she was also given access
to MedlinePlus’ Google Analytics account to analyze the data from the links in the
email bulletins.
To effectively identify what changes in an email were engaging to users, the
Project Team and Associate Fellow decided to do A/B testing with the emails. A/B
testing is a method in which subscribers are divided into two groups and both
receive the same email with only one small difference (the variable) between them
(e.g. an email with an image and then another one without an image). Data, from
either the email marketing services or Google Analytics, would be used to
determine which change lead to more emails being open or more engagement with
users (Duquet, 2016).
During meetings with the Project Team, they and the Associate Fellow
selected which health topics would receive an email bulletin and what variable
would be tested. The health topics chosen from March-June are as follows:
Diabetes, Autism, Vitamin D and Ticks. Two emails with content tailored to the
interests of the subscribers (customized emails) were created with only one
variable changed. Then using the “Subscribers Added” criteria in GovDelivery, the
subscriber list for each topic (and some related topics) was divided in half by the
date in which a subscriber signed up to receive an email bulletin for a health topic;
one email was sent to the first half of the group and the other email to the second
half of the group to determine which version people responded to best.
Using the Google Analytics campaign builder, the Associate Fellow created
urchin tracking module (UTM) codes for each link so Google Analytics could retrieve
data on the activity from the webpage once the link has been clicked. A UTM code is
code attached to a customized URL to capture data (e.g. number of users, bounce
rate, time spent viewing a page, etc.) from an email or website (Launch Digital
Marketing, 2012). Data collected from GovDelivery and Google Analytics were
utilized to identify which variables were more engaging. The metrics used to assess
engagement were:
• Unique open rate (GovDelivery): Rate in which an email was opened for the
first time by a subscriber (Granicus, 2017c).
• Click rate/click-through rate (GovDelivery): Total number of times a link was
clicked on in the email (Granicus, 2017c).
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 8
• Unique click rate (GovDelivery): Rate in which a link was clicked on for the
first time by a subscriber (Granicus, 2017c).
• Bounce rate (Google Analytics): Rate in which a user viewed a page before
leaving. The smaller the bounce rate, the longer a user was viewing a
webpage (Granicus, 2017a).
• Users (Google Analytics): Number of people who visited a webpage
(Batesole, 2017).
• New users (Google Analytics): Number of people who are viewing a webpage
for the first time (Batesole, 2017).
• Average session duration (Google Analytics): Average number of interactions
a user has on the website within a specific time frame (Goggle, 2017a;
Google, 2017b).
The data informed what elements to include in the email designs produced for the
hallway usability test. The data also supports best practices for future MedlinePlus
email initiatives.
Hallway Usability Testing
On July 14, 2017, Tyler Moses, Katie Chan, Kate Masterton, and Joanna
Widzer (Specialist Librarian who has experience in hallway usability tests)
conducted a hallway usability test on the south side of the Clinical Center at the
National Institutes of Health (NIH). Hallway usability testing is where researchers
set up a booth in an area with a lot of foot traffic and ask bystanders to test and
provide feedback on their product(s) (USAJOBS Team, 2014). For the project,
participants were solicited to take two different tests where they were asked to look
at two types of email layouts: a customized information on a specific topic (design
with a blue banner versus design without blue banner) or a newsletter (2
storyboards versus 5 storyboards). Participants were tasked to look at the email
layouts and provide feedback on which designs or features of a design they felt
were engaging. The data would be utilized to construct possible email design
templates that could be used for future email notifications from MedlinePlus.
Participants were asked about the frequency of their email usage. To track the
number of non-federal employees to stay under the Paperwork Reduction Act limit,
participants were asked whether they were a federal employee and about their
profession. Four email layouts were constructed and tested: three by Alla
Vysokolova (E-Health Information Support Section at NLM) and one by Tyler Moses
using an email design layout created prior to the project by an email marketing
consultant at Granicus. See Appendix B for a copy of the script and the document
used to record participants’ feedback. Also, see Appendices C-F for copies of the
email designs used during the test.
Results
Outcomes from Email Bulletins
Table 1 compares the GovDelivery’s average open rate of the emails sent to
subscribers before the project and those sent during the project.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 9
Table 1: Average Open Rate of Emails Sent Before and During the Project
Granicus Accounts
Average Open
Rate Prior to
Project
Average Open
Rate of
Customized
Emails
English
6.19%
6.33%
Spanish
5.65%
6.1%
The data illustrates a slight increase in how frequent the emails were opened before
and during the project. However, there was no significant change in the click rate
between the customized emails and the emails sent prior to the start of the project.
The vitamin D email illustrated a difference between the unique open rate for
both the statement and question subject headings among the English (statement:
12% unique open rate; question: 7% unique open rate) and Spanish (statement:
8% unique open rate; question: 6% unique open rate) subscribers. There was no
significant difference in the data among tick bite English subscribers who received
an email with or without an image. However, tick bite Spanish subscribers who
received an email with an image clicked on the link to a MedlinePlus’ image of a tick
on person only five times compared to sixteen times from those who received an
email without an image.
Per the data gathered from GovDelivery, 37.5% of English subscribers who
received the customized tick email clicked on the link to the new MedlinePlus article
on the increase in the number of ticks during the summer. The data from Google
Analytics shows that the bounce rate from the MedlinePlus Lyme disease page that
was linked to the email was about 67% for Spanish subscribers compared to the
80% bounce rate for English subscribers. The data illustrated a difference in the
level of engagement by health topics among Spanish and English users. For
example, GovDelivery’s unique open rate for English subscribers to diabetes
complications was 4% while the unique open rate for Spanish subscribers was 8%.
Furthermore, Google Analytics’ bounce rate from the webpages attached to the
diabetes email ranged from 22.22-50% among Spanish subscribers while the
bounce rate for the same links ranged from 0-100% among English subscribers;
Google Analytics showed that the average session durations (44-389.33 seconds)
were higher for Spanish users compared to English users (0-214 seconds) for the
same diabetes related links in the email.
The data also illustrated that new subscribers were more compelled to open
emails compared to subscribers who have been receiving emails for a long period.
See Table 2 for a comparison of the average open rate between English and
Spanish subscribers slotted in the before (the group of individuals who subscribed
to a health topic prior to a specific date) and after (the group of individuals who
subscribed to a health topic after a specific date) groups for the tick email.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 10
Table 2: Comparison of the Average Open Rates of the Before and After
Groups of the Tick Email English and Spanish Subscribers
GovDelivery
Subscriber
Accounts
Average Open
Rates of Before
Group
Average Open
Rates of After
Group
English
3%
5%
Spanish
4%
5%
Only one email (insect bites and stings English subscribers in the before group) had
the most clicks for the first link (Health Daily news article: 6 unique clicks).
Results from the Hallway Usability Testing
A total of 20 (13 participants for news and 7 participants for customized
design on a specific topic) individuals participated in the hallway usability testing;
four of the participants were federal employees and sixteen of the participants held
other positions (e.g. interns, students, post doctorates, etc.). On average, the users
spend about 36 hours a week viewing emails.
Similar elements were identified from all participants. Participants from both
test groups felt that:
• Emails should contain relevant pictures (customized emails: 2 out of 7
people; news: 6 out of 13 people);
• Have a clear call to action (customized emails: 1 out of 7; news: 1 out of
13);
• Bigger font (customized emails: 3 out of 7; news: 2 out of 13) and;
• Designs should be colorful (customized emails: 2 out of 7; news: 1 out of
13).
Participants viewing the customized designs on a specific topic commented that
they wanted less information (3 out of 7 participants) while those viewing the
newsletter emails wanted more articles (6 out of 13 participants).
Overall, 10 participants preferred the five-panel newsletter while 5
participants were drawn to the two-panel design. For the customized email on a
specific topic, four participants liked design without the blue banner but three
participants favored the email with the blue banner. However, the customized email
test may have been biased by the researchers. The design without the blue banner
was printed from an email making the font bigger than the design with the blue
banner; many of the participants commented that they selected the design without
the blue banner because of the larger font.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 11
Discussion
Information Acquired from Email Marketing Resources
The challenge in email marketing is engaging users and compelling them to
go to an institution’s website. For federal institutions, they face the additional
challenge of competing for users’ attention with private and non-profit institutions;
for instance, a person, on average, receives about 100 email a day and they check
their inbox at least 74 times (Wales, 2017). In an area where information derives
from diverse groups, federal institutions need to learn how to approach their users
in an effective manner.
One way to pique users’ attention is to create a unique subject heading that
contains either a question or an intriguing statement; however, the subject heading
should not contain certain terms, such as “Free”, that will cause it to be placed in
the Spam folder (Saghafi, 2017). To further engage users, the content must align
with the interests of the users while staying focused on the topic and being succinct
(K. Blaine, personal communication, March 22, 2017). The content should contain
several calls to action to compel users to explore additional information on the
institution’s website (Duquet, 2016).
While images are an excellent method to engage users, they should be used
sparingly for some browsers block images from appearing within the body of the
email (S. Dailey, personal communication, March 20, 2017). Furthermore, using
links can be an excellent method to guide users to pertinent information but, the
links need to stand out from the rest of the content (Saghafi, 2017). Also, the first
link in the email should contain the most pertinent information that users need to
know for about 89% of users click on only one link within an email (O’Phelan,
2014). It is also beneficial to organize content in the email from the mindset that it
will be opened on a mobile device since many users open emails on their tablets or
smartphones (Duquet, 2016).
Though there is an overlap in best practices for engaging with individuals
through email, these elements are subjective for each institution due to the
diversity of their audience (Duquet, 2016). To determine which elements are
appropriate for users, institutions need to be willing to experiment with their emails
to identify what works for engaging their users. Lastly, institutions need to define
what success looks like for their email campaign and select the metrics, such as
open rate (percentage of total number of people who opened the email) and click-
through rate, that will be able to capture that data; by establishing what makes a
successful email campaign, institutions can uncover whether they effectively
engaged with users (Duquet, 2016).
Best Practices for Email
The data from both the email campaign and the hallway usability testing
highlighted the elements subscribers find engaging in an email. For instance,
subscribers were more likely to open emails with a statement subject heading
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 12
versus those with a question. While English subscribers did not exhibit any change
in engagement if the email contained an image, Spanish subscribers were more
inclined to click on links if the email lacked an image. Spanish users are more
inclined to select links that directed them to images and/or quantifiable information
(e.g. intake of vitamin, testing measurements, etc.); English subscribers are geared
towards links that yield timely information, like news articles, that pertain to their
health interests. As seen from the data, there is a deviation between Spanish and
English subscribers as to what health topics they find interesting. Contrary to the
published literature and information acquired from the interviews, subscribers tend
to click on links that lead to content relevant to their interests instead of the first
link. Overall, subscribers tend to stay longer on webpages that contain information
that they have not seen before. The data seen from the hallway usability testing
suggests subscribers prefer emails to have a clear call to action, larger font, colorful
designs, and images that are relatable to them. Also, email subscribers want to
receive email newsletters with a variety of news stories but want customized emails
with more succinct, topic-oriented content.
The information acquired from these tests should be incorporated into future
emails disseminated from MedlinePlus. Due to the divergence of interest in health
topics between English and Spanish subscribers, members from the MedlinePlus
team should utilize Google Analytics data to identify the most popular health topics
among both groups and create emails pertaining to those topics; in addition, the
content within the emails should predominately focus on new information to engage
subscribers who have been receiving email notices for a long time. For emails being
dispersed to Spanish subscribers, there should be a mix of links that guide users
towards visuals and information with values (e.g. number of calories consumed,
etc.). These changes could not only lead to higher open rates from current
subscribers, it could also increase the volume of subscribers that frequent
MedlinePlus as well as the time spent reading the content on the webpages;
furthermore, it could possibly lead to more subscriptions to MedlinePlus if users are
circulating their emails to other individuals.
Limitations
The data pulled from the links in Google Analytics does not always reflect
how the groups are divided in GovDelivery. Google Analytics groups the email
campaign data together by the UTM code assigned to the links. This grouping of
data prohibits researchers from conducting an in-depth analysis of the data based
on the segmented groups. The challenge is to devise a method where the division
of the subscribers in GovDelivery can be reflected in Google Analytics while
negating the bias that newer subscribers are more inclined to open emails.
Another limitation is the time in which it takes to create a customized email
in Granicus; the glitches in the system (inability to sometimes resize or move
images, resetting to default formats when inserting bullets, etc.) can extend the
time it takes to design an email. Also, the create and review process for an email
takes about a month. Adjustments should be made to the process if more emails
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 13
are to be produced within a single month; one adjustment could be selecting the
health topics to receive an email at least one week before the completion of the
current email bulletin. Lastly, having to rely on GovDelivery and Google Analytics to
collect the necessary variables leads to some of the data being lost between the
two systems for they utilize diverse metrics to analyze and organize the data.
Recommendations
To ensure the continuity of the project, here are the various phases that individuals
coming onto the project should explore:
Table 3: The Phases to Explore for the Email Campaign Project
Phase
Additional Details
Phase One:
Education
and Training
Lynda.com.
Name of courses:
• Email Marketing;
• Google Analytics and;
• 508 Compliance.
videos.
Videos to watch:
• GovDelivery Basic Bulletin Editor;
• Creating an Advanced Bulletin
Template and;
• Sending an Advanced Bulletin.
collected from the
GovDelivery project
(uploaded in to the
2016-2017 Associate
Fellow’s report folder
Materials include:
• Interview notes and;
• Granicus documents.
wiki page on creating
Topics to pay attention to:
• MedlinePlus GovD Home and;
• Spanish Procedures.
accounts for
GovDelivery and
Permissions to use the accounts will be
given from the MedlinePlus team.
review material on
A/B testing.
Online resources:
• Intro to A/B testing
• Diving Deeper into A/B testing: 30
Tests to Drive Engagement
Look at the 2016-2017 Associate Fellow
reports page on Sharepoint for additional
A/B testing material from Granicus.
Phase Two:
Creation of
Emails
Methods/Resources to select health topics:
• National Health observances;
• Seasonally relevant health topics;
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 14
Phase
Additional Details
• Topic Category Report created from
GovDelivery and;
• Looking at the Site content data
(under the Behavior category) from
Google Analytics to determine which
MedlinePlus pages are being viewed.
when portions of the
email should be
Can use the calendar format in Word.
topic.
Use MedlinePlus and other federal
institutes’ websites as sources for
information.
email bulletin in
Set up the content as it would appear in an
email.
Repeat this step as much as necessary.
in the email using the
Google Email
This will help track electronic traffic in
Google Analytics.
insert coded
hyperlinks into a new
email bulletin in
GovDelivery under the
appropriate health
If the email is going to multiple subscriber
groups, copies of the email must be created
and the subscriber list changed in the “To”
section of the email bulletin.
Phase
Three:
Distribution
of Email
select which variable
to test.
Can be anything from image/no image,
links/no links, etc. However, only one
variable should be tested at a time.
the email bulletin with
the appropriate
changes to the
No additional details needed.
subscribers by 2 to
This information yield the number of
subscribers that make up half the list.
“Subscribers Added”
option in the criteria
No additional details needed.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 15
Phase
Additional Details
and “After” option,
select the date where
the number of
subscriber list
matches half of the
subscribers on the
The number of subscribers will
automatically change as the dates are
changed.
email will go to the
Before and After
No additional details needed.
option at the bottom
to receive a draft
version of the email.
A test copy of the email will be sent to
editor of the email’s inbox. This will allow
the editor, and other selected users, to see
the email as it will appear in the
subscribers’ inbox.
correction to the
email bulletins in
No additional details needed.
subscribers
Click on the “Send Test” at the bottom of
the email bulletin and then click “OK”
button on the message that appears.
Phase 4:
Data
Collection
reports generated
through GovDelivery.
Look to see if there are any noticeable
differences in the reports especially the
open rate, click rate, and the number of
unique clicks for the links.
data in Google
The data will provide an in-depth analysis of
the Internet traffic to the coded URLs.
in Google Analytics for
the Internet traffic
data on the
MedlinePlus webpages
that were used in the
Compared with the campaign data from the
email bulletin, this will give an idea as to
whether the emails lead to more
engagement for the webpages.
together.
Will help to identify whether there is an
increase in engagement with the content
and identify which variable engaged the
subscribers more.
In addition, it would be beneficial to do additional A/B testing to determine what
other variables (e.g. time of day, day of the week, etc.) could lead to an increase in
engagement. The data from these tests, along with the data from the hallway
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 16
usability testing and the customized emails, can be used to create an email design
using the elements that were found to be engaging.
While not essential for the continuity of the project, it may be useful to
compare the different functionalities of other email distribution products, such as
MailChimp and Hubspot, widely used by other federal agencies. This comparison
can help determine which product offers the best features and services to meet
MedlinePlus’ goals of engaging their email subscribers; a brief comparison of these
products can be found in Appendix G.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 17
Bibliography
Atherton H, Huckvale C, Car J. Communicating health promotion and disease
prevention information to patients via email: a review. J Telemed Telecare
[Internet]. 2010 May 28 [cited 2017 Jul 1]; 16(4):172-5. Available from:
http://journals.sagepub.com/doi/pdf/10.1258/jtt.2010.004002.
Batesole B. Google Analytics Essential Training [Internet]. Santa Barbara (CA):
Lynda.com; 2017 April 28 [cited 2017 Aug 4]; Available from:
https://www.lynda.com/MailChimp-tutorials/Optimizing-Marketing-Emails/433519-
2.html.
Duquet M. Optimizing Marketing Emails [Internet]. Vancouver (BC): Lynda.com;
2016 April 7 [cited 2017 Jul 20]; Available from:
https://www.lynda.com/MailChimp-tutorials/Optimizing-Marketing-Emails/433519-
2.html.
Google. How a web session is defined in Analytics [Internet]. [place unknown]:
Google, Google Analytics; c2017 (a) [cited 2017 Jul 31]; Available from:
https://support.google.com/analytics/answer/2731565?hl=en.
Google. Session Duration, Avg [Internet]. [place unknown]: Google, Google
Analytics; c2017 (b) [cited 2017 Jul 31]; Available from:
https://support.google.com/analytics/answer/2731565?hl=en.
Granicus. A Guide to Content Marketing for the Public Sector. [place unknown]:
Granicus; 2017 (a). 20 p.
Granicus. About Us: The largest cloud solutions provider for government [Internet].
Denver (CO): Granicus; c2017 (b) [cited 2017 Jul 1]; Available from:
https://granicus.com/about-us/.
Granicus. Bulletin Analytics Report [Internet]. Denver (CO): Granicus; c2017 (c)
[cited 2017 Aug 4]; Available from:
https://insights.govdelivery.com/Communications/CommunicationsCloud/Sending_
Messages/Using_Reports_to_Track_Communication_Effectiveness/Bulletin_Analytic
s_Report.
HubSpot. HubSpot Software: Pricing Packages [Internet]. [place unknown]:
HubSpot; c2017 [cited 2017 Jul 31]; Available from:
https://www.hubspot.com/pricing/marketing#?currency=USD.
Launch Digital Marketing Campaign [Internet]. Naperville (IL): Launch Digital
Marketing Campaign; 2012 Sept 26 [cited 2017 Aug 17]; Available from:
https://www.launchdigitalmarketing.com/what-are-utm-codes/.
MailChimp. MailChimp Pro: The only marketing platform with true Multivariate
Testing [Internet]. Atlanta (GA): The Rocket Science Group, MailChimp; c2001-
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 18
2017 (a) [cited 2017 Jul 31]; Available from:
https://mailchimp.com/features/multivariate-testing/.
MailChimp. Pricing: Pro Marketer [Internet]. Atlanta (GA): The Rocket Science
Group, MailChimp; c2001-2017 (b) [cited 2017 Jul 31]; Available from:
https://mailchimp.com/pricing/pro-marketer/.
Moses, Tyler (Library of Operations Division, National Library of Medicine, Bethesda,
MD). Conversation with: Karla Blaine (Public Relations and Communications Office,
National Center for Complementary and Integrative Health, Bethesda, MD). 2017
Mar 22.
Moses, Tyler (Library of Operations Division, National Library of Medicine, Bethesda,
MD). Conversation with: Stephanie Dailey (Office of Communications and Public
Liaison, National Institute of Aging, Bethesda, MD). 2017 Mar 20.
O’Phelan M. 3 Steps to Awesome Email Design [Internet]. Denver (CO): Granicus;
2014 May 21 [cited 2017 Jul 20]; Available from:
https://granicus.com/blog/uncategorized/3-steps-to-awesome-email-design/.
Rowe A, Burridge L. Ten inbox secrets: What eye tracking reveals about designing
better emails. J Direct, Data Digital Marketing Practice [Internet]. 2012 Aug 13
[cited 2017 Jul 1]; 14(1):46-65. Available from:
https://link.springer.com/content/pdf/10.1057%2Fdddmp.2012.23.pdf.
Saghafi, Michelle. RE: Email Design Policies for Engaging Consumers [Internet].
Message to: Tyler Moses. 2017 Mar 29 [cited 2017 Jul 20]. [4 bullets].
USAJOBS Team. 10 Tips for Better Hallway Usability Testing [Internet]. Washington
D.C.: U.S. General Services Administration (US), Office of Products and Programs,
Technology Transformation Service; 2014 Feb 19 [cited 2017 Jul 1]; Available from
https://www.digitalgov.gov/2014/02/19/10-tips-for-better-hallway-usability-
testing/.
Wales VB. Email Marketing: A/B Testing [slide]. Washington D.C.: U.S. General
Services Administration (US), Technology Transformation Service; 2017 May 31. 33
slides: color.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 19
Appendix A: A Table of the Individuals Interviewed Regarding Email
Marketing Strategies
Table 4: Individuals Interviewed Regarding Email Marketing Strategies
Name
Office/Division/Title
Institution
Kathleen Cravedi
Director of the Office of
Communications and
Public Liaison (OCPL)
National Library of
Medicine (NLM)
Melanie Modlin
Deputy Director of the
OCPL
NLM
Mary Ann Leonard
Public Affairs Specialist
NLM
Wanda Whitney
Reference and Web
Services Division
NLM
Brooke Dine
Reference and Web
Services Division
NLM
Irene Liu
Public Liaison Officer
National Center for
Complementary and
Integrative Health
(NCCIH)
Karla Blaine
Public Relations and
Communications Office
NCCIH
Stephanie Dailey
Director of the Office of
Communications and
Public Liaison
National Institute on
Aging
Victoria Donnelly
Management/Program
Analyst
Food and Drug
Administration (FDA)
Michelle Saghafi
Public Affairs Specialist
FDA
Joshlyn Ross
Digital Marketing & Social
Media Strategist
Bixal
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 20
Appendix B: Hallway Testing Script and Notes
Hi, may I ask you what’s your name?
My name is ___________ and I’m from the National Library of Medicine. Do you
have 10 minutes to share your opinions on some email layouts with us?
Great! We’re asking people to give their feedback on a few email designs from
NLM’s health information website for the public, MedlinePlus, to find out which
design people like the most.
As you compare the two designs, I’m going to ask you to try to think out loud as
much as possible: to say what you’re looking at and what you’re thinking. This will
be a big help to us in terms of understanding your experience.
Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to
improve the appearance of our emails, so we need to hear your honest reactions.
Any comments about the designs are welcome.
If you have any questions, just ask them. I’ll answer them immediately if I can, or
after the test. And if you need to take a break at any point, just let me know.
Do you have any questions so far?
OK. Before we look at the designs, I’d like to ask you just a few quick questions.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 21
Test Variables A and B
Participant #_____
Introduction Questions
1. What do you do for a living? What keeps you busy?
2. Are you a federal employee?
3. Now, roughly how many hours a week would you say you spend looking
through email, at work and at home?
4. Do you subscribe to email listservs to receive health information from websites?
5. Have you looked at the MedlinePlus website before?
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 22
Testing Session (Testing less than 10 participants)
OK, great. We’re done with the questions, and we can start looking at things.
1. What do you think about the email designs?
2. Are they something you would like to receive in your inbox?
3. Of the two designs, which design do you like the best and why?
4. If you saw this design in your inbox, what would be the first thing you’d want
to do and why?
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 23
Test Variables C and D
Participant #_____
Introduction Questions
1. What do you do for a living? What keeps you busy?
2. Are you a federal employee?
3. Now, roughly how many hours a week would you say you spend looking
through email, at work and at home?
4. Do you subscribe to email listservs to receive health information from websites?
5. Have you looked at the MedlinePlus website before?
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 24
Testing Session (Testing less than 10 participants)
OK, great. We’re done with the questions, and we can start looking at things.
1. What do you think about the email designs?
2. Are they something you would like to receive in your inbox?
3. Of the two designs, which design do you like the best and why?
4. If you saw this design in your inbox, what would be the first thing you’d want
to do and why?
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 25
END TEST PORTION
Thanks, that was very helpful.
Do you have any questions for me, now that we’re done?
Thank you so much for your help today. Your comments will help us create better
email notifications.
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 26
Appendix C: Two Panel Newsletter Layout
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 27
Appendix D: Five Panel Newsletter Layout
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 28
Appendix E: Customized Email with a Blue Banner Layout
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 29
Appendix F: Customized Email without Blue Banner Layout
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 30
Appendix G: Comparison of HubSpot, Granicus, and MailChimp
Table 5: Comparison of HubSpot, Granicus, and MailChimp
Name
Pricing
Customize
Emails
Segment
Users by
Topic
A/B
Testing
Run Email
Campaigns
Additional
Features
HubSpot
• $2,400 per
month for
10,000
contacts
(HubSpot,
2017)
• One-time
fee: $5,000
(HubSpot,
2017)
• $10 added
for each
additional
1,000
subscribers
(HubSpot,
2017)
• Plan
Subtotal:
$11,600 a
month
(HubSpot,
2017)
Yes
Yes
Yes
Yes
• Send
suggestions
on how to
optimize
emails to
engage
subscribers
more
(HubSpot,
2017)
• Create
personas and
receive a
custom
persona-based
report
(HubSpot,
2017)
Granicus
$82,000 a year
for the library
Yes
Yes
Not
with
current
package
Done in
conjunction
with Google
Analytics
• Email
consultants
can help
create a
customized
email
template
• Generate
Subscriber
activity report
• Can remove
subscribers
who have not
opened emails
in over 6
months
LEVERAGING GOVDELIVERY FOR MEDLINEPLUS EMAILS CAMPAIGNS 31
Name
Pricing
Customize
Emails
Segment
Users by
Topic
A/B
Testing
Run Email
Campaigns
Additional
Features
MailChimp
• $3,950 per
month for
922,501-
930,000
subscribers
(MailChimp,
2001-
2017b)
Yes
Yes
Yes
Yes
• A/B testing on
up to eight
variations of a
single
campaign
(MailChimp,
2001-2017a)
• Email API to
send emails
(e.g.
automated
and
personalized
messages)
(MailChimp,
2001-2017b)
Note: Costs could potentially change if subscribers who have not opened an email
in 6 months are removed from the subscriber lists. Costs could also change if a
tailored email marketing package is ordered that provides services that are
appropriate for MedlinePlus staff and subscribers.