Success of a 1st Year Program

Sarah Gorman, Senior Manager, Digital Marketing, HSN

Resources Engage Success of a 1st Year Program

Video Transcripts:
TV Land, so put it together for Sarah Gorman from HSN.


All right. So you killed—can you guys hear me? Yeah? Oh, there we go.

Hi there. So Eric, you killed my potential temperature read on the room, because I use the HSN acronym. But not everybody knows what that is. So just a show of hands. Does everybody know who HSN is? Home Shopping Network?



All right.


How many of you shop there?


The only person that raised their hand was a person on my team.

That's great.


Just kidding.

So HSN. You know who we are. We're based in Florida. Originated in 1977. We've been selling products starting on the radio and now have evolved to a digital commerce platform.

So about a year ago, we signed on with Localytics. And what I'm going to talk to you a little bit about today is our story. We'll cover that first, here. So we've done a lot. We've accomplished a lot. We've tested a lot. So we're going to walk you through, high level, what that first year really looks like. All right. So that's me. Success of our first-year program. So high level, we did a lot in our first year.

So it's November 2017. And November 2016 we launched our app with the Localytics SDK on November 18. So everything we've done had been in one calendar year, which, it's been a lot. It's been the story of Tiffany and I's lives in the last year. So we started with our integration and two new hires. So I was a new hire. My first day at HSN was October 1. So shortly thereafter, we started our integration. And shortly after that, I hired Tiffany.

So we also created a new inbox. We integrated our data with our internal CRM and datalinks,

applied universal control groups. We launched an entirely new brand new-- iOS app, internally built. We created new app technologies, created an onboarding strategy, a retention strategy. We leveraged predictive modeling, custom reporting, and we have launched over 450 campaigns. So that's a lot. It's a lot that we've done.

So I'm going to walk you through our path to success over the last year. And it's sort of broken out into six components. So we started with our integration. We move on to foundation and planning and how we really wanted to set the foundation for the future. Staffing for success. Optimizing for growth. Engagement and retention. How do we measure that success, and then what's next? How do we look forward into our second year? So our integration. Standard six-week integration. I think what was unique about our integration with Localytics was we had some of our Localytics folks on site, some of which are in the room here. Thank god for Caleb, because he's like family. Yeah.  Caleb is definitely family to the HSN crew. And he was on site, along with Harley and a few others, to help us step-by-step in that integration and really work very closely with our developers. And that was really crucial to set the stage and make sure everybody was familiar with what we were trying to do and what we were trying to implement. It required a lot of teams. So project management led the entire project.

Product management was involved. Our digital systems, our IT team, digital, marketing.

There were a lot of different groups involved, and it required a lot of breaking down silos, because a lot of those teams work in silos, just on their own. And so this integration, because it was so large for us  and new for us, required us to work very closely together. So that was just the beginning. So we launched our SDK on November 18th, and then began all of the work. So we set a very solid foundation. So we defined what our key strategies and KPIs were. So we broke this out into four different groups. We wanted to activate, nurture, monetize, and retain.

Standard strategy stuff, right? And how we measured each one of those strategies was around session engagement, session length, session usage, app version, upgrades to

apps, user state, revenue, conversion, top of the funnel all the way down to the bottom of the funnel. We also identified what those key behaviors and attributes were that we really wanted

to be able to target on. And I'll talk a little bit about personalization and how we also leveraged and tested into that in our first year. And then I was learning the dashboard. So we all had to do that, right? It's pretty easy, but it takes some time. And I think it takes a special person to hit that Activate button, right? We all have to train ourselves. We were just talking earlier about who has the authority to hit Activate in an organization and defining that role and making sure it's consistent. And we were also talking earlier a little bit about second guessing yourself. Because I initiated all of our campaigns to start. And it was almost like you have to spell check yourself, make sure everything is accurate, which is very daunting in the beginning. It's very tactical. But there is a lot to it, right? And getting yourself ramped up and getting your organization ramped up. It's really important to spend the time to set that foundation. So then we started planning. So we became comfortable with the tool and built a plan. And most of the Localytics folks know how much of an Excel guru I am and how much I love Excel. So I built a plan in Excel, which unfortunately, my legal department made me remove from this presentation. But it was a very detailed document that included all of the aspects and components to all of our messaging. So high level strategies down to the tactical level and what we were building to support each one of those strategies, all the way from what the message

said, what the word count was, what our key value pairs were, what our deep link was, time of day. And then we appended that with our response data. So taking all of the open, click, click through, conversion, revenue, gross profit. And so it really became an aggregator for all of these different things, in the planning stage all the way through the conversion funnel. And that's what we started to use to build our plan and to help influence our future messaging.

All right. We initiated dozens of A/B tests. So as any marketer should, test into everything. It was new for us, so we wanted to test into messaging placement. Do we include a header? Do we just go straight into the body of a message? Do we include emojis, personalization?


I know you're all going to be very surprised to hear this, but marketing works, and personalization works. So we tested all of that and tested into it, just to prove to the

organization and to validate a lot of those things. Which as an enterprise organization, sometimes we're held back a little bit by things like that, because we do have to prove

it to all of our leadership. So that was really important for us. We also created a QA process. So I know that there's a beautiful summary screen on the last page of those campaign setup pages in the dashboard, but QA was really important for us, because we were not going to send anything wrong. And some of our competitors, who we monitor very closely, we also monitor sometimes when they have errors. So it's really very important for me and for my team for

the organization to make sure that we didn't have any of that. And I'm proud to say that after

450 messages over the course of 12 months, we have not deployed anything wrong, which is really a very hard thing to do. The only limitations that we had were around our inbox, which I'll

talk a little bit about later. So being a highly promotional organization, we had a lot of offers,

and offers change. And leadership changes, and direction changes, and strategy changes. So sometimes when we put something into market, we let it live for a period of time, it can become outdated. So that's where the area of opportunity was for us, and the challenges

that we're still overcoming. Then we move on to staffing. So I said my first day was our first day with Localytics. And I was handed the mobile program, and we just sort of ran with it from there. So I am the senior manager. I then realized that we needed to staff to support this program. I think we definitely have a lot of opportunity to grow still, but we had to define

what that role really was and how we created our entire team. Some of the challenges for

that, at least in our market, being in Florida, is mobile is still so new and getting candidates who

have experience in mobile was a little bit challenging for us. So we knew that going in, and we knew that if we could get somebody who was eager and somebody who was willing to learn, that we could train them in that space, because I was learning to and the organization

was learning as well. So then came along Tiffany, thank god. Success! We found her. So we hired Tiffany, and we started building campaigns and talking about HSN strategy, our KPIs and our objectives. And she ramped up really quickly and has been deploying messages ever since. But staffing to success and staffing to the expectations of the organization was really important

for us because we had to make sure that we had the manpower to support the expectation of

the enterprise organization. And I think that there's still a ton of opportunity for us to learn and a ton of opportunity for us to grow. We've set the foundation. We have a solid team, and now we're just continuing to build blocks on top of that. Then we move into optimizing for growth, engagement, and retention. So I've got a few use cases that I'm going to share with you all today, starting with the app technology that we built. So we built a first ever app-only offer

or app-only coupon, which was new for HSN. And we needed something to differentiate our app, because our app and our Wapp and our desktop experience are all identical. So why does she need to download the app? And yes, we use that pronoun to describe our customers, because they're predominantly women. So we needed to differentiate the value of the app. And so we created some app-only technologies. So that required a variety of teams to be involved and develop the capability for us to create offers available only in the app. So from an omnichannel execution perspective, this was really important for us, to make sure that we

delivered on this message across all of our channels. Now, we're a unique organization in that we have a television network, and we have a few channels, and we're also in the digital space. So we started with email. We started with a very targeted list of our key customers, our best customers. We started with targeting them. Now, we didn't know whether or not she had the app yet, because we hadn't integrated our data from Localytics back into our internal CRM data lake. So we were just targeting our best customers via email. We created app-only landing pages and app-only experiences, including in-app inbox and push messages. Standard campaign, starting in email, really evolving into the mobile space. Small, but really starting with email and trying to expand into the mobile space, which I have responsibility for the email team at HSN as well. So when we talk about omnichannel, that's the immediate-- from our perspective, the immediate opportunity. I think we have a ton of opportunity to expand to all of our channels, but definitely, the most immediate opportunity for growth. We then moved on to create an entirely new app. So our team redesigned the iPhone app, which took--I don't know--8 to 10 months of the last year. And it was a large effort, and so we were tasked with promoting that. And so we did that in any standard way, right?, our own landing ages. Then we started to get into email. And at this time, we had data integrated from Localytics into our internal CRM system, which I'll talk about a little bit later. So we could do some targeting here. And what we really wanted to do, because we don't have deep linking enabled yet

in our e-mails-- efinitely an opportunity for us in 2018--but we were able to identify if a customer had the app, which version of the app she had, and if she had made a purchase in the app. So now we're taking that data, and we're filtering that in and using that in our email targeting. And that's how we were able to one, deep link for those of our customers who we knew had the app, and two, target an offer only to people who had not made a purchase. So we weren't sending an irrelevant message to customers who couldn't use it or were ineligible. We had package inserts. So we're a retail company. We send packages to customers all the time.

So we have the ability to customize package inserts and package labels and things like that. It just seems a natural fit to really try to drive top-of-the-funnel engagement and users into our app. Of course, Localytics initiated push and in-app messages supporting the app and the new launch of the app, and then television, which believe it or not, even though we started in television and eventually expanded into the digital commerce space, television is a very

challenging channel for us, and it's very challenging to get airtime. So we created commercials and spots that would lay over the actual television and things like that, that we started

to promote on air, which was great, because we were seeing a spike in downloads every time [AUDIO OUT].

So those are just two use cases to get us started. The next is our onboarding strategy. So activating and nurturing new users. We created some automated drip campaigns and really started to pull our customers out into that onboarding phase and keep her and our customers

focused on what we really wanted to educate them about during that onboarding phase. So we started with--excuse me. Some of our offers. So our app-only technologies for a targeted user base, targeting customers who we know hadn't made a purchase in the app. We also would put those messages in the inbox. So we are a very heavy inbox user. We send the majority of our

push messages to our inbox and try to train our customers to go back to that inbox, sort of

her source of truth, of all of the offers that are relevant for her at any given moment in time. We expanded that to push messages things that were unique to HSN, like a Today's Special,

being the one product of the day that was the most important or the best deal. And also things

like Flex-Pay, that are also very unique to our company, that we offer to consumers. But educating her through that journey in the first couple of weeks of her experience in the app,

to get her familiar with the lingo that we wanted her to learn about HSN, and the behaviors that we wanted her to adopt. So we wanted her to adopt coming back to her inbox and seeing that little red badge to know that there was a message there for her in the app. We did then after move on to some of our targeted-- or excuse me. Our mass messages after our onboarding strategies. So our onboarding strategy really is nurturing her in the very beginning, and then our mass messages, which come after that, we suppress on all or our onboarding customers. And we're really trying to customize and drive the behaviors that we want. We want her to enable push. We want her to comeback and use the inbox. We want her to make repeat purchases and valuable purchases. The third use case that I've got here is growing the value

of our customers. So encouraging them to make those repeat purchases and reminding them to come back. And we use we use tagging, so our profile and behavior attributes, to really get

into some liquid templating to customize some of our messages. So again, we tested this very early on. First name personalization. We've got experience in email, we know that works. Is that transferable in the mobile space? Test it, yes, no, great. Implement it moving forward. That's how we evolved in our initial stages. So we do a lot of first name personalization. Engagement is next. So again, I mentioned the inbox. So we do a lot in the inbox and integrating our push messages with our inbox. And so it really was important for us to be able to replicate and/or

support those messages that were showing to her as a push message and display them in her inbox, so that she has a source, and she can come back and use them and see them. And then the next phase of that was how to measure that. So we implemented some additional tagging

around our inbox messages to really be able to quantify is it helpful. And we were able to do that. So we knew that users who engage with an inbox message--and engagement for us was

just a click on the message in the inbox--users were more likely to make purchases in the app by five times, which is huge. It's huge. If we can get her to engage in the app we can get her to

engage with the inbox, she's going to make five times the amount of purchases. So great insights and great learnings for us. Push opt in. I think that's something the majority of us are doing, right? We want to be able to speak to our customers. So we have to ask permission for that. So we deployed a number of messages, mostly in-app messages, trying to encourage our users to opt in to push notifications. This is just an example of one of them. We saw an increase of an average of 56% in our push enabled rates across all of our platforms. Retention. So we've on boarded her. We're trying to engage with her. And we're trying to avoid churn, so we want to make sure that she doesn't leave us. So the evolution of our retention campaigns, we started small. We started with, hey, we've missed you. Which is a pretty common one, I get it. But hey, we've missed you. Then we added first name personalization, which again, we've proven, and we knew that it worked. So we added first name personalization, and we saw a 10% increase

in seven-day retention. And then we applied some additional tags with liquid templating. So storefront, which is a category for us, like shoes or beauty. So we added first name and storefront. So now we're getting even further into personalization and the experience that the customer wants, because we want to be talking to her about what's relevant to our customers. And when we did that, we saw almost a 14% increase in seven-day retention. We also have-- since we created, I guess, a while ago, we did deploy the predictive modeling. So we are now

leveraging Localytics as predictive modeling tool to help us identify users who are more likely to churn, and that's what we're using as our targeting for that specific message, inclusive of first name and storefront. So then we move on to measuring success. So data is critical. We all know that. We've been having a lot of conversations about data and why it's so important. For us, it began with aggregating the data of all of our campaigns, because we knew we were going to send so many of them that we had to have a source for that. So we actually worked with Localytics to build a custom report that gets fed to us every morning that is an aggregation of all

of our campaigns over the last 60 days across all of our app keys. And we look at to click through conversion and revenue, and we use that data to supplement the plan that I spoke about a few slides back. And then we move on to incrementality. So just quick read on the crowd. For those of you who are held to a revenue objective, are any of you using incrementality to prove lift in your messages? No? OK. So I'm going to walk a little bit through incrementality. So this is really important for HSN and our organization. So our definition

of incrementality is would she have performed that behavior if we didn't send the message. And you think you'd be able to get to that with just a standard test and control cell. But our organization believes we have to go a little bit further, because we want to make sure that we're not just shifting attribution from one channel to another. Because right now, we're focused on last click attribution. So maybe she just clicked on the push message, and that was her last click, and so we're going to attribute the revenue there, but we're just shifting revenue

from one channel to another. So we were tasked with proving incrementality. And so quick mathematics on how to get to calculate incrementality, just to give you some context. So we were looking at our total audience size, responders and net sales. We looked at data across our

test cell and our control cell. And these are very small numbers, just to prove a point. So if our test cell had an audience of 25 people, with 2 responders and $100 in net sales, and our control cell had an audience of 5 people, with 1 responder that generated $5, we are challenged with proving the behavior of the test cell versus the control cell and proving whether or not sending that actual message showed that we are actually driving behavior. So how we did that was we

normalized the control. And so if you talk to statisticians--they make my head spin--but I had to get a very basic understanding of this for myself, my team, and all of my leaders to really talk about how are we proving incrementality, so that we're not just a channel that's shifting last click attribution from one to another. And so we apply a multiplier to the control cell to normalize it or bring it up to the level of the target cell. Because you can't just calculate

the difference between a target cell and a control cell. You have to normalize the control. And so adding a multiplier to the control cell, you can then calculate the difference. So after we normalize the control cell for responders, we take the test cell, which is 2, and our normalized

control cell, which is 5, which actually proved a negative lift. So this is saying we would have

gotten more responders if we hadn't sent the message. However, if we do the same

calculation for net sales, after we've normalized that control and calculate the difference, we've

proven that if you take the test cell of $100 and the normalized control cell of $25, there was

an incremental net sales lift of $75. So that's how we as an organization prove incrementality and are tasked with making sure that we do so again, so that we're not just shifting attribution from one channel to the other. So what's next for us? Omnichannel, everybody is talking about it. Everybody wants to do it. Data is critical for us to do that. And our next step is to get

data fed from our internal data lakes to Localytics so that we can do more personalization and more targeting. It's also leveraging site behavior and preferences and next best offers and some

of those types of things, and also, of course, machine learning. So Henry talked a little bit about machine learning.