Chartbeat: AI-Generated Headline Suggestions

Overview: Chartbeat, the content intelligence platform for publishers, empowers media companies to build loyal audiences with real-time and historical editorial analytics across desktop, social, and mobile platforms. Chartbeat helps digital publishing organizations understand what, within their content, is keeping people engaged. Partnering with over 60,000 media brands across 60+ countries, Chartbeat’s real-time and historical dashboards, in-depth headline testing and in-page optimization tools, robust reporting, and more help the world’s leading media organizations understand, measure, and build business value from the attention earned by their written and video content. Problem Statement: Customers of Chartbeat already have a robust Headline Testing product, but we would like to expand it to allow AI Generated Headline Suggestions in order to save time and generate the most efficient headlines possible. However, we also know that AI has its limitations, and we wanted to mitigate any potential cases of AI issues in a field where user trust and honesty are vital. Users/Audience: Writers, Reporters, Headline Editors, and Homepage Editors who are curious about trying AI to help their workflow but may be hesitant to implement it.

Goal:

To seamlessly integrate AI generation into the existing Headline Testing Product in a way that is accessible and not intimidating to our users. How We Will Measure Success:

After a refining period of several months with various early access clients, we would ideally like to see this product become an add-on feature that can generate profit for the company. Proof-Of-Concept:

Because of complications with working directly in the more popular existing HUD (Heads-Up Displaly) product, we chose to execute or proof-of-concept in the less-often accessed "Headline Testing" tab. We hoped that the excitement generated by a highly requsted AI product would negate the problems associated by testing the product in a part of the website with lower traffic.

This is what the Headline Page test looked like after our changes, and the following text is how we explained the changes to our customers on our blog:

The top of the page visualization shows your total number of completed tests, broken out into two sections.

• On the left, the pie chart shows you the percentage of the time that original or alternative variants won (including AI-Suggested Headlines) as well as how often tests resulted in no winner.

• The bar graph in the middle takes the breakdown of original vs. alternative winner (including AI-Suggested Headlines) and splits it over the course of the past selected time range so you can see on a day by day basis which headlines won more often.

1. Beneath the top-of-page metrics, you can see details on all your active and past 50 headline and image tests.

   • We have added a new "type" column which displays whether a headline test was "User Generated" or "AI-Assisted".

2. If you want to run deeper analysis on your headline testing results, beneath the top right of the Completed Tests module you can export all your results data as either of two CSVs.

   • We have added a new "type" column which displays whether a headline test was "User Generated" or "AI-Assisted".

When the "Generate Headline" button in the top-left was clicked, the following modal popped up:

We hoped this modal would be mostly self-explainatory but in case it wasn't we had the following explanation on the Chartbeat Blog: 1. Navigate to the Headline Testing Dashboard: Here, you'll find the essential features for creating headline suggestions via the Headline Testing dashboard.

2. Locate the "Generate Headlines (Beta)" Button: Positioned in the top right corner, this button is your gateway to AI-suggested headlines.

3. Select Homepage, Search, and/or Social: We are exploring the ability to create headline suggestions for Search and Social along with Homepage suggestions.

4. Input and Generate: Paste the body text, current headline, and article section into the tool. The tool will then generate headline suggestions.

5. Review and Select: Carefully evaluate the suggested headlines to determine their suitability for testing.

6. Headline Testing in the Heads Up Display: Navigate to the Headline Testing area of the Heads Up Display to paste your chosen headlines and commence the testing process. Make sure and click the "AI-Assisted" robot icon which indicates that the headline was created using AI. Phase 2:

After months of successful testing and tweaking, we chose to went ahead and incorporate the modal we designed in the existing HUD product mentioned earlier. This way it would be more accessible to people as the HUD is the usual starting point for most people in this workflow.

Here is how this tool was explained on the blog:

Navigate to the Heads Up Display: Here, you'll find the essential features for creating headline suggestions in the Heads Up Display. 

1. Headline Testing: Select a headline on the page where the test will run, and click on its Heads Up Display pin. Next, toggle into the Testing tab in the pin card.

2. Generate Headlines: Click the "Generate Headline" button. This should automatically create headline suggestions. If we aren't able to access the article content to generate headlines, you’ll be prompted to paste the article’s body text into the tool. The tool will then generate headline suggestions.

3. Review and Select: Carefully evaluate the suggested headlines to determine their suitability for testing and then select the + sign.

4. AI Robot: In order to track which headlines are "AI-Assisted" the robot icon next to the headline you have chosen will be automatically selected.

Monitor and Analyze: Visit the Headline Testing results page to monitor the experiment live or view results from past experiments.

Success:

After months of refining and tweaking, this product made our early access customers happy enough that they expressed interest in having this be an add-on product.