Measuring success and acting with data – What counts?

Updated 19 Sep 2026

Defining Measurement

Measuring success is deciding, and undeciding, what matters. Every metric in what follows was the right answer once. The harder half of the job is noticing when it stops being one.

So how do we measure success? There are a couple of ways. One is pure audience metrics, pageviews. But that mostly tells you if you wrote about a topic people are already searching for. It also reflects things like whether your headline was strong, your promo image worked, or the story got prominent placement on the homepage. To complement that, we look at how much time someone spends on a single story. We call that attention time. We have built models to estimate how much time we would expect a reader to spend, factoring in things like story length, visual content, and where readers are entering the article. We refer to this as an attention time score.

Anna Dubenko, The New York Times

Every newsroom in this project measures something, and no two of them measure the same thing. Three questions set the baseline.

1. Purpose: Why are we counting, and what decision depends on the answer?
2. Substance: How deep does the relationship have to go before we call it success?
3. Practice: Who sets the goals, and who keeps the definitions honest?

The second one is newsletter subscribers. A pretty quantitative number, obviously not really measuring engagement, but our theory is that we know through other audience data points that our newsletter subscribers are much more likely to engage with our journalism.

Alexandra Smith, The 19th

The Arc of Measurement

Measurement has gone through five eras. Each begins with a decision about what will count as success. Then the technology shifts underneath the metrics and the meaning drains out. The old metric is rarely dropped. A new one is stacked on top of it, and the next era starts. Metrics can stack. North Stars cannot, and a newsroom has to choose the one it steers by.

Pre-digital era (through 1994). The first systematic count of newspaper circulation in Britain was a by-product of taxation. From 1712 every sheet had to be physically stamped and the duty paid per copy, so the tax returns counted newspapers whether or not anyone in the trade wanted them counted. By the nineteenth century the duties had a name and an opposition, and publishers were campaigning against what they called the taxes on knowledge.

Advertisers built their own instrument a century later. In 1914 advertisers, agencies and publishers together founded the Audit Bureau of Circulations in Chicago to stop false and misleading circulation claims.

Neither number counted readers. Circulation audited copies that had been bought. Nobody knew who had read what, and nobody expected to.

Early web era (1995–2005). Every request to a web server left a record in the log, with a timestamp, an IP address and the page the visitor had come from. Reading the logs took software, money and somebody in the IT department willing to spend time on it. In 1997 a large company could still need a full day to process its own traffic.

Hit counters were the easy option. The counter was a small picture of a number, and the picture was stored on the counter company's own machine rather than the publisher's. So every time a visitor loaded the page, the request for that picture went to the counter company, and it was their records that did the counting. Through the late 1990s the same arrangement spread to proper analytics tools, which also gathered the data on their own servers and showed the publisher a report.

Counters had their moment. Then richer pages and smarter browsers broke the hit itself. A hit was one request to the server, so while a page was a single file of plain text, a hit meant a page. Then pages started carrying images, audio and video, and one visit generated a dozen requests or more. At the same time browsers began caching files, so repeat views stopped generating requests and disappeared from the log entirely. The number was too high and too low at once, and it went on being reported either way. The industry moved to counting pageviews instead.

The unit of sale arrived in the same window. HotWired ran the ad that became known as the web's first banner in October 1994, and AT&T paid a flat $30,000 for three months of it, the way you buy space in print. Eight months later CNET began selling banners with guaranteed audience deliveries, which meant somebody had to count impressions to prove the guarantee had been met. From then on the pageview had a price.

Platform era (2006–2018). Google bought Urchin in 2005 and turned it into Google Analytics, free to anyone who wanted it. Ad exchanges changed how the space was sold. A single impression went to auction every time a page called for an ad, and the auction closed in the milliseconds while the page was still loading.

By the time the Tow Center published Caitlin Petre's The Traffic Factories, a decade of development in analytics tools, and of a business built on impressions, had made metrics ubiquitous in news organizations. Newsrooms were genuinely excited about the granularity, and about how much they could now know about story-level performance and user behavior. But the worry about what data was doing to judgment and to the mission was real. Metrics-driven decision-making, leaning heavily on pageviews, was in the DNA of data-natives such as Gawker. At the New York Times metrics were still peripheral, and editors told Petre that topping the most-emailed list was not what got a story onto the home page. That decision rested on which stories were the most interesting, or the most important, for their readers.

Measurement evolved where the money was. Bucks for impressions.

Then one reader stopped being one number. The desktop browser was joined by an app, a tablet, and half a dozen platforms with browsers built inside them. Mobile and tablet devices passed desktop in October 2016, reaching 51 percent of internet use worldwide.

Analytics counted browsers and devices, not people. As early as 2007 Comscore was warning that counting cookies could overstate a site's audience by a factor of two and a half, because people delete cookies and use more than one browser. One set of eyes could show up as ten unique visitors in a month.

Then the devices multiplied, and a publisher stopped having one audience. A Knight Foundation study tracking 9,000 smartphone users for two years found that USA Today reached ten million people on the mobile web and 2.6 million in its app, and that the app users spent about an hour with it over a month against four minutes for the web users. Jeff Sonderman put the choice to publishers directly: who is more likely to subscribe, the mobile web user who spends four minutes with you in a month or the app user who spends over an hour, and who is the more valuable advertising audience, the 2.6 million app users or the 10.2 million on mobile web?

We know that roughly 70% of unique visitors don't return within the time period that we can track them, which is typically 30 days. So saying we had however many millions of unique users in a given month — even in a really high month driven by a major news event — means little if only a handful of them come back with any regularity, if at all. That is why we focus on readership, frequency, and deeper engagement. Metrics like newsletter signups, email database growth, and app usage tell us more about loyalty. In a subscription model, unique users can be a diminishing metric because most will not return.

Ross Maghielse, The Philadelphia Inquirer

By the mid-2010s, a noticeable shift began as leading publishers and industry thinkers pushed back against the dominance of pageviews. Raw traffic volume was increasingly seen as a poor measure of journalistic success or business sustainability, especially with the rise of digital subscriptions. A user who bounced after a few seconds held far less value to both publishers and advertisers than someone who spent several minutes engaging with an article. In 2014, Chartbeat's CEO Tony Haile suggested that the industry pivot from counting clicks to measuring attention, arguing that time spent actively engaging with content was a more meaningful metric of value.

We measure the number of UPVs, which is unique page views, so the number of stories consumed. That's not the same as page views. A gallery counts as one. And then we measure time spent. That's really the engagement one.

Sarah Marshall, Condé Nast

By prioritizing engagement, newsrooms aimed to redefine success around deeper reader satisfaction. For example, The Financial Times introduced a metric called Quality Reads, which tracked whether readers consumed a meaningful portion of an article, not just clicked on the headline. The New York Times went further and modelled how much attention a story should expect to earn, then measured against that. It also began treating sharing as evidence in its own right, counting whether subscribers thought a piece valuable enough to pass on through their monthly gift links.

Post-platform era (2018–2023). Apple introduced tracking prevention in Safari in 2017 and by 2020 was blocking cookies for cross-site resources. Firefox turned the same protection on by default in 2019. Google announced in January 2020 that Chrome would phase out third-party cookies within two years, delayed it repeatedly, and in the end kept them.

Even though the long-predicted cookiepocalypse never came, many publishers built strategies and capabilities to collect first-party data. For a while, CDP (Customer Data Platform) was the acronym of the moment. FIPP's playbook for publishers, published in 2022, reported that most media companies could recognise fewer than 3 percent of their online users as known. The work was starting from almost nothing, which is why it took years rather than a quarter.

Publishers turned their focus to known users. Anonymous measurement had been unreliable long before the cookie was threatened. A reader who logs in is the same reader on a phone and on a laptop, so they get counted once instead of twice. Their history goes back as far as the account does, not as far as the cookie survives. Piano found the subscriber conversion rate ten times higher for known users than anonymous ones, and by 2023 the Guardian was rolling out a registration wall. Thomas Baekdal made the measurement case rather than the advertising one, telling Digiday you cannot do good churn analysis without first-party data.

After decades of chasing scale and immediate clicks, many news organizations in the late 2010s and early 2020s pivoted to a more sustainable vision of success: building loyal audiences and long-term relationships. After pressuring a generation of journalists with an intense focus on pageviews, the pivot to reader revenue has shown that quality matters more than scale when it comes to subscriptions. A study by the Lenfest Institute found that readers who view eight or more articles per month are significantly more likely to subscribe.

In addition to just total subscription growth and retention, we also track what we classify as story driven subscription conversions. That means subscriptions that result directly from someone consuming Inquirer content and hitting our smart paywall, as opposed to converting through marketing promos, ads or other sign-up mechanisms. In the newsroom, we focus heavily on the stories and coverage areas that lead people to subscribe most consistently.

Ross Maghielse, The Philadelphia Inquirer

The same study reported that, for the typical publisher, only 4 percent of unique visitors qualify as "regular readers," meaning they view more than five articles per month. In this context, the growing focus on engagement becomes clear.

AI-discovery era (2024–now). Search has always been based on crawling, so for as long as there has been traffic on the internet, there has been a bot reading the page first. Publishers could always refuse. Robots.txt has existed since 1994, so the ability to prevent crawling is as old as the practice, and it has always worked by asking rather than enforcing. Few refused, and why would they. Publish, wait for the crawler to find the new piece, get visitors in return. That exchange of access for distribution gave crawling a positive note for thirty years. AI gave it a new one. Not all crawling is the same.

What became measurable was the crawling itself. Cloudflare began publishing crawl-to-refer ratios in 2025, counting how many pages an AI company takes for every visitor it sends back, but the measure is unstable. The same ratio for the same company has been reported at wildly different values depending on the window and on what counts as a crawl, ranging from roughly 70,900:1 down to 2,237:1 inside about thirteen months. Crawl purpose matters, because a crawler gathering training data was never going to send a referral. Trusted Reviews found OpenAI hitting its site 1.6 million times in a day, taking it down repeatedly, while ChatGPT sent about 300 clicks over the same day. Chris Dicker, CEO of parent company Candr Media, said there is no value exchange.

Cloudflare launched pay-per-crawl in July 2025, then a year later said charging per crawl was not enough and began moving towards paying when content actually appears in an answer. A crawl is easy to count, but use is not. Paying for it would mean knowing which sources shaped an answer, and apparently nobody can tell that yet. Being cited is what publishers are left asking for instead. It may tell you that you had interesting, high-quality content. It also means the platform now has a copy.

For the majority, the ratio is evidence to use as an argument, not a goal or an asset that can be turned into business. Most are already acting on it, either by adapting their content strategies to the new normal, or by deciding which bots are invited and which are not. By early 2026, 79 percent of the hundred biggest news sites in the UK and US were blocking at least one training crawler and 71 percent were blocking retrieval bots. The strongest can use it as leverage when cutting licence deals. People Inc's CEO Neil Vogel told investors that being able to block almost everyone through Cloudflare is what brought AI companies to the table, because they then have to pay.

Chartbeat's 2026 data has ChatGPT referrals growing more than 200 percent year on year while AI sources stay under 1 percent of all pageviews. Nieman Lab found AI sources accounted for 0.7 percent of its pageviews. Despite the small overall effect, news sites of all the categories are getting the highest total pageviews from AI platforms alongside the lowest engagement per article. Chartbeat reads that as AI surfacing many different articles for factchecking and context, with readers getting their answer and leaving.

What the arc shows

Every metric in this chapter broke the same way. The hit meant a page until pages started carrying images. The unique visitor meant a person until one person carried three devices. The pageview meant value until a three-second bounce and a five-minute read counted the same. In each case the number kept being reported after the thing it stood for had moved. It is hard to come up with metrics that serve the business and the newsroom at the same time. Unlearning them, once the ecosystem has shifted, is harder.

Choosing what to count

What counts depends on the strategy, the business model and where the newsroom is in its life. Audience teams combine the most relevant metrics to better understand the complex nature of engagement, and two newsrooms working on their purpose can both be right while sharing no metrics at all.

The Financial Times is a clear example of a learning organization, with metrics that keep changing as the ecosystem and the organization evolve. It has changed its North Star twice since 2016, moving from RFV to lifetime value in 2020 and then to Global Paying Audience in 2023. Each built on the last rather than replacing it. Above an RFV index score of 18.2 the FT counts a subscriber as engaged, and engaged readers cancel 10 percent less often. For the FT, the North Star led to action when they built myFT, a way to follow chosen topics. Merely becoming a myFT user raised engagement by 86 percent.

Mediahuis went another way in its Belgian newsrooms, reporting the aggregated time all its readers spent on an article. Part of the reasoning was that editors understood the number and accepted it, and that time is harder to win with a misleading headline than a click is. The time a reader spent also predicted whether they would convert and whether they would renew.

The New York Times identified attention time as its metric in 2024. The newsroom wanted deeper engagement and the business wanted more time on the platform. A raw number was not enough. The Times set it against context, using previous periods and times of day, similar stories and sections, stories given similar promotion, reader type and traffic source, and seasonality. It also modelled how much attention an article of a given length should expect and plotted each story above or below that line.

How it was presented mattered as much as what it measured. The Times decided against treating the data as a scorecard and against marking stories as over- or under-performing. No red text, no green text, everything in black. Hayley Arader, executive director of data and insights for the newsroom, sums it up by saying the newsroom is "data-informed, not data-driven," with editorial judgment first.

The newsroom receives no subscription data at all, and no data on churn. Part of the reason is that the strongest predictor of whether someone subscribes to the Times is whether there is a sale on that week, so conversion figures handed to editors would describe pricing while looking like they described journalism.

I am not allowed to give the newsroom any sort of subscription data really at all, but definitely not any data about what kinds of stories convert readers into subscribers, or churn, like unsubscribes. We don't share that at all. And that's deliberate on the part of the publisher. He wants to be able to say honestly that the newsroom does not make coverage decisions based on business imperatives.

Anna Dubenko, The New York Times

From metric to decision

Metrics without actions are numbers. The New York Times audience team sends a note to the masthead every morning, before the news meeting, setting out where the opportunities in the coverage are, then follows up the next day on how those stories did. Reports run daily, weekly and monthly.

The audience team isn't an assigning desk. You might be talking to some news organizations where the audience center has reporters that they run. We don't. We think about audience as a partner to newsroom leadership and desk leadership in providing data that is actionable. Not just kudos, not just a report, but finding the data that tells us what we should do next.

Anna Dubenko, The New York Times

Getting a metric used takes explaining. When the Times put attention time at the top of every story's dashboard, it also circulated materials and wrote an FAQ so the whole newsroom understood what the number meant. The managing editor wrote to the staff about why it mattered, and audience editors worked with journalists directly. According to Arader, it worked because it came from the bottom up and the top down at once.

The Future of Measuring Success

As the relationship with the audience brings in the money instead of single hit articles, the focus of measurement shifts from the article to the user. Repeat use, habit, trust, loyalty, lifetime value. Janis Kitzhofer, who runs editorial insights at Axel Springer, explains why the pageview outlasted every argument against it: it has an immediate advertising revenue impact and translates easily into money, while a relationship or a return visit is much harder to put a dollar sign behind.

Trust is the hard one, and Kitzhofer's answer is not to measure it directly. He is building a pyramid with three layers. Original journalism at the base. Engagement in the middle, meaning scroll depth, video playback rates and expected reading time. Relationships and commitment at the top, meaning frequency and retention. Trust is what you infer from the stack holding together, and the journalism sits underneath it rather than beside it.

Where the pyramid points is at what happens next. At BILD most readers arrive directly at the home page, so Kitzhofer watches what they do after that, whether they sign up for a newsletter, play a video or take a quiz. A visit that starts nothing is worth far less over time than a visit that starts a relationship. As he puts it, everyone would now rather have a reader coming six times a week for five minutes than once a week for half an hour.

How many metrics it takes differs from newsroom to newsroom. Kitzhofer has one rule for any metric: it has to be real time and simple enough for everyone in the newsroom to understand, or it will not change a single decision. Sofia Delgado, formerly director of audience at Metro, built a golden metric combining volume, dwell time and engagement, and defends compound metrics on the grounds that a newsroom cannot watch everything, so one number that does most of the job beats twenty that nobody reads. Her condition is that the business has to have done the work to know it is the right one.

Most audiences have got used to personalisation built on behavioural data. What you click is what you get, learned from social and now standard on the home pages of legacy media too. Personalisation has long meant mainly news selection or dynamic paywalls. It is now being framed as an answer to news avoidance and to differing user needs, by serving the same journalism in multiple formats. The Reuters Institute reports publishers looking beyond the article and making their content liquid so it can be reformatted and personalised, and reporting video and audio formats as the clearest areas of increased investment.

For now the publishers want this more than the audience does. Interest in AI-driven news personalisation runs below 30 percent for every option Reuters tested across 27 markets, and the most wanted were the ones that save time rather than the ones that choose on your behalf. Some already worry that algorithmic recommendation will mean missing out on important stories.

Missing out may be the smaller problem. Personalisation might do more damage to the audience's sense of fairness. In 2026 Nieman Lab reported subscribers finding a line in their renewal email saying the price had been set by an algorithm. Hearst confirmed it runs dynamic pricing across all its newspaper markets at renewal. The Financial Times personalises the price but not the journalism. Fiona Spooner, the FT's managing director for consumer revenue, gives the reason: you do not want to lose the serendipity that comes with editorial influence. The line publishers draw and the line readers would draw may not be the same one. A personalised article is a service. A personalised price, set by a model that knows where you live and how much you read, is not.

The same capability points the other way, and the examples are harder to find. After years of spending the best offers on strangers, the algorithm could just as easily thank the reader who has been there six years. Who rewards loyalty with a price cut first?

For thirty years the focus was on gathering the data around the product. Some publishers are now putting that data to work. At an INMA summit in 2026, Robert Whitehead described agents watching every subscriber from the moment they pay and responding without anyone deciding to. Débora Pradella of Grupo RBS reported running over 300 email and 200 WhatsApp campaigns in a quarter with a team cut by 40 percent, and said the results held level against human-made campaigns.

The Financial Times has run one of the most-watched experiments. Its AI paywall reads demographic and behavioural data, how often someone has hit the paywall, where they are and whether they have lapsed, then decides which product to offer them. Among the audience segments exposed to it, conversion rose 290 percent and lifetime value between 7 and 10 percent. Spooner has described the same capability being turned on retention, catching people who are thinking about cancelling and offering them a newsletter or a cheaper product instead.

The FT itself admits the results might look better than they are. The paywall only reaches the 30 to 40 percent of readers who consented to tracking, so it may be working on people who were going to subscribe anyway. Graham MacFadyen, the consumer marketing director, says they suspect an intent bias in the sample, and they are running control groups of consented readers who do not see it to find out what is actually incremental.

This test matters. The FT is already using the AI, but it still compares readers who see the AI paywall with readers who do not. Only that comparison shows whether the AI really brought in the extra subscribers.

Are these the questions for the AI era?

1. Purpose: What decision does this number serve, now that a model may be doing the deciding?
2. Substance: Should the depth of the relationship set the price a reader pays?
3. Practice: Who checks that the definitions are still right, when AI acts on them without anyone reviewing each decision?