Home > TV > Ofcom PSB Review > Ofcom reports > Public Service Broadcasting > Assessing the value produced by PSB

Assessing the value produced by PSB

Measurement of the delivery of PSB, as described above, will tell us about how effectively PSBs are fulfilling the purposes set out in the Act. It will not, however, tell us how much the various components of PSB are valued by viewers, nor will it show whether the cost of the relevant PSB interventions is justified by the benefits ultimately produced. Measuring the costs and benefits of PSB should be the final part of our analysis. It is also possibly the most important, and almost certainly the most difficult, part of the work.

Its importance stems from the fact that in the current political and economic climate it is no longer excusable to justify such a major use of public funds as public service broadcasting without hard evidence regarding the value for money delivered by that expenditure. Few other forms of public spending have such an easy ride as PSB has enjoyed in the past (arguably because few other forms of public intervention have resulted in the provision of services that are as widely enjoyed as PSB programmes).

The difficulty of the work is due in large part to the intangible nature of the benefits of PSB. As with many other publicly-funded services, such as defence and other forms of arts funding, it is extremely difficult to translate the benefits provided by PSB into a monetary value that could then be directly compared with a cost figure to provide a net benefit (or net cost) to society of PSB interventions.

Nevertheless, after examining a number of different approaches that might be taken to assess these benefits, we do think that powerful tools exist which can be deployed to shed more light than previous studies have done on the relative strength of the benefits of different components of PSB. This in turn may be used to inform decisions as whether money is being spent on the right things.

ITV1 pilot study

As an indication of the approach that might be taken, the ITC has carried out a pilot study that examines the costs and benefits of certain genre-specific PSB obligations imposed on ITV1. The purpose of the project - carried out with the help of consultancy firms Human Capital and The Knowledge Agency - was to calculate, for the first time, detailed estimates of the opportunity costs incurred by ITV as a consequence of its PSB programming obligations, along with an assessment of the benefits of these programming obligations enjoyed by UK audiences in their capacity both as consumers and as citizens.

This section describes the approach and results of the research, and concludes with a look at what the study seems to be saying about the overall value of PSB programming as currently delivered by ITV1.

ITV1's genre-specific obligations in 2002 are shown in the table below.

Genre	Hours of programming	ITV1 PSB obligations in 2002
Network News and Weather Current Affairs Religion Children's (excl acquired) Arts	Annual programme hours 365 (obligation) 78 (obligation) 104 (obligation) 391 (obligation) 39 (obligation)
Regional - average across licensees
Regional News Regional Current Affairs Regional Other	342 (all programming) 29 (all programming) 290 (all programming)

Approach to calculating opportunity costs

At the heart of our approach lies a simple but fundamental formula for calculating the 'opportunity cost' to ITV1 of showing PSB programmes. This formula aims to capture the true cost of PSB, which is not simply how much these programmes cost to make. Rather, it is the margin (or contribution) generated by these PSB programmes compared to what might have been shown if the PSB obligation did not exist. More specifically, the 'opportunity cost' of a PSB obligation is made up of:

the additional production costs of the relevant PSB programmes that were actually shown, compared to those of alternative non-PSB programmes that might have been shown in their place; plus
the net advertising revenue (NAR) foregone by showing the relevant PSB programmes rather than more commercial non-PSB programmes.

All further references to the costs of PSB are to this notion of opportunity cost.

In order to complete the specification of this approach, we needed to take a view on what types of programmes ITV1 might show if it were not under obligation to show PSB programming. We deliberately developed a mechanistic rules-based approach, based on the underlying principle that, in the absence of PSB obligations, ITV1 would simply show more of the types of commercial programming that it already shows in similar time slots. This approach has the merits, firstly, of simplicity (in conceptual terms); secondly, of objectivity, in that it avoids the need for subjective or arbitrary judgements; and thirdly, of being intuitively plausible (more so than other candidate rules that we considered).

We thus created a stylised schedule that represented the types of programmes that ITV1 actually showed in 2002, along with a series of alternative schedules showing what might have been shown had each of the PSB obligations not existed(-6-). Using detailed data (provided by licensees to the ITC, along with BARB viewing data), we then estimated average costs-per-hour and revenues-per-hour for each programme genre, separately for each time of the day and each day of the week in the stylised schedules.

Opportunity costs to ITV1

The results of this exercise are shown in the chart below, which shows the total annual opportunity costs to ITV1 associated with each of the PSB genres. The chart also shows the number of hours of programming each year stipulated by each of the obligations.

Opportunity costs of PSB obligations by genre, £ million (2002)

The most immediate conclusion is that regional programming emerges as by far the greatest burden for ITV1. The explanation for this is relatively straightforward: this type of programming is inevitably expensive compared to alternative commercial programming given the regional variations required, with production costs incurred separately in each region. Moreover, a high volume of hours is required to be broadcast under the ITC's obligations (on average, the ITV regions each showed 660 hours annually across the three regional PSB genres). Moreover, regional news in particular is broadcast at times in the day when significantly higher revenues could be earned by showing more popular commercial programming. So both elements of the opportunity cost formula - extra cost of production and foregone advertising revenue - are significant when it comes to regional programming.

The chart shows that the next most expensive genre is children's programming, again due to a mixture of high opportunity costs-per hour and a large volume of output. Interestingly, further analysis showed significant differences between the costs associated with ITV1's children's output in its two main slots: weekday afternoons and Saturday mornings. The weekend programmes do not in fact impose significant opportunity costs to ITV1, while the costs associated with weekday slots are substantially higher than the all-week averages shown in the chart.

Turning to the other genres, network news and current affairs are relatively low-cost genres. For news programmes, this reflects average programming costs that are low compared to what might be shown in their place (particularly for the peak-time bulletins). Also, audiences and hence revenues for network news remain relatively high. For these reasons, the total annual opportunity cost of news provision remains low despite the high volume of output.

The costs associated with religion and arts programmes are the lowest of all the formal PSB obligations, due in part to their relatively low volume of output and to their off-peak scheduling (primarily Sunday lunchtime and late evening respectively).

Having discussed the main results, it is important to highlight some of the most important caveats to this work. The size of this list is an indication both of the complexity of the task (and therefore the need to specify rules and simplifying assumptions to make it manageable) and the care we have taken to be honest about the limitations of the approach.

First, as an overarching point, although we believe the methodology to be robust and rigorously-applied, and the modelled data to be extremely comprehensive, the results remain estimates based on a series of both significant and minor assumptions.
In particular, the results are highly dependent on the rule we applied for what ITV1 would show if it were not faced with its PSB obligations. We also considered alternative rules, such as replacing all PSB programming with acquisitions. While we believe such alternatives to be less plausible, it is important to realise that some of them would have generated substantially different results.
We did not model the impact of 'inheritance' effects, whereby a change in the schedule might have knock-on effects on the audiences of subsequent programmes. This means, for example, that we did not capture the entire potential benefit that ITV1 might enjoy, if it were able to drop its weekday afternoon children's programmes, by reconstructing its subsequent early evening schedule.
We also did not model the impact of counter scheduling, whereby a change in ITV1's schedule might lead to a response by other broadcasters (and a possible further response by ITV1). Whereas both this and the inheritance effect would inevitably have some impact on the true opportunity costs, note that these two effects will tend to work in opposite directions and thus cancel each other out to some extent. This means that excluding them from the model becomes less problematic.
Our model was constructed to estimate the impact of removing each PSB genre separately, for two reasons. Firstly, this enabled us to estimate the costs for each individual genre. And secondly, this meant that the volume of programming being removed and inserted in each scenario would be much smaller than if all the genres were removed at once. This in turn means that the implicit assumption that average costs-per-hour and revenues-per-hour for each genre remain constant as the volume of programming changes is likely to be a reasonable simplification. This would not necessarily be the case if we were to remove all the genres at once, and so adding up the opportunity costs for each genre to reach an estimate of the total cost imposed on ITV1 by all its PSB genre obligations taken together is likely to give a less reliable estimate than those for the individual genres.
Finally, our pilot study was limited to analysing genre-specific programming obligations on ITV1. There are additional conditions that ITV licensees face, but which we excluded from the scope of the study, such as the requirement to maintain a network of regional production centres.

Assessing the benefits of PSB programming on ITV1

The approach that we adopted to analyse the costs of PSB was conceptually straightforward, notwithstanding the laborious nature of such a data-intensive study. Estimating the value of the corresponding benefits, by contrast, is a much trickier task, as a couple of fundamental problems arise when it comes to determining an appropriate methodology. The first problem is that benefits cannot be captured by a simple formula, analogous to the one specified for the opportunity costs, which may be evaluated by collecting and analysing relevant hard data. Rather, market research tends to be the standard way to assess the benefits of PSB. And second, as noted above, the value of the benefits cannot readily be expressed in monetary terms. The benefits are typically assumed to confer some kind of 'utility' (as economists would say) on to consumers, and mapping utility on to a monetary scale is a far from simple exercise.

Given these fundamental problems, we felt that it should be feasible at least to calculate the relative benefits of different PSB genres - how important is network news compared to, say, children's programming. Of course, many studies in this area have been conducted in the past, including those by the ITC. An example of the types of results that are produced is shown in the next chart.

Example of results from earlier ITC studies

Do you think ITV should have to show this type of programme?

Source: What Viewers Want, ITC, January 2001

Notice that in this example the results are highly clustered, with five of the eleven genres generating scores of over 90%. This highlights what we believe to be an important shortcoming of this type of study. Because of the nature of the questions that respondents face, there is no downside or cost to them agreeing (in this instance) that ITV1 should show every type of programme, as a result of which there will be a strong tendency for the percentage of 'yes' responses to be very high.

Our ambition was to address this issue by conducting research using a framework that forces people to make actual choices between different genres and reveal their real strength of preference between different PSB genres. Such an approach would put people in a position where they realised that they could not have more of everything for 'free' and would therefore have to choose what was most important to them.

In order to do this, we turned to a market research technique called conjoint analysis. This technique is well-established in other areas of consumer products, and is used to help manufacturers and other service providers understand consumers' preferences over the attributes of products such as package holidays, bank accounts and cars. It does this by offering respondents a series of simple trade-offs between different levels of the attributes that make up the product. Using the classic example of cars, these attributes might include number of doors, safety features, specification of the in-car stereo, luxuriance of the interior, etc. The individual trade-offs are then aggregated to reveal consumers' relative preferences about the attributes of the relevant product.

For our purposes, the relevant product was PSB programming on ITV1. The challenge was to adapt a technique generally used for products about which consumers have to make conscious purchase decisions to a service that they receive for free. To add to the challenge, we also wanted respondents to think about the different genres not only from their own personal viewpoint but also from a wider societal perspective.

To this end, we sent a questionnaire to more than 4000 respondents across the UK (-5-), in which they were presented with a series of pairs of different scenarios about the amount of PSB programming that could be shown on ITV1 each week. Each scenario specified a certain number of hours of each PSB genre, with the number of hours in each pair varying across some or all of the genres. The nature of each genre was explained to respondents, with (region-specific) programme examples for each one. We asked respondents to decide, for each pair, which alternative they preferred both from their own position as viewers (personal or consumer benefits) and also from the position of UK society as a whole (social or citizen benefits).

Relative benefits of PSB programming on ITV1

The aggregate preferences expressed by respondents as a result of the choices made in the questionnaire are shown below. As there was no absolute measure, the results are expressed as a percentage of the total preference for network news (which itself therefore had a score of 100%).

Relative benefits of PSB programming on ITV1: results of conjoint analysis

Index: News=100

We found these constrained-choice results to be extremely interesting. Compared to previous research, the importance attached to network news on ITV1 is stark. Along with previous studies, this was shown to be the most valued genre. But what the conjoint analysis revealed was that - with the exception of current affairs (which was also highly valued) - network news provides more than twice as much value to ITV1 viewers as any other PSB genre. Note the contrast between this result and those from the previous study, which showed news to be rated only slightly higher than other genres. The new results appear to indicate that network news really is the very bedrock of PSB on ITV1, with current affairs emerging as a strong second.

The other results that stand out are those for regional programming. When forced to make a choice based on the actual delivery of regional programming, respondents appeared to place a much lower value on ITV1's regional programming than in previous studies. Indeed, the results imply that they would rather watch any other PSB genre than so-called 'Regional Other' (which includes a mixture of regional sports, factual, entertainment and arts programmes). When asked about the importance of the various genres, however, regional programming scored highly (consistent with previous studies), indicating that further work is needed to understand the key drivers of these results, including perhaps a review of the quality of the programming delivered.

Weighing up costs and benefits

Bringing together the costs and benefits studies, it is possible to represent the results on a chart which gives some insights into the relative (but not absolute) performance of these genres on ITV1. This chart is shown below.

Comparing costs and benefits of ITV1's PSB obligations: pilot results

The chart plots the opportunity costs of each genre against their relative benefits.

These results need to be interpreted carefully. Because the benefits are ranked relative to each other rather than in absolute terms, it is not possible to subtract the benefits score from the opportunity cost to derive a net benefit. Indeed, it is not possible to infer from this chart whether the net benefits for any genres are positive or negative. So we cannot state in absolute terms whether some genres are proving good value for money and others not. What we can do is draw conclusions about the relative performance of different genres. That is, we can say that genre X is performing better than genre Y by delivering greater benefit at a lower cost.

Bearing this in mind, three distinct groups of genres emerge. The first includes arts, religion and regional current affairs. While the personal and social benefits derived from these genres appear to be comparatively low, the opportunity cost of these genres is also low, so these genres are of least concern in terms of the burden imposed on ITV1.

The second group - network news, current affairs and children's - contains those genres which are most highly valued by viewers. Although the opportunity cost associated with these genres is not insignificant, they are certainly not the most costly genres and therefore these may be thought of as the best-performing PSB genres.

Regional news and regional other programming, which make up the third group, are the genres revealed as being of most concern, due to both the particularly high cost of delivery and the relatively low associated benefits. Interestingly, if the conclusion of ITC research in 2002 (Pride of Place) - that viewers would prefer fewer hours of higher quality regional programming on ITV1 - were adopted, we might see a move both upwards and to the left for these genres, into a higher-benefit, lower-cost area of the chart.

As with the costs work, it is important to note a few caveats to these results.

First, to the best of our knowledge, this is the first time that conjoint analysis has been undertaken to assess different genres of television programming. While we believe that the approach that we have taken is robust (and was tested carefully with two pilot studies), the innovative nature of the work means that it has not been possible to cross-check the findings with other studies.
Second, while we asked respondents to make their choices based on both personal preferences and taking a wider societal view, it is difficult to check to what extent the altruistic element informed their decisions. Further research could probe this further by conducting separate conjoint studies focusing on the personal and social elements.
And third, a general concern that applies to all consumer-oriented market research is that the approach is based on the assumption that consumers are best-placed to judge the full extent of what PSB should provide. Consequently, this research does not identify the 'merit good' aspects of PSB, the benefits of which, by their nature, are not fully recognised by consumers. Moreover, there may be some wider social benefits that are not understood or recognised by individuals, even when challenged to include them in their responses. As noted above, it would be possible to complement this work by seeking the views of industry experts and/or of specialists in other fields directly relevant to each genre (e.g. parents and teachers for children's programming).

Extending the work to other channels

Cost-benefit analysis of this nature could be extended to the other network channels, although varying approaches may be needed. The ITC has already undertaken work on the opportunity costs of the PSB genres on Five, and it would be possible to undertake research on the corresponding benefits along the same lines as the ITV1 study. However, this methodology does not translate so easily to Channel 4 or to the BBC channels.

Channel 4 faces a range of genre-specific obligations, like ITV1, but its remit also specifies a series of additional obligations, e.g. to 'encourage innovation and experiment', that apply across all its output. Therefore, it is not so straightforward to split Channel 4's output into PSB and non-PSB programming. Also, as it is not intended to be a commercially-driven profit-maximiser, the assumptions regarding the programming that Channel 4 would show if its obligations were to be relaxed would not necessarily be the same as those that applied to ITV1. For these reasons, some modifications to the approach would need to be made. For example, it may be necessary to hypothesise a 'profit-maximising' schedule for Channel 4 (free of all obligations), rather than modelling incremental changes to the programme mix, and derive from this an estimate of the value to a fully commercial operator of Channel 4's licence.

Turning to the BBC channels, a different framework is likely to be necessary. As with Channel 4, the PSB elements of these channels cannot be reduced to specific genres. More fundamentally, given the nature and funding of the BBC, it would be difficult to apply an opportunity cost formula in the same way as was done for ITV1. One approach for assessing the value of these channels would be to conduct market research that presented respondents with a range of bundles of BBC services, consistent with different levels of the licence fee, and ask respondents which they would prefer. The challenges of such an approach would be to devise meaningful packages or bundles, and to elicit honest responses that were not biased in either direction.

Footnotes

5. To give a simple numerical example, the stylised schedule might have shown that between 2-3pm on Mondays over the course of 2002, ITV1 showed 40% of light entertainment (LE), 40% acquired programming and 20% regional current affairs PSB programming (these figures are purely illustrative, and are not intended to represent what ITV1 actually showed in that time slot). To estimate what ITV1 might have shown in place of regional current affairs, if the obligation to show those programmes had not existed, we constructed the alternative schedule by simply removing the PSB programming and scaling up the remaining commercial programming, to 50% LE and 50% acquired.

6. A self-completion postal survey was sent to a broadcast panel recruited to be representative of the UK population in terms of age, sex, socio-economic class, working status, region and multichannel take-up (Quest panel). A total of 4,479 questionnaires were returned.