How do we evaluate cultural success?

Last week I attended a conference on the evaluation of Hull’s UK City of Culture year. Many people in the cultural sector have recognised Hull 2017 as a success - and data, research, and evaluation were a core part of this.

I found out a lot about how Hull 2017 produced such a high-quality evaluation. But I also left with some questions and new ideas around how and what we measure, and how we should best make the case for more investment in culture.

Commitment to research and evaluation

If you take a look at the full Hull 2017 evaluation report, it’s clear that data and research were a massive focus for the team over the whole project.

Here are some of their findings:

• An audience of 5.3 million attending over 2,800 events, cultural activities, installations and exhibitions.
• 95% of Hull residents attended at least one event.
• Nearly 800 new jobs have been created in the visitor economy and cultural sector since 2013.
• Over 60% of ticket-buying audiences were first-time bookers.
• 75% of those who visited Hull in 2017 stated that it changed their perception of the city for the better.
• Volunteers supporting Hull’s City of Culture year have so far given around 300,000 hours of their own time – equating to 34 years.

It is this commitment to data the enabled them to see if their work was really having an impact. It’s easy to count numbers of events put on – but without a clear data and audience plan, you can’t know who those events and activities are reaching.

Importance of volunteers

Hull 2017 had a much-celebrated team of 2400 volunteers, and 120 of these were given specialist fieldwork training.

I talked to two of these volunteers who had played a part in the research and evaluation of Hull 2017. They were both extremely positive about the entire volunteering experience, particularly the online platform which was used to match volunteers’ interests and skills to particular events and tasks. Once they had been given fieldwork training, they attended events and shows to conduct visitor surveys via iPads. And they also did some more creative surveying involving jars, beans, and sand.

Arguments for Cultural Investment

But after all the evaluation is done… where next? When cultural organisations get good data in their hands they tend to use it to improve customer experience and grow their audiences. But next on the list is using impact evidence to argue for more investment.

In the cultural sector, we all know the transformative benefits that culture can have – we see it first-hand. But there’s a question around how best to argue for cultural investment. Should we emphasise the economic benefits, the intrinsic value of the arts, or the social and wellbeing impacts?

Several speakers talked about the importance of highlighting economic benefits as a way to get the council and public on board at the start of the bid process. They sold the bid as a way to create jobs and make Hull an attractive city for outside investment. Hull city council literally put economy and culture workers side by side in the same office - building a strong relationship between the two and widening agendas for both areas.

Equally, there was a lot of research on the wellbeing impacts of Hull 2017. Some speakers argued that the conversation needs to be shifted to emphasise social impacts instead of solely economic ones.

If this happens, we need to start thinking about who we as cultural organisations should be partnering with to deliver these outcomes. If culture delivers wellbeing, health, and social impact – then funding sources should reflect this fact.

New Approaches to Impact Measurement

There were also suggestions for changing the way impact is measured. Hasan Bakhshi spoke about ‘contingent valuation’ – an assessment approach that asks the public what they would be willing to pay for culture, versus something else. This approach wasn’t used in the Hull evaluation report, but he argued that it should be, because it can show benefits of culture that aren’t captured by financial transactions.

Contingent valuation has been used in heritage a lot, but less so in arts and culture. You can see an example of contingent valuation used in this Tate Liverpool report.

There was also discussion around what we define as ‘culture’ when we are measuring levels of engagement. We may not know how much culture if happening if we are using one definition of it or looking in the wrong places. For example, the 16 – 34 age-group had a very different definition of culture than arts programmers, who tended to be older. And it is important to properly audit the city’s cultural diversity too – we may not know what is going on culturally if it happens outside traditional venues.

What next?

The final evaluation report is due to be published in 2019. There are calls now to keep the evaluation going to see what the impacts of the year will be beyond 2017 – going back in one, five, and ten years’ time.

The strength of any research project is doing something with the results and sustaining success, so I think everyone in arts, culture, and heritage will be looking to Hull in the next few years to see the long-term impacts.