This report is a deliverable of the GODAN Action project, and outlines the main lessons from a review of approaches developed to evaluate the impact of open data.
The review took into account a range of academic and policy oriented empirical studies that examine different aspects of initiatives incorporating open data, as well as methodological and conceptual frameworks designed to support open data evaluation.
Lessons and insights from this report will form the basis for the development of an approach to evaluating initiatives that incorporate open data, in the context of the GODAN
The overview of empirical studies of open data revealed that:
- The methodological repertoire of the majority of studies is limited.
- Many empirical studies on open data examine perceived impact and use. Although this approach can be useful, it has significant limitations and is heavily influenced by the rigour of the research design.
- More objective usage metrics, such as government website and application analytics, can be valuable but may be difficult to obtain as they require a certain level of access.
- Retrospectively estimating the economic value of open data can be a struggle. It can be difficult to isolate the net benefits, as open data is often used in conjunction with proprietary data.
- There are important lessons to be learned on how to define and operationalise impact from other social science fields, for particular goals and uses of open data.
- Also relevant, albeit with some adjustments, are lessons from the study of open data’s impact in developed countries.
- Given the many uses of open data, an approach to the study of impact that centres the user/use is a valid starting point for inquiry.
- Understanding the conditions of impact is important in order to grasp how the benefits of open data are moderated.
The main insights afforded from the dedicated frameworks for evaluation are:
- It is important to distinguish between different types of results, and the different kinds of outputs open data initiatives may generate, whether they are outcomes or impacts. For example, open data can result in improved service delivery and citizen satisfaction. A more equitable society could be an example of a longer-term impact of open data.
- The process of evaluation needs to take into account the stakeholders’ ideas of success and how success is measured, rather than relying solely on readymade criteria and indicators.
- It is vital to establish the context and the conditions of open data innovation in order to fully understand the character and contributions of the examined initiatives, services and products.