Despite the great appeal that citizen participation has had for political authorities, a culture of evaluation in citizen participation seems far from being instituted on a worldwide basis. There is general agreement that challenges in providing robust policy evaluation theories and empirical evidence refer to the difficulty of defining what participation is and how it should be implemented (Rosener 1981; Abelson & Gauvin 2006). However, the lack of attention paid to the evaluation of participatory processes is one of the major hurdles to the development of this field of study, as expressed by international and transnational sponsors. The OECD (2005: 10) affirms that ‘there is a striking imbalance between the amount of time, money and energy that governments in OECD countries invest in engaging citizens and civil society in public decision making and the amount of attention they pay to evaluating the effectiveness and impact of such efforts’. As the creation of spaces for collective bargaining on policy issues can be captured by political and financial elites, and cleavages of socioeconomic resources in civil society can further condition these processes, S. Arnstein (1969), early on, warned of the necessity to guarantee that the have-nots (i.e. participants that have limited access to standard decision-making) gain actual power of decision in the face of the powerholders (i.e. political sponsors and decision-makers). Therefore, the evaluation is expected to expose value biases and ‘hidden agendas’, while certifying whether goals are being pursued and how this is taking place.
Assessments of participatory processes in urban regeneration and other policy domains are mainly conducted via place-based methods. The spatial and temporal fragmentation of these experiments, together with little (international) dissemination often means that their overall contribution to this field of study is overlooked. In fact, the lack of transferable and replicable models of evaluation reduces the chance of wider debate on the conceptualisation and operationalisation of accessible and shared metrics (Chess & Purcell 1999; Delli Carpini, Cook & Jacobs 2004). As G. Rowe & L. J. Frewer (2004: 551) put it, ’without typologies of mechanisms and contexts, and an attempt by researchers to adequately define the exercise(s) they are evaluating against these, little progress will be made in establishing a theory of “what works best when”’. The scarceness of evaluation compromises a more comprehensive understanding of the mechanisms and outputs of the increasing number of participatory processes worldwide. In the case of participatory processes in urban regeneration, this condition further weakens the chances of knowing what really works to balance the socio-spatial inequalities of deprived areas (Soja 2010). Multiple forms of social exclusion and injustice necessarily rely on the multidimensional aspects of disadvantage at the urban level. As N. Fraser (2009: 15) put it ‘overcoming injustice means dismantling institutionalized obstacles that prevent some people from participating on a par with others as full partners in social interaction.’. To promote the wider participation of local communities in the multiple trajectories that urban regeneration can take to countervail the marginalisation of equity concerns, it is therefore necessary to expose its agenda and provide robust information on procedures and outputs for the fair redistribution of resources.
Bearing this in mind, the article discusses the conceptualisation and operationalisation of the participation index of the participatory programme for urban regeneration, BipZip, which has been implemented by the municipality of Lisbon since 2011. The index was devised to gather multiple and complex information within one measurable parameter for each urban area where the programme has been implemented. The goal of the index is to make different types of data converge and provide the Municipality, as well as all the agents involved in the programme at large, with a consistent metric of evaluation. In order to provide the necessary inputs for discussion on the construction of this index, the article is structured as follows. First, a general outline of participatory processes in urban regeneration is provided by shedding light on key scientific literature. Secondly, some of the most contentious issues on the evaluation of citizen participation in policymaking from scientific and grey literature are presented. Thirdly, the institutional design of the BipZip programme is described in the light of the broader socio-political context of Portugal. Fourthly, the construction of the participation index is described and key highlights are put forward for the advance of the evaluation of the programme. Finally, the conceptualisation and operationalisation of the participation index is discussed in the light of international debate and evidence from the field.
An outline of participatory processes in urban regeneration
The equitable distribution of resources in urban contexts is influenced by and in turn influences the organisation of social relations and opportunities for citizens to live in the city and to have the right to produce their urban space (Lefebvre 1968; Purcell 2002). If cities have historically been the locus of the expansion of citizenship, patterns of urbanisation can create unequal provision of structures and infrastructures, as well as impaired access to adequate standards of socioeconomic life that end up marginalising some citizens. The relationship between the spatial configuration of cities and local communities is a complex phenomenon that changes through time, although in some cases inequalities are installed and accumulated in specific areas in a way that requires targeted interventions (Power 2000). The decrease in control over decisions and the production of space inhabited by citizens has encouraged public agencies to reflect on current challenges for wider and stronger forms of enfranchisement. As greater commitment to social inclusion should be pursued with those who are the most deprived of urban spaces, P. Marcuse (2010) argues that the fulfilment of the ‘right to the city’ should account for those who are the most deeply affected by its absence (see also Soja 2010 and Iveson 2011). Against this backdrop, international and transnational agencies have increasingly urged urban authorities to practice new forms of citizen engagement to tackle such a complex array of social, economic, and political phenomena (Lowndes & Sullivan 2008).
Participatory processes with local communities have opened up a new socio-political scenario where distributive policies rely on procedural arrangements that are expected to elicit different configurations of material and immaterial goods. In general, citizen participation in urban regeneration assumes that the reinforcement of the social capital assets of local communities can be a means and an end to solving emerging inequalities. As stated by N. Bailey & M. Pill (2011), the broad policy trend to approach the ‘local’ in reducing socio-spatial inequalities relies on the assumption that urban areas contain varying degrees and types of social capital to be enhanced through capacity-building measures with direct pay-offs in terms of economic development and political participation. Generalised trust, shared norms, and values characterise strong and weak ties between citizens, the former joining people on familiar and friendly bases, the latter opening up to other forms of relations (Granovetter 1974); supportive and bridging ties within communities (Briggs 1998); bonding and bridging relationships that may reinforce either internal or external connections between the community and the city as a whole (Putnam 2000). The social capital, thus, provides information about the capacity of the local community to act upon inequalities and resist external pressures that influence (or create) disadvantages.
The unequal distribution of resources behind the phenomena of citizen isolation, ghettoization, marginalisation, and segregation is deemed to be disentangled by the enhancement of participatory processes for urban regeneration, which can address either more physical or social goals. Processes can be diversely socio-economic, ethnic, gender-specific, mobility-specific, etc. and seek to act upon the physical environment, like the presence and/or stability of local institutions, such as churches, parks, recreation facilities, employment centres, supermarkets (Curley 2010), and access to facilities and services (Dekker & Bolt 2005). Others can focus more extensively on the enhancement of social links within the community and outside, fostering place attachment and feelings of safety (Curley 2010), community spirit to act collectively (Forrest & Kearns 2001), and face (perceived) stigmas (Wallace 2010), racial discrimination or harassment (Buck 2001), as well as the aim to promote greater inclusion of the most disadvantaged social groups, such as the elderly and those living in precarious housing conditions (Durose 2011).
The need to improve the living conditions of the most disadvantaged groups is frequently addressed to poor and elderly people (Forrest & Kearns 2001), as well as to those with difficulty in accessing the labour market and the unemployed (Atkinson & Kintrea 2001). Operationally, the enhancement of social capital in deprived areas sees the role of community leaders as central in the participatory process, as well as in everyday life. The theory is that community leaders work to gain trust from local agents and access to elite groups, acting as brokers connecting informal and institutional networks (Maloney, Smith & Stoker 2000). As they are expected to ‘embody’ the social capital and sit at the table with other relevant agents of decision-making, participatory processes are frequently organised through local networks and partnerships. The aggregation of different agents and the constitution of partnerships have been argued to strengthen the social capital, as well as the citizenry’s trust of public institutions (Aitken 2012). However, considering that power inequalities may persist between local authorities and community leaders (Purdue 2001), risks of co-option of local agents can compromise strategies of inclusion, while an overemphasis on community leaders as new ‘social entrepreneurs’ for community capacity-building may shift responsibility from state to local communities and individuals.
Critical considerations are crucial when considering the global growth and dissemination of participatory processes that aim to reduce burdens and enhance the benefits of local communities in deprived urban areas within the recent financial crisis that has dramatically strengthened global trends of inequality (Falanga 2018). As the imposition of global forces of the neoliberal market structure urban life opportunities in less prosperous areas, the engagement of local communities can be part of wider processes of commodification of a community lifestyle. The segregation of socioeconomic problems and the encouragement of forms of self-governance that, however, are decoupled from wider strategies confirm that greater attention should be paid to participatory processes in urban regeneration (Forrest & Kearns 2001). There is increasing competition among cities for the advantages brought about by financial support in initiatives for urban regeneration and participatory processes in general (Swyngedow 2005). The imbalance in the provision of financial capital is mirrored in the unequal distribution of resources in urban areas that can be in conflict with one another. Accordingly, scholars have become alerted to the perverse effects of localism and devolution in communities, as people may gain more access to local state institutions and expand their political rights without guarantee of redistributive equity (Diamond 2001; Taylor 2007; Wallace 2010).
Measuring participatory processes
The unpredictability of citizen participation is often assumed to be a major hurdle in defining how participatory processes are expected to lead to new ways of framing problems, new solutions and new ways of working. The difficulty in identifying common strands of citizen participation in policymaking is reflected in the hard task of defining what constitutes the success of these processes. According to T. Brannan, P. John & G. Stoker (2006), the general lack of data on participatory processes is due, inter alia, to their experimental nature; the scarce attention given to and funding provided for monitoring and evaluation; the difficulties in identifying and mapping the complex range of investors and beneficiaries involved; the presence of hidden costs that are hardly detectable. Authors argue that ‘what is needed is on-the-ground research that tells both academics and policy-makers what to do and not to do – what works and what does not – in a way that builds knowledge about this new policy area’ (Brannan, John & Stoker 2006: 1001). On the contrary, anecdotal evidence, compounded by a general lack of adequate description of the designs and methods used in participatory processes in favour of a more normative statement on citizen participation as good per se, is often found in this field of study. Scientific knowledge frequently produces arguments based on a self-evident legitimation of participatory processes and, as highlighted by L. Häikiö (2012: 432), ‘discursive structures presented in scientific texts have an effect on the ways legitimate citizen participation is understood and how participation is enacted in the local settings.’
The need to untangle the reasons for moving local authorities to sponsor participatory processes seeks reasonable responses. One of the main reasons behind the need to construct methodologies for the evaluation of participatory processes is to allow citizen participation agendas to ‘go public’, which is expected to diminish the inherent risks of manipulation by economic and political elites (Rosener 1978). It is not only the assessment of costs and benefits for both policy-makers and citizens that is legitimate, as in addition the choice of goals and the positioning of participants (and/or their representatives) entitled to be the gatekeepers of local communities – which are never neutral – require tools to measure the degree and intensity of power shared by political, policy and social agents (Cornwall 2004). According to the main literature, a clear definition and problematisation of the success achieved or not achieved by participatory processes is the first step to critically understanding whether and to what extent normative goals of democratic enhancement and/or policy-based goals of public service improvement are being pursued in urban regeneration (Rosener 1978; Fiorino 1990; Abelson & Gauvin 2006). This conceptual and operational endeavour competes with the recognition that the evaluation of participatory processes needs to explain the ways in which the intended goals are achieved, as well as which political, societal, economic and administrative factors drive the design and implementation of specific procedures (cf. Weiss 2004; Burton, Goodlad & Croft 2006).
The evaluation of participatory procedures and outputs inherently depends on the construction of criteria against which measures can be selected within the wide range of extensive and intensive research methods (Rowe & Frewer 2004). Participatory procedures can be evaluated by taking into account the arrangements for public negotiation of interests and values among different agents, as well as between policy areas and levels of administration (Delli Carpini, Cook & Jacobs 2004). Acknowledging the peculiarity of participatory settings, T. Webler (1999) focuses on the potential of social learning acquired through cognitive enhancement and moral development. Procedures are also characterised in terms of fairness (what people are permitted to do) and competence (construction of the ‘best’ possible understandings and agreements among the agents) (Webler & Tuler 2002). Outputs are more clearly linked to either normative (i.e. enhancement of democratic values) or policy-based (i.e. improvement of public service effectiveness) goals of citizen participation (Fiorino 1990). Criteria on the results in the policy area should cover technical issues, such as timings to develop regulations and the reduction of judicial challenges, to substantive issues, such as agency responsiveness to participants’ demands and the impact of the process in general.
An overview of the socio-political context in Portugal
The end of the dictatorial regime in Portugal, precipitated by the Carnation Revolution, and social pressure for the installation of democracy, led to the definition of the national Constitution in 1976, which includes principles of participatory democracy. After years of increasing social mobilisation and engagement in public life at the end of the 1970s (Fishman 2011), the 1980s and 1990s were characterised by spreading disaffection with the political class, generally perceived as corrupt by society (Caldas 2012; de Sousa, Magalhães & Amaral 2014). Negative outlooks on citizenry’s trust of democratic institutions and political representatives were paired with high abstention rates in political elections, peaking in the 2013 local elections with 47.4 % and followed by 44.1% in the legislative elections in 2015 (cf. OQD 2012).
The disruption of the sovereign debt crisis in the last few years met with an unstable banking system, the growth of fiscal deficit, public debt, and unemployment. To tackle this situation, the European Union pressured the Portuguese government to adopt austerity measures, which were eventually agreed with the Troika (International Monetary Fund, European Central Bank, and European Commission) in 2011 under the requirement of a €78 billion bailout package (Caldas 2012; OECD 2015). The main goal of the Memorandum of Understanding on Specific Economic Conditionality (MoU) between Portugal and the Troika was to stabilise domestic finances and improve international competitiveness in order to regain a complete access to the bond market. In order to achieve these aims, the cost of the labour market was lowered together with public-private partnerships and state-owned enterprises, while renewed incentives were given to the financial sector through the capitalisation of banks (Falanga 2018).
As C. Lapavitsas et al. (2012: 113) concluded, ‘the mix of austerity and liberalisation within the eurozone has been harsh on working people but also dangerous for economy and society’. Economic recession and an increase in poverty, aggravated by forms of growing political alienation and disaffection, led to the spreading of grassroots protests and mobilisations between 2011 and 2012 (Baumgarten 2013). Social perception and legitimisation of the adjustment programme worsened throughout its implementation due to the erosion of trust in national and international decision-makers (Fonseca & Ferreira 2015). When Portugal exited the adjustment programme in 2014, a new alliance referred to as ‘Geringonça’, a term that means an unstable and unformed thing, was appointed to form the new government following parliamentary elections in October 2015. The socialist party assumed the new mandate with the support of the communist party and the left block party committed to restoring the purchasing power of workers via a new expansionary agenda, in striking contrast with the austerity measures. Notwithstanding these changes, great socio-economic cleavages persist and the economic growth appears to be increasingly dependent on international capital investments, especially in the metropolitan areas of Lisbon and Porto (Falanga & Tulumello 2017).
Participatory processes in Lisbon and the Bip/Zip programme
The role of participatory processes, in particular those concerned with urban regeneration, should be understood by considering bottlenecks (and opportunities) arising from the impacts of the crisis in local contexts. In 2007/2008, Lisbon city council inaugurated the first participatory budget implemented at municipal level by a European capital city. The decision was consistent with the intention to place values and mechanisms of citizen participation at the centre of the local political agenda (Falanga & Lüchmann 2019). The initiation of the Bip/Zip programme in 2011 confirms the political commitment to enlarge the spectrum of policy areas covered by participatory processes in urban regeneration. The programme is focussed on the involvement of local associations, NGOs and parish governments in the implementation of local initiatives for the regeneration of the so-called priority areas, including neighbourhoods and areas with critical socioeconomic, infrastructural, and environmental outlooks. The mapping of the priority areas was first led by the Department of Local Housing and Development of the Municipality of Lisbon, and secondly supported by public consultation with local associations and NGOs via survey, as well as by public meetings with citizens in 2010. The survey collected 1039 responses (22% online and 78% in public meetings) that generally confirmed the classification of the indicators (87% agree; 3% disagree; 10% not responded), while new priority areas were proposed by 76 out of the 244 comments annexed to the survey. As a result of this process, 67 priority areas were identified and divided into four typologies: social housing neighbourhoods (= 29); historic centre (= 13); illegal housing (= 7); and other/mix (= 18). The result was the Bip/Zip Chart, which was issued as a constituent part of the city’s master plan (Fig. 1.).
The programme aimed to tackle long-lasting deficiencies in the provision of public services, which in the end were aggravated during the years of the crisis. Towards this end, initiatives led by local partnerships are encouraged to address issues emerging in the priority areas and engage communities with the goals of
enhancing community life, improving job skills and encouraging entrepreneurship, regenerating public space, and promoting social inclusion. Initiatives can be given funds of up to €50,000 each, with a total budget for the programme of around two million euros per year. The annual selection of proposals by local partnerships is run by an independent jury of external experts who are invited to select them according to predefined criteria: the participation of local communities approached in the formulation and implementation of the proposal; the degree to which the proposal solves the issues identified; the improvement of local development and social cohesion in the area (and beyond); the sustainability of the initiatives; and the implementation of innovative methods, objectives and partnerships.
Evidence emerging from the field suggests that some initiatives have been effective in attaining their goals. As the programme has become a major source of public funding in the city, this model of urban regeneration has become an outstanding brand of the municipality, which has increasingly sponsored the programme internationally. The programme received an award from the International Observatory on Participatory Democracy in 2013 as an example of good practice in citizen participation, and interest in it has grown accordingly at both national and international levels. Acknowledging all these elements, the Department decided to design an evaluation model with the help of a team of consultants. With the goal of providing tangible data on the success of the programme, the consultants drew the necessary inputs and inspiration from scientific and grey literature on the evaluation of citizen participation by seeking to match conceptual inputs with evidence. Furthermore, considering the policy-oriented mandate for the evaluation modelling, consultants were challenged to provide a parameter for the measurement of citizen participation in the programme. The privileged position of the author of this article as a researcher supporting the team of consultants provides this text with an original reflection on the construction of an index dealing with participation, which was, as far as the author has been able to establish, the first of its kind ever to be conceptualised and operationalised in this field of study.
Conceptualising and operationalising the participation index
The BipZip programme engenders normative goals for the enhancement of local democracy through the mobilisation of local communities, and policy-based goals towards the improvement of public services in priority areas. The commitment of local partnerships in the design and implementation of the initiatives is designed to take advantage of existing ties within and beyond local communities, fostering trajectories of social capital and finding innovative solutions to priority issues. While the exposure of the programme has attracted the interest of national and international scholars, practitioners, and policymakers, the need for more robust knowledge about the programme was mooted by the department of local housing and development, and the consultants. Accordingly, the evaluation was agreed to provide the municipality with an instrument of knowledge and information about the procedures and outputs of the programme.
Considering that the programme covers a wide range of factors that contribute to determining how local agents participate in combating socio-spatial inequalities within and between the neighbourhoods, funded initiatives are expected to enhance multiple aspects connected to the quality of life (Granovetter 1974; Briggs 1998; Putnam 2000). The initiatives can be implemented in one or more priority areas, promoting the (re)construction of both the internal and external ties of local communities, and increasing socio-territorial cohesion. The identification of the issues to be addressed in the initiatives is autonomously selected by local partnerships, and their solution relies on multiple forms of self-governance with local communities. Physical interventions together with social and cultural activities have been designed to generate impact on the availability and use of public arrangements in the localities, as well as on the living conditions of those who live there. On occasion and whenever the priority areas have required specific interventions with more disadvantaged groups and/or in-need sectors of civil society, a variety of initiatives have addressed goals of combating poverty, discrimination and stigma.
The programme as a whole has sought to diminish the long-standing disparities between and/or the emerging conflicts among the prosperous and disadvantaged areas of the city (Swyngedow 2005; Bailey & Pill 2011). This aim relies on a policy design that encourages the self-organisation of local partnerships to lead initiatives of urban regeneration and act through informal and formal networks to support local communities (Atkin 2012). While assets of social capital stand behind initiatives designed and implemented by local partnerships, the active engagement of local communities should rely on and reinforce trust, shared norms, and common values among citizens for the successful implementation of initiatives on the ground. Until 2017, the main outputs from the seven editions of the programme have encompassed the engagement of 1015 local partners in the field (out of 2159 local partners applying to the programme), the implementation of 270 initiatives composed of 1683 activities, and the provision of an amount of almost fifteen million euros in funding.
Consistent with the inputs retrieved from the main literature, the programme aims to improve socio-territorial cohesion within and between local communities by addressing socioeconomic and physical issues that might otherwise compromise residents’ quality of life. Acknowledging that the programme should add some tangible value to standard policymaking and provide the impetus to rectify the critical issues identified and classified in the priority areas, three main aspects were identified: the local partnerships created through the programme; the initiatives implemented within the programme; and the public funding provided by the programme. These aspects were identified to provide a consistent understanding of the way the programme acts as a whole in order to (re)generate priority areas in Lisbon. Accordingly, the constitution of local partnerships and the requirement of active participation of local communities in the design and implementation of the initiatives are expected to mobilise human and social resources in the localities. The initiatives and their activities are expected to create new or reinforce existing social capital in the field. The provision of funding permits the shift of economic capital from the municipality to the priority areas giving the necessary support (Fig. 2.).
The identification of the three aspects helps to substantiate the definition of success, as well as the criteria and methods for the measurement of the procedures and outputs of the programme. Criteria will be clustered within the three aspects as the policy pillars of the programme, and their measurement will be compounded for both procedures and outputs. The measurement will be supported by quantitative and qualitative research methods and the estimated product of the evaluation of this programme will be a participation index. As the three aspects are meant to guide the construction of the participation index, each aspect will be provided with information from the measurement of procedures and outputs that will be worked through the evaluation. In sum, the index is understood as the parameter within which a complex range of information should converge and make sense of the success and performance of the programme from 2011 to 2017.
While data gathered through quantitative and qualitative methods are expected to provide a deeper understanding of the three aspects (and beyond), to date, the evaluation has not yet been completed and, therefore, there are no final results available for publication. Nevertheless, the reflection on the construction of the index can contribute to the ongoing debate on the evaluation of participatory processes in urban regeneration. On the quantitative side, three indicators will be calculated covering local partnerships, initiatives, and public funding, one for each aspect. The first indicator will define to what extent the programme encourages the generation of local partnerships. The indicator will consider raw data on local partnerships and initiatives (Fig. 3.), and provide disaggregated data on each priority area by means of a weighted calculation based on the number of local partners – i.e. the networking of agents and agencies corrected for duplications of local associations, NGOs, and parish councils (Portuguese freguesias) from annual programme to annual programme – and initiatives in the seven annual programmes.
The second indicator will provide disaggregated information on initiatives, and corresponding activities developed within the initiatives, during the seven annual programmes. As with the first indicator, raw data will be measured through a weighted calculation for each priority area (Fig. 4.).
Finally, the third indicator will measure the funding provided by the programme per resident population in the 67 priority areas. The resident population has not been included in Figure 5 as the corresponding value of 131418 residents in priority areas (out of 547733 residents in the municipality) was retrieved from the 2011 Census. To date, there are no available sources on the scale of the priority areas taking account of the demographic variations occurring throughout the seven annual programmes.
On the qualitative side, the index will consist of information retrieved from surveying and interviewing local agencies and agents engaged in the programme. To date, only the surveys of local associations, NGOs, and parish governments have been conducted. The survey was conducted both in open meetings and online, and addressed a total number of 567 local associations and NGOs registered in the municipal database of the programme, plus the 24 parish governments of the city. The mobile application ‘Sli.do’ allowed the surveying and collecting of answers during the Workshop for the wider public Bip/Zip ‘Divulgação e Capacitação: O Desafio do Desenvolvimento Local’ (‘Dissemination and Capacity building: The Challenge of Local Development’) organised by the Municipality of Lisbon on the 7th April 2018 . The survey contained the same questions as the online survey constructed using Google forms and distributed by the Municipality of Lisbon via email. A total of 159 completed surveys were collected online, while an average of 73.5 participants used the Sli.do application in the open meeting as it was not obligatory to respond to each question. The resulting average of local agencies surveyed reached 232.5, which ensures the validity of the results and represents almost 50% of the statistical universe of local associations and NGOs registered in the programme. It should be noted, however, that 21 of the responses were provided by parish governments, which slightly reduces the average to 211.5.
As part of the multidimensional evaluation of procedures and outputs, the surveys were designed to retain information about the three aspects identified above. Accordingly, the surveys posed questions concerning the making of local partnerships, the implementation of initiatives, and the provision of public funding. Emerging data from the surveys will be considered as part of a more comprehensive collection of information through quantitative and qualitative methods. One point highlighted from these surveys is that one third of the respondents declared that the local associations and NGOs they represented were created in the last seven years, which coincides with the introduction and development of the programme (2011–2017). An additional feature highlighted regards networking opportunities and connections with the same and/or other associations and NGOs beyond the programme, which is also considered as a positive result of their participation. Regarding the perception of performance and outputs of the programme, respondents showed high levels of satisfaction. The programme is generally referred to as a necessary tool for socio-territorial cohesion,
achieving the goal of combating socio-spatial inequalities. The programme allows for the inclusion of new initiatives in their work plan, which means that there has been consistent investment by partners in accomplishing their objectives. The consistency of the initiatives and the involvement of end-users and local communities at large in the deployment of the initiatives are considered to be positive outcomes. Despite assessing the funding allocated for the implementation of initiatives as adequate, when asked about needs that associations and NGOs would like to see answered in future annual programmes, responses are divided between more funding and more time for implementation.
The consolidation of information from quantitative and qualitative methods within the participation index is, at best, challenging. This operation will require that the data retrieved are weighted whenever needed, collocated within a common interval scale structured with ranges corresponding to degrees of participation to be agreed between the Municipality and the consultants, and transformed into a measurable parameter. Considering that the construction of the index is ongoing, it is not possible to share more detailed information about this process. Nonetheless, discussion on the conceptualisation and operationalisation of the index can positively contribute to the international debate on the measurement of participatory processes in urban regeneration.
Can scholars and practitioners find a reasonable way to translate highly complex socio-political phenomena characterising actions for urban regeneration into a measurable parameter? Is there any inherent risk in this ambition? Is this ambition likely to produce reliable results? These questions need to be addressed to understand whether the experience described in this article is scientifically valid. The participation index results from the application of a methodology that draws inspiration from the main literature on the topic and aims to accommodate theoretical inputs to empirical knowledge emerging from the field. Furthermore, the limited coverage on the evaluation of citizen participation in public policymaking in scientific and grey literature makes this index, as far as the author was able to establish, the first contribution of its kind in this field of study. The acknowledgment of the positive stimulus that the conceptualisation of the index can give to scholars, practitioners, political sponsors and citizens engaged in participatory processes in urban regeneration should not, however, underestimate the necessity of reflecting upon such a complex theme and its relevance in practice.
The need expressed by the local authority to provide policy-oriented information to allow greater access to, broader understanding of, and wider dissemination of the way the programme performed should be considered as paramount in this case. Consultants have taken into account this need, and have agreed on the concrete opportunity to conceptualise and operationalise the participation index by committing their work to designing a robust methodology for its evaluation. However, from the point of view of the research, the transformation of the range of data into one measurable parameter cannot help but raise some concerns. While this article shares this endeavour to improve discussion on the topic, it is useful to shed light on some questions that remain rather one-sided in this operation and that can be tackled in future research.
The measurement of participation promoted through the programme responds to the need to provide an evaluation with an extensive set of information on a multidimensional range of issues. While acknowledging the complexity of the programme, three aspects were identified to guide the definition of criteria and methods for the measurement of procedures and outputs, and ultimately provide policy-oriented knowledge through the participation index. The three aspects – partners, initiatives, and funding – will cluster the criteria selected to evaluate the programme, and it is legitimate to question whether this selection has prevented other aspects from gaining visibility. At first glance, there are issues concerning the effective improvement of the socio-territorial cohesion within and between local communities, and the ways in which socioeconomic and physical issues are addressed to enhance the quality of life in priority areas. Some of the concerns emerging from the conceptualisation and operationalisation of the participation index should be contrasted with the contribution to the international debate on the evaluation of participatory processes. While the index is unlikely to adequately cover all the multiple and complex dimensions that interplay within the programme, it seeks to provide an accessible and measurable range of information on procedures and outputs on the basis of an accurate collection of data.
Despite the quantitative and qualitative measurements converging into the index of participation and providing insights that partially respond to some of the following concerns, the evaluation provides a brief analysis of some elements of the broader context wherein the programme has been implemented. A first element regards the extent to which the implementation of the participation of local communities has been affected by the global financial and political crisis and, in turn, the ways the programme has effectively counteracted some of its harshest effects in priority areas. While the years of the financial crisis deeply compromised the quality of life in less prosperous urban areas, the post-crisis shows evidence of reversal and economic recovery. Portugal is today taken as an example of successful exit from the crisis, although commenters do not agree on a common version of this story and what actually led Portugal to recover over recent years. Was it austerity? Was it the expansionary economic agenda of the new government? Consistent answers to these questions should be critically considered according to the classification of priority areas. In fact, the impact of the crisis and the recent boom of the tourism industry is not the same everywhere, as it rather reflects the distribution of incoming financial capital in specific areas of the city that, in some cases, overlap with the priority areas.
This issue relates to the need for a clearer assessment of whether the engagement of local communities can be considered, globally, as part of the pervasive processes of devolution to communities via forms of self-governance; processes that are often decoupled, however, from wider debates on the impact of transnational strategies on socio-spatial inequalities. Considering that power inequalities may persist between local authorities and communities, more attentive analysis should be applied to the role of local partnerships in this programme. In particular, there should be space in the evaluation to consider how and to what extent the promotion of social entrepreneurship and capacity building through the initiatives involves a risk of conformity to some dominant ideas about local participation. As transnational agencies instil national and local authorities with apparently consensual conceptualisations of urban regeneration, it is legitimate to question if the programme has resulted in redistributive equity and what expected or unexpected forms of power have actually gained local partnerships through their action in the field.
Related issues that may be investigated in future research concern the emerging coincidence between the calendar of the programme and the work plan of some of the local partners. Was the decision to create associations and NGOs influenced by the possibility of applying for public funding via the programme? Were they created as a result of other initiatives funded via the programme? Future research could also show whether the NGOs simultaneously contributed to the regeneration of the priority areas via other policy programmes, as well as via grassroots initiatives. This information is a key factor in extending the potential of the programme so that it can contribute to wider action by both governmental agencies and civil society in urban regeneration. At the same time, this information could provide greater clarity on the mutual learning of the agents involved in the implementation of different initiatives, and the development of their capacities and skills to actively contribute to the regeneration of the territories.
Will the participation index respond to what the evaluation is expected to do in the field of citizen participation in urban regeneration in Lisbon? Will it provide robust information on the success of the programme, and the ways it reduces the extent and degree of priority issues, through a measurable parameter?
According to the main literature in this field of study, the evaluation should provide necessary data to understand whether participatory processes improve decision-making. However, participatory processes do not merely concern procedural aspects, as they encompass the ambitious aim to enhance democratic values at the local scale. Acknowledgment of both the policy-based and normative goals of the Bip/Zip programme has led the Municipality of Lisbon together with the consultants to identify three aspects to be considered the pillars of the programme: the local partnerships created through the programme; the initiatives implemented within the programme; and the public funding provided by the programme. The measurement of data through quantitative and qualitative methods will converge into the participation index, which is an ongoing process that aims to provide an accessible and measurable parameter for evaluation of the programme. The article discusses the conceptualisation and operationalisation of the participation index and recognises the necessity of supplying the local council, as well as the agencies and agents engaged in the overall programme with robust information about the performance and success of the seven annual stages of the programme.
Acknowledgment of the inherent complexity of participatory processes should not and cannot become a major hurdle for future debate on its evaluation. This contribution can positively stimulate sponsors and practitioners to transfer and replicate the methodology to other participatory processes and other contexts. In line with the claim for the systematisation of methods of evaluation of citizen participation, this article is intended to contribute to making progress towards the establishment of a theory of ‘what works best when’ or, at least, an approximate attempt at that.
Fundação para a Ciência e Tecnologia (SFRH/BPD/109406/2015); Department of Local Housing and Development of the Municipality of Lisbon; Nelson Dias, consultant coordinator.
Abelson, J. & Gauvin, F. P. (2006) Assessing the Impacts of Public Participation: Concepts, Evidence and Policy Implications Research Report P|06, Public Involvement Network. Available from: https://www.researchgate.net/publication/5100037_Assessing_the_Impacts_of_Public_Participation_Concepts_Evidence_and_Policy_Implications [accessed: 11.11.2016].
Aitken, D. (2012) Trust and Participation in Urban Regeneration People, Place & Policy, 6(3), 133–147. DOI: 10.3351/ppp.0006.0003.0003.
Arnstein, S. (1969) A Ladder of Citizen Participation Journal of the American Institute of Planners, 35(4), 216–224. DOI: 10.1080/01944366908977225.
Atkinson, R. & Kintrea, K. (2001) Disentangling Area Effects: Evidence from Deprived and Non-Deprived Neighbourhoods Urban Studies, 38(12), 2277–2298. DOI: 10.1080/00420980120087162.
Bailey, N. & Pill, M. (2011) The Continuing Popularity of the Neighbourhood and Neighbourhood Governance in the Transition from the `Big State’ to the `Big Society’ Paradigm Environment and Planning C: Politics and Space, 29(5), 927–942. DOI: 10.1068/c1133r.
Baumgarten, B. (2013) Geração à Rasca and beyond: Mobilizations in Portugal after 12 March 2011 Current Sociology, 61(4), 457–473. DOI: 10.1177/0011392113479745.
Brannan T., John, P. & Stoker, G. (2006) Active Citizenship and Effective Public Services and Programmes: How can we know what really works? Urban Studies, 43(5/6), 993–1008. DOI: 10.1080/0042 0980600676626.
Briggs, X. de S. (1998) Brown kids in white suburbs: Housing mobility and the many faces of social capital Housing Policy Debate, 9(1), 177–221. DOI: 10.1080/10511482.1998.9521290.
Buck, N. (2001) Identifying Neighbourhood Effects on Social Exclusion Urban Studies 38(12), 2251–2275. DOI: 10.1080/00420980120087153.
Burton P., Goodlad, R. & Croft, J. (2006) How Would We Know What Works? Context and Complexity in the Evaluation of Community Involvement Evaluation, 12(3), 294–312. DOI: 10.1177/13 56389006069136.
Caldas, J. C. (2012) The Consequences of Austerity Policies in Portugal Friedrich Ebert Stiftung, Berlin. Available from: http://library.fes.de/pdf-files/id-moe/09311.pdf [accessed: 10.09.2018].
Chess, C. & Purcell, K. (1999) Public Participation and the Environment: Do We Know What Works? Environmental Science & Technology, 33(16), 2685–2692. DOI: 10.1021/es980500g.
Cornwall, A. (2004) Spaces for Transformation? Reflections on Issues of Power and Difference in Participation in Development [in:] S. Hickey & G. Mohan, eds., Participation: from Tyranny to Transformation? Exploring New Approaches to Participation in Development Zed Books, London/New York, 75–91.
Curley, A.M. (2010) Relocating the Poor: Social Capital and Neighbourhood Resources Journal of Urban Affairs, 32(1), 79–103. DOI: 10.1111/j.1467-9906.2009.00475.x.
de Sousa, L., Magalhães, P. & Amaral, L. (2014) Sovereign Debt and Governance Failures: Portuguese Democracy and the Financial Crisis American Behavior Scientist, 58(12), 1517–1541. DOI: 10.1177/0002764214534666.
Dekker, K. & Bolt, G. (2005) Social Cohesion in Post-War Estates in the Netherlands: Differences between Socioeconomic and Ethnic Groups Urban Studies, 42(13), 2447–2470. DOI: 10.1080/0042 0980500380360.
Delli Carpini, M., Cook, F.L. & Jacobs, L.R. (2004) Public Deliberations, Discursive Participation and Citizen Engagement: A Review of the Empirical Literature Annual Review of Political Science, 7(1), 315–344. DOI: 10.1146/annurev.polisci.7.121003.091630.
Diamond, J. (2001) Managing Change or Coping with Conflict? Mapping the Experience of a Local Regeneration Partnership Local Economy, 16(4), 272–285. DOI: 10.1080/02690940110078274.
Durose, C. (2011) Revisiting Lipsky: Front-line Work in UK Local Governance Political Studies, 59, 978–995. DOI: 10.1111/j.1467-9248.2011.00886.x.
Falanga, R. (2018) Critical Trends of Citizen Participation in Policymaking: Caveats from Portugal [in:] M. Costa Lobo, F. Carreira da Silva & J.P. Zúquete, eds., Changing Societies: Legacies and Challenges Volume II. Citizenship in Crisis Imprensa de Ciências Sociais, Lisbon 295–318.
Falanga, R. & Lüchmann, L. (2019) Participatory Budgets in Brazil and Portugal: Comparing Patterns of Dissemination Policy Studies, ahead of print. DOI: 10.1080/01442872.2019.1577373.
Falanga, R. & Tulmello, S. (2017) Portugal and Austerity: What European Model? CURA Blog. Available from: http://cura.our.dmu.ac.uk/2017/11/20/portugal-and-austerity-what-european-model/[accessed 23.11.2018].
Fiorino, D. J. (1990) Citizen Participation and Environmental Risk: A Survey of Institutional Mechanisms Science, Technology and Human Values, 15(2), 226–243. DOI: 10.1177/016224399001500204.
Fishman, R.M. (2011) Democratic Practice after the Revolution: The Case of Portugal and Beyond Politics & Society, 39(2), 233–267. DOI: 10.1177/0032329211405439.
Fonseca, P. & Ferreira M.J. (2015) Through ‘seas never before sailed’: Portuguese government discursive legitimation strategies in a context of financial crisis Discourse and Society, 26(6), 682–711. DOI: 10.1177/0957926515592780.
Forrest, R. & Kearns, A. (2001) Social Cohesion, Social Capital and the Neighbourhood Urban Studies, 38(12), 2125–2143. DOI: 10.1080/00420980120087081.
Fraser, N. (2009) Social Justice in the Age of Identity Politics [in:] G.L. Henderson & M. Waterstone, eds., Geographic Thought: A Praxis Perspective Routledge, New York, 72–91.
Granovetter, M. (1974) Getting a Job: A study of Contacts and Careers Harvard University Press, Cambridge.
Häikiö, L. (2012) From innovation to convention: legitimate citizen participation in local governance Local Government Studies, 38(4), 415–435. DOI: 10.1080/03003930.2012.698241.
Iveson, K. (2011) Social or spatial justice? Marcuse and Soja on the right to the city City, 15(2), 250–259. DOI: 10.1080/13604813.2011.568723.
Lapavitsas, C., Kaltenbrunner, A., Labrinidis, G., Lindo, D., Mead-way, J., Michell, J., Painceira, J.P., Pires, E., Powell, J., Stenfors, A., Teles, N., & Vatikiotis L. (2012) Crisis in the Eurozone Verso, London/New York.
Lefebvre, H. (1968) Le Droit à la ville Anthropos, Paris [in French]. Lowndes, V. & Sullivan, H. (2008) How Low Can You Go? Rationales and Challenges for Neighbourhood Governance Public Administration, 86(1), 53–74. DOI: 10.1111/j.1467-9299.2007.00696.x.
Marcuse, P. (2010) From critical urban theory to the right to the city City, 13(2-3), 185–197. DOI: 10.1080/13604810902982177.
Maloney, W., Smith, G., Stoker, G. (2000) Social Capital and Urban Governance: Adding a more contextualized “top-down” perspective Political Studies, 48(4), 802–820. DOI: 10.1111/1467-9248.00284.
OECD (2005) Evaluating Public Participation in Policymaking Paris. OECD (2015) Reducing Inequality and Poverty in Portugal. Economics Department Working Papers 1258 OECD, Paris.
OECD (2015) Reducing Inequality and Poverty in Portugal. Economics Department Working Papers 1258 OECD, Paris.
Power, A. (2000) Poor areas and social exclusion [in:] A. Power & R.W. Wilson, eds., Social Exclusion and the Future of Cites CASE Paper 35, London School of Economics, London, 1–20.
Purcell, M. (2002) Excavating Lefebvre: The right to the city and its urban politics of the inhabitant GeoJournal, 58(2-3), 99–108. DOI: 10.1023/B:GEJO.0000010829.62237.8f.
Purdue, D. (2001) Neighbourhood Governance: Leadership, Trust and Social Capital Urban Studies, 38(12), 2211–2224. DOI: 10.1080/00420980120087135.
Putnam, R.D. (2000) Bowling Alone: The Collapse and Revival of American Community Simon & Schuster, New York.
Rosener, J.B. (1978) Citizen Participation: Can We Measure Its Effectiveness? Public Administration Review, 38(5), 457–463. DOI: 10.23 07/975505.
Rosener, J.B. (1981) User-oriented Evaluation. A New Way to View Citizen Participation Journal of Applied Behavioral Science, 17, 583–596. DOI: 10.1177/002188638101700412.
Rowe, G. & Frewer, L.J. (2004) Evaluating Public-Participation Exercises: A Research Agenda Science, Technology & Human Values, 29(4), 512–556. DOI: 10.1177/0162243903259197.
Soja, E. W. (2010) Seeking Spatial Justice University of Minnesota Press, Minneapolis.
Swyngedouw, E. (2005) Governance Innovation and the Citizen: The Janus Face of Governance-beyond-the State Urban Studies, 42(11), 1991–2006. DOI: 10.1080/00420980500279869.
Taylor, M. (2007) Community Participation in the real World: Opportunities and Pitfalls in New Governance Spaces Urban Studies, 44(2), 297–317. DOI: 10.1080/00420980601074987.
Wallace, A. (2010) New Neighbourhoods, New Citizens? Challenging ‘Community’ as a Framework for Social and Moral regeneration under New Labour in the UK International Journal of Urban and Regional Research, 34(4), 805–819. DOI: 10.1080/00420980601074987.
Webler, T. (1999) The Craft and Theory of Public Participation: A Dialectical Process Journal of Risk Research, 2(1), 55–71. DOI: 10.1080/136698799376989.
Webler, T. & Tuler, S. (2002) Unlocking the Puzzle of Public Participation Bulletin of Science, Technology & Society, 22(3), 179–189. DOI: 10.1177/02767602022003002.
Weiss, C.H. (2004) Rooting for Evaluation: A Cliff Notes Version of My Work [in:] M. Alkin, ed., Evaluation Roots Sage, Thousand Oaks, 154–168.