On Evidence-Based Practice

Revellers dressed as Mozart (or the female equivalent) dig up a gas main in central London sometime in the 1930s. Evidence of not much at all.

   "Now, what I want is, Facts. Teach these boys and girls nothing but Facts. Facts alone are wanted in life. Plant nothing else, and root out everything else. You can only form the minds of reasoning animals upon Facts: nothing else will ever be of any service to them. This is the principle on which I bring up my own children, and this is the principle on which I bring up these children. Stick to Facts, sir!"
    - Thomas Gradgrind, in Hard Times by Charles Dickens (1854)

Several days before I wrote this piece, the British Government suppressed a report it had commissioned on immigration into the United Kingdom. News of this was leaked to the press, and BBC television interviewed a government spokesperson, Mr Vince Cable, Secretary of State for Business, Innovation and Skills and a Liberal Democrat politician. Mr Cable described the report as "one-sided" and was very much at ease with the fact that the Government had had it rewritten twice and was reluctant to issue the final product in any shape or form. The report (which I cannot reference) consisted of a wide-ranging survey of the evidence on immigration to the UK. According to the television news, it concluded that immigration has had a range of positive benefits for British society and economy. It also noted that immigrants use the National Health System less than indigenous Britons and contribute very much to it in terms of their skills and labour. This is in contrast to the Government's legislation to curb "benefits tourism" (a phenomenon that the report notes is largely non-existent).

The reason for mentioning this case here is that the British Government commissioned a review of evidence and then tried to alter and suppress it because the evidence ran counter to its policies. The evidence could not be used to support an anti-European or xenophobic stance, nor to increase the anxieties of voters about multiculturalism and the shortage of jobs.

As it happens, I do believe in "evidence-based practice". Logic demands that we take experience into account and that we consider all relevant knowledge pertaining to a problem before we decide how to solve it. Without such an approach, policy makers risk blundering around in the dark, and their policies risk being, at best, inefficient, and at worst downright injurious. However, there are two main problems with evidence-based practice. One concerns the nature of evidence and the other refers to the way in which it is, or is not, used.

What is evidence?

It is axiomatic that policy and practice should be based on as complete a knowledge of a problem as the evidence will allow. That is why policy formulators use academics and advisors, because they have a wide-ranging knowledge of the problem in question, its connotations and the evidence that, properly interpreted, can lead to a solution.

However, for any problem in society, economy and ecology that begs to be solved, there are at least nine important questions that may well lack an adequate answer. They are as follows.

• What exactly is evidence?
• To what extent is evidence a surrogate for direct experience, or, alternatively, how much evidence should be derived from experience and how much from indirect sources?
• How should evidence be verified?
• Leading on from the previous question, is 'evidence' merely objective data, or does it include subjective experience?
• Evidence of what? To what should the evidence be attributed?
• What is evidence capable of proving or confirming?
• What is the connection between evidence and wisdom?
• Can we do without evidence?
• Lastly, how much evidence is enough before decisions can be made?

Clearly, the answers to these questions will differ from case to case. In general, 'evidence' is information that is capable of contributing to the solution of the problem, which has been obtained by objective methods and that paints an objective picture of the situation under examination, and one that is as complete as needed in order to draw conclusions, formulate policy and develop strategies to implement a solution.

Merely trawling for data does not adequately define the process of compiling evidence. On the other hand, the inevitable resort to selectivity risks the introduction of bias into the process of accumulating evidence. Moreover, as risk analysis involves risk perception, and as risk perception has a strong influence on how risks are communicated and managed, then subjective experience is clearly part of the 'evidence' in some way. 'Wisdom' is therefore the process of sifting and selecting evidence in an impartial and even-handed manner. In the modern world, information technology has promoted a massive return to the kinds of inductive science that were common in the times of the Encyclopédistes of the eighteenth century. Computers have taken the hard work out of blind analysis of data, but they have also removed the thinking. In intellectual terms, there is nothing more feeble and pathetic than data mining.

How should evidence be used?

There are three kinds of evidence:-

• precise and decisive
• equivocal, ambiguous and puzzling
• uninterpretable (evidence of what?).

Data  are a low-level form of evidence and may not be enough to form an adequate generalisation about a phenomenon. Disaster impacts have two unfortunate features: over time, they are spiky and they have a (somewhat ambiguous) trend. Hence, it can be difficult to make a generalisation about the future on the basis of evidence derived from the past. This was illustrated by an editorial in a journal, which congratulated the world on reducing disaster death tolls to 59,000 a year over the previous five years (Wilson 2005). It was published just as the Indian Ocean tsunami killed more people in one catastrophe than had died in all disasters during the previous 60 months. Unfortunately, for many phenomena, evidence alone will never be sufficient to characterise them, especially if their mean values trend over time. Hence, we need evidence, models and inspiration. As all use of evidence is selective, the criteria by which facts are selected should be made explicit so that they can be evaluated. In short, evidence can constrain uncertainty, but it cannot eradicate it.

A cautionary tale

The Irish engineer Robert Mallet developed a strong interest in earthquakes. Indeed, he is to some extent the "Father of Seismology". One of his greatest achievements was to compile all the known evidence of earthquakes into a catalogue and map. Mallet knew  the location of plate boundaries before anyone knew of the existence of tectonic plates. In December 1857 the Italian region of Basilicata was struck by a major earthquake that killed about 5,000 people. Mallet organised an expedition there and assiduously collected evidence, often at great personal hardship, which he published in two volumes that have become classics of observational science (Mallet 1862). Mallet missed no piece of evidence, however trivial, but he was unable to deduce the cause of earthquakes, which he thought had something to do with subterranean steam. It took John Milne (1850-1913) to do that, in concert with a number of other scientists, using a new and more sensitive kind of seismograph.

Let's ignore the evidence

Here is a more modern case, but one that harks back in its structure to the immigration question with which I started this piece. Since 2006 there has been a set of international regulations that prohibit passengers from taking bottles of liquids larger than 100 ml onto flights. It stems from some assumptions about how liquids could be mixed on board an aircraft to make a bomb. To begin with, the standard size of bottles, at least in the European Union, is 60 an 120 ml. Hence, vast numbers of the latter have had to be thrown away at airports (the container size is what matters, not the amount of liquid inside the bottle). Secondly, in terms of concocting a bomb, 100 ml is definitely not a magic number. I asked a highly experienced counter-terrorism artificer about this and he told me that 25 ml of certain substances would be sufficient. I do not know whether one could buy the relevant substances in the airport pharmacy, having already passed security.

I mention this example because there is virtually never any attempt to evaluate the efficiency and effectiveness of counter-terrorism measures. The evidence is secret, or perhaps merely lacking.

Let's ignore the evidence when it hits us in the face

A report from the UN International Strategy for Disaster Reduction states that "The City of Venice joined the [Safe Cities] Campaign as a role model for cultural heritage protection and climate change adaptation" (UNISDR 2013). Indeed, such an example is Venice that it appeared on the cover of this document. In reality, Venice is severely threatened by the arrival of cruise ships of up to 140,000 tonnes in size, which navigate within one metre of the historical urban fabric, causing damage with their bow waves and creating a massive risk of collision and shipwreck. Despite the example of the Costa Concordia (the world's most expensive shipwreck), in 35 years of debate the city council has refused to legislate on this issue. Protests by Venetian residents have turned violent but there has been no change. Indeed, no attempt was made adequately to regulate the chaotic water transport on the Grand Canal until an eminent German was crushed and drowned in the collision between two boats. Meanwhile, the mayor, Sig. Giorgio Orsoni, has resigned after being arrested in a corruption investigation regarding the city's flood defences. So much for evidence-based practice, both within Venice and looking in.

Conclusion

Evidence-based practice is a good idea providing we are not too naive about it. Any attempt to collect, martial and interpret evidence on a particular problem needs to be transparent, fair and impartial. It must state the criteria by which evidence is included and excluded, and must ensure that an objective, balanced view of the problem is compiled. Besides the fact that they are grossly inefficient, inductive and aductive processes will not automatically ensure this. A 'blind' approach to evidence will not make it objective or comprehensive, because choices inevitably have to be made in the way that evidence is collected.

Lastly, examples described in this essay illustrate the fact that evidence alone does not "shame" policy makers into adopting a better, more objective approach. They are perfectly at liberty to use evidence selectively, or ignore it altogether.

Hence, we need an evidence-based investigation of exactly how and why policy makers ignore or manipulate the evidence.

References

Mallet, R. 1862. Great Neapolitan Earthquake of 1857: The First Principles of Observational Seismology. Chapman and Hall for the Royal Society, London, 2 vols.

UNISDR 2013. Making Cities Resilient: Summary for Policymakers. A Global Snapshot of How Local Governments Reduce Disaster Risk. United Nations International Strategy for Disaster Reduction, Geneva, 20 pp.

Wilson, H.C. 2005. Editorial. Disaster Prevention and Management 14(1).
http://www.emeraldinsight.com/journals.htm?issn=0965-3562&volume=14&issue=1&articleid=17096777&show=html (accessed 21 July 2014).

Transitional housing, time travel and 'lost worlds'

Transitional housing, time travel and 'lost worlds'

While conducting fieldwork in central Italy some years ago, I met a priest whose name was Dante Paolino. Don Paolino was very fond of his namesake, the 14th-century poet Dante. He presented me with a copy of a book he had written and published, entitled The Divine Comedy Brought Up to Date. In this, he used Dante's great masterpiece as a vehicle for commenting on the wiles of the modern world. Readers with a literary disposition may remember that in the Divine Comedy Dante recounts how he is conducted around Hell, Purgatory and Heaven by his mentor, the Roman poet Virgil. The poem is very much a commentary on life and mores in mediaeval Tuscany. In Don Dante Paolino's 20th century version, one of the circles of hell is entirely populated with earthquake victims from the South of Italy (he wrote it shortly after the 1980 Irpinia-Basilicata earthquake, which damaged or destroyed the homes of 400,000 people).

In examining transitional shelter, one is motivated to ask what is 'transitional' for people who lack the resources to get out of it and have been abandoned by their government? Is the answer Hell or merely Purgatory? Sudden-impact disasters can abruptly cast people from a stable existence, one that is normal in terms of their expectations, and those of the communities in which they live, to one that is anything but stable and normal. Stability may be achieved by a form of 'suspended animation', in which a temporary situation 'freezes' and becomes permanent.

At the turn of the millennium, the city of Messina still had some of the temporary accommodation erected after the 1908 Strait of Messina earthquake and tsunami. Granted, it was no longer used as primary housing, but it was still there. Likewise, vestiges of the temporary accommodation can be found in the vicinity of Avezzano, central-southern Italy, even though a century has passed since the 1915 earthquake, which killed 32,500 people and devastated the town.

Romagnano al Monte is one of the smallest municipalities in the Region of Campania, southern Italy. Its fiscal position is gloomy and its political weight is negligible. Hence, 35 years after the 1980 earthquake that devastated the town and necessitated complete evacuation of the original urban fabric, the wooden prefabs are still in situ. The town hall and local coffee bar are still in prefabricated buildings erected by the Italian government at Christmas 1980. To be fair, part of this is a matter of convenience. Prefabs that served as people's transitional homes have been turned into holiday accommodation, but they are still a visual reminder of a long phase of waiting in "transitional" circumstances, while reconstruction slowly, painfully got underway.

The town coffee bar at Romagnano al Monte, southern Italy,

a prefab erected after the 1980 earthquake and photographed in 2011.

Italian Government policy on transitional shelter in the aftermath of the L'Aquila (central Italy) earthquake of 2009 is described in more detail elsewhere in this book and is interesting because it is so radical. Lavishing very large sums of money on transitional shelter can be interpreted as a message to the beneficiaries that the temporary situation is designed to last. It is an expression of pessimism in the ability to find the necessary money and political drive to achieve full recovery in any reasonable length of time.

Once, during a tour of the backwoods of rural Calabria (southern Italy), I drove around a corner and came face to face with a group of about 20 prefabs, nestling in a valley. I found out that they had been built for families made homeless by a large landslide about 25 years previously. There they lay, fully inhabited and completely forgotten by the rest of society. How many more "lost worlds" of this kind are there?

Transitional shelter is an artefact of wealthy countries. Units usually cost upwards of US$20,000, and possibly much more than that, with transportation and site preparation costs included. Whereas the problem of transitional shelter that does not adequately fit the needs of its beneficiaries is well known in the context of donor countries supplying it to poor nations that have suffered disaster, there is a parallel set of issues associated with the domestic use of shelter in rich, or relatively rich, countries.

Barrack-style temporary housing was supplied to homeless survivors of the 1968 earthquakes in the Belice Valley of western Sicily. It was unpleasant to live in, cramped and poorly situated with respect to people's needs (such as travel to work, agricultural activities and shopping for basic necessities), but it was a roof over their heads at a critical moment. Predictably, as soon as alternative accommodation was arranged, it was abandoned. On a blisteringly hot day in the summer of 1983 I visited the prefabs and found them silent, deserted and empty amid the parched fields. Later, with the influx of illegal migrants from North Africa, they were recolonised by groups of people who were lower down the social scale than the original beneficiaries. In similar manner, in central Italy, the land abandoned after the massive landslide of 1982 in the city of Ancona was briefly colonised by itinerant groups of Roma, who had nowhere else to pitch camp.

These are situations in which transitional housing has contributed to a 'ghettoisation' of the disaster area, which may, as time goes on, accumulate a sense of relative permanence. Thus, transitional housing is capable of creating a new stratification in local society—between the upper caste, who did not lose their permanent homes, and the lower caste, who did, and are forced to live in the prefabs. Community spirit and solidarity are not well served by such a distinction and one hopes that recovery policies will eliminate it as soon as possible by doggedly pursuing full reconstruction and eventually dismantling the prefabs. Yet this is often not the case.

In many countries where there is substantial poverty, a significant proportion of the population lives in so-called 'informal' housing. In cities such as Rio de Janeiro this can attain remarkable levels of social and architectural sophistication, while at the same time retaining all the crude drawbacks of precarious, unplanned urbanisation. At its very worst, the permanence of transitional housing in richer countries could be regarded as a regression to something akin to the 'informal' settlements of less fortunate countries. Granted, it does not go all the way and is a far cry from the lawlessness and destitution that characterise such places in many of the world's developing metropolises. However, if nothing else, the presence of these situations, frozen in time, and the social realities that they represent, call into question the definition of welfare.

Where it exists, welfare is the social safety net. It can be defined as "the provision of care to a minimum acceptable standard to people who are unable adequately to look after themselves." This begs the question of what the 'minimum acceptable standard' should be, and what it entails? The answer is highly variable from one country to another, even among wealthy nations. With increases in global mobility—and people-trafficking—there is also a burning question about to whom welfare applies. In many respects, the solution is a matter of determining what welfare is not, instead of what it is.

Transitional housing is usually donated to its users. They do not have to buy it and probably do not pay rent to live in it. Local authorities may be charged with maintaining it, unless a central government agency takes over this function. In the case of the Great Eastern Japan earthquake and tsunami of March 2011, transitional housing was supplied to many people and families whose homes had been swept away by the waves. It was designed and supplied under central government auspices. it was highly standardised and a balance was struck between functionality and cost. Living in the transitional units was unpleasant, but it embodied a pact between the residents and the Japanese Government: endure these conditions for up to seven years, but no more, and you will move into safe, acceptable permanent housing. Hence, the key to understanding transitional housing lies in whether there is a relationship of trust between the users and the government (at whichever level), an how strong that relationship is.

Transitional housing near Ishinomaki, northeast Japan, in 2013.

The welfare function of transitional housing in cases that differ strongly from the Japanese example is, in effect, truncated. The Government is telling survivors, in so many words, "we will start you off on the road to recovery by providing basic shelter" but the rest is up to you, and if you do not have the resources to progress beyond this stage, this is where you will remain."

Paradoxically, this sort of failure of welfare induces dependency. Beneficiaries can go neither forward nor backward. They are politically weightless, not empowered, not listened to. It is to be expected that such situations are most common in areas that are backwaters in a country's political and economic life. From the point of view of a government administrator, or a politician seeking re-election, it does not matter if the inhabitants of a forgotten mountain valley are left in limbo. Yet occasionally such situations turn around. The inertia in western Sicily after the 1968 earthquakes became a festering problem that forced the Italian Government to act 15-20 years after the disaster, which it did using what Americans will recognise as 'pork-barrel legislation', the enactment of measures on the back of provisions for other disasters. Elsewhere, the plight of the 'transitional tribes' will become a matter, not so much for anthropologists, but for archaeologists!