Since evaluation and good project management go hand in hand, you should be able to answer all of the following questions, and in the process design both your project and your evaluation. For illustrative purposes, examples have been drawn from the Graphic Science project “Science on the Buses”.

Aim

What is the aim of your project? Can it be easily stated in a few words? Can you point to some outcome that you believe can be achieved?
E.g. To increase the level of participation in informal debates around science.

Audience

Do you have a clearly defined audience in mind? Are they able to deliver the aims above?
E.g. Young adults (16 – 35)

Message

What message do you wish to take to the audience? Does it support the aims above?
E.g. Science is a major part of daily life.

Medium

How will your message be taken to the audience? What sort of activities will you use?
E.g. Posters inside buses

Indicators

What are the indicators of success? How will you know if you have reached the audience in mind with the message you had planned?
E.g. Members of the target audience will suggest that science is an integral part of everyday life.

Measures

How will you measure the indicators above? What results will be considered a success?
E.g. In questionnaire based interviews conducted on buses showing the campaign, a substantially higher proportion of the target audience will respond as above when compared to other age groups.

With all of the above built into your project plan, it should be clear to you where and how you will need to collect data to evaluate the project. There are a number of methods available, and your choice should be determined by the nature of the information you wish to illicit and the nature and needs of the audience. Some of the most popular methods are:

Count them

A simple record of the number of people who have attended an event, been exposed to a poster etc. Very little can be said about the experience of the audience or what impressions they take away with them. This method is cheap, simple and most useful when describing the scale of a project.

Questionnaires

A simple questionnaire can produce some fairly detailed information on how an audience has reacted to a project. However, questionnaires must be very brief if they are to be filled in by members of the public, relying for the most part on “closed” questions (i.e. responses are chosen from a list). Response rates will usually be very low (less than 10%)

Interviews

Talking to people and filling in questionnaires as you go along (so called questionnaire based interviewing), will generally allow you to probe more deeply and ask more “open” questions (i.e. responses are statements framed by the respondent). Care should be taken with questionnaire design and with the conduct of the interviewers to avoid bias. This technique is often very resource intensive.

Focus Groups

Focus groups can produce very detailed information. However, they are difficult to arrange and manage and results should be analysed with great caution. Ideally, this work should be undertaken by professionals.

Press Cuttings

Media surveys will tell you very little about the effectiveness of your project (unless the media were your target audience). However, press cuttings can be a very influential chapter of your final report, especially if you need to demonstrate that the project really took place.

Depending on the scale of your project, you may find yourself with a greater or lesser amount of raw data. However you choose to analyse your data, it is advisable to always enter it into a spreadsheet; this has the advantage that other people can check your results or possibly suggest new ways to investigate the data you already have.

Data entry is made much easier if you number, not only the questions on a questionnaire, but also each of the possible responses in closed questions.

Open responses can also be entered into spread sheets for keyword searches and so on. Open responses can be “coded” for easier analysis. This involves reading through those responses and looking for commonalities. Where it seems to you that a group of respondents are broadly in agreement, each of these comments can be assigned a standard code letter, and treated as a statistical group.

Coding is a delicate matter of interpretation and should conform to a well thought out set of coding criteria. It is advisable after coding to ask an impartial colleague not familiar with the project to code the same responses using your criteria and then check the two versions for conflicts.

Reports should be written in a clear, accessible style, and should be as short as possible, without skipping any important details. The length of the evaluation report should be proportional to the overall budget for the project. Try to make your evaluation report separate from your reports on financial and other management matters.

You should always include a short summary of your main findings and conclusions. Describe your methodology briefly and say why you chose it. Present your findings clearly, using tables and charts ONLY if they are obvious and will reduce the word count substantially. Always ensure that your conclusions actually do refer to findings presented in the body of the report.

You should try to be self critical throughout this process. Remember that any mistakes you have made will certainly be repeated by others if they cannot learn from your experience. Try to disseminate your report; for example, you might offer it, via the PSCI-COM email list, to anyone interested in seeing it, or, if you can summarise your experiences fairly fluently, why not offer it to Science and Public Affairs?

Above all, be honest. Your report is your opportunity to tell the world exactly what you did and why. No project has ever been an effortless road to glory, and many others will be interested to learn from your experience. Remember that funders, in particular, are always pleased to find project managers who have learnt valuable lessons from their experiences.