GEM TOOL> PHASE 2> STEP 5

How to Decide which Methods/Tools to Use?

Deciding the appropriate data-gathering tools should be based on the identified intended use/s and user/s; on the kind of data intended users will find relevant and useful; and on how well the intended use and users of your evaluation results have been articulated.

Still another basis for selecting and designing data-gathering tools can be based on the indicators that were formulated in the previous phase. Based on these, define what type/s of data are needed, and the main source/s of information.

ACTIVITY

EXPLORING EXAMPLES OF PRACTITIONERS’ METHODOLOGIES

Below are examples on how two GEM testing partners, Women Mayors’ Link and Mothers 4 Mothers, developed their data-gathering strategies.

WOMEN MAYORS’ LINK (WML)

WML is an initiative of the Stability Pact Gender Task Force (SP GTF), a project developed in 12 countries and territories of the Stability Pact (SP) Region with the Equal Opportunities for Women Foundation (SEF) as the lead organisation. Set up in 2002, the purpose of WML is to foster cooperation between women mayors and local governments and local women’s networks in preparing small projects for improving the quality of life of women and children in local communities. Its aims are to initiate and facilitate regional and international exchange of best practices in similar projects; lobby for a better representation of women in local government; and support efforts of women mayors in increasing people’s participation in the problem-solving process of their communities.

The WML’s evaluation team used a combination of methods to collect both quantitative and qualitative data. The busy schedules of the mayors and geographical location of the countries and territories were factors that influenced the methodologies that were used. (The respondents were 50 mayors who came from the Stability Pact countries and territories: Albania, Bosnia and Herzegovina,Bulgaria,Croatia, FYRoMacedonia, Greece, Hungary, Kosovo/a, Moldova, Romania, Serbia and Montenegro, Slovenia.) The team drafted two sets of surveys. One questionnaire explored the networking process within the WML which was posted on the web and list-serve, and was also sent by email and snail mail to all the women mayors in the network. The other questionnaire probed how the initiative helped mainstream gender in local governance which was distributed at the international conference “Twinning and Partnerships Initiatives of Women Mayors – Matching Conference” held in Romania, November 20-23, 2003. Most of the responses were also gathered there.

In addition to the surveys, the team also collected data on the types of internet connection the women mayors used, put together a comparative analysis on the costs of telecommunication services in their localities, and conducted background research on gender mainstreaming in local governance.

In sum, below is the list of tools WML used to collect data:

• Questionnaires on the networking process that were sent by email, posted on the web and list-serve and sent by post to all women mayors WML were in contact with
• Questionnaires related to the Partnerships’ Building to foster mainstreaming of gender in local governance that were distributed at the international conference
• Research to identify types of internet connection used by women mayors involved in the project
• Research to develop a comparative analysis on the costs of telecommunication services
• Research on mainstreaming gender in local governance

 

(Read the full evaluation report at http://www.apcwomen.org/gem/
practitioners/reports.shtm?x=51601 )

 

MOTHERS 4 MOTHERS (M4M)
COLLECTING QUANTITATIVE INDICATORS	FINDING QUALITATIVE INFORMATION
M4M developed a survey with a long questionnaire that looked into the factors affecting teleworking. A total of 141 potential respondents for the survey were reached by phone, fax or email from 8 July to 1 September 2003. The potential respondents all worked at home: running their own registered business, freelancers, part-timers, and those employed by M4M. Questionnaire form The questionnaire had a total of 83 openended and factual (dichotomy/multiplechoice) questions. Forty-two questions or 50% required the respondents to use the Likert scale, 37 or 45% were factual questions and the remaining four or five percent were open-ended questions. Mode of execution From the M4M network, a contact list of potential respondents was drawn up together with their email addresses and contact numbers. Email was the primary mode of getting in touch with respondents. Phone calls were resorted to where email addresses were unavailable. There were two respondents who requested the documents be faxed to them because they could not retrieve the documents due to software incompatibility or problems with their dial-up connection. A random screening on the current job status of potential respondents was made to determine their suitability as respondents to the survey. Potential respondents who met the survey requirements were asked to fill up the questionnaire form. Follow-up action The first set of emails sent to potential respondents introduced the purpose of the survey. Phone calls were made to follow-up their willingness to participate in the survey. Reminders were sent in the absence of a reply from respondents after seven days. Some respondents were sent up to four reminders to respond to the questionnaire form. On average, each respondent was reminded at least twice by email and at least once by phone.	Seven face-to-face interviews with the virtual team (VT) and one interview using online Yahoo messenger were held with the respondents. The objective was to identify the challenges faced by teleworkers within a structured organisation. Identify problems faced by the VT To find out the problems of the VT in teleworking full-time; to get VT’s suggestions on the possibility of teleworking full-time and ways of improving their situation Impact of teleworking on women’s lives To find out the impact of teleworking on women’s lives: Does it change genderrelations at home? To what extent do gender issues affect teleworking and how does teleworking through ICTs change women’s lives and status? Observe home office set-up and home situation To observe home office set-up and home situation and ask the possibility of promoting teleworking full-time Home office set-up To find out the conditions of computer availability (e.g. sharing computer), location (e.g. stable internet access), etc. Home situation To find out the conditions in the working environment (e.g. noisy, full of distractions); or pinpoint other barriers that impede work Identify personality of teleworker To determine the most common traits of teleworkers (e.g. whether teleworker loves working with a computer to check email, to surf, etc.) even during holidays Identify training and support needs of VT To find out the kind of training and support needed through observation and interviews

Additional Guidelines for Selecting Methods for Gender Evaluation:

• Choose appropriate and relevant methods

Evaluations carried out from a gender perspective of ICT initiatives include telecentres in rural or indigenous communities, a women’s global network or an online resource centre. Choose data gathering tools based on their appropriateness for different kinds of initiatives. The most effective methodologies are those that are flexible and adaptable, simple to administer, designed to draw meaningful results, and are appropriate and relevant to the intended use and users of the evaluation.

• Choose methods that are participatory

Participatory methodologies are those that allow all the defined users/stakeholders to submit data and information.

Think about the intended respondents and their context when deciding which methods to use. For instance, while online surveys are economical and time-efficient, it is an inappropriate method if the intended respondents do not have regular access to the internet. Make sure that the tools used are accessible to the respondents.

• Use multiplicity of methods/tools

Use multiple methods to help test, correct and correlate messages and data from different sources of information.

“When thinking about choosing sources of information keep in mind that what matters in the end is not the validity of individual sources of information, so much as the coherence and consistency of the information from different sources taken together. It is the pattern that matters

Using multiple methods can help test, correct and correlate messages coming from different sources of information.” [ Evaluation and Effectiveness 36, 50]

In all cases, methodologies should focus on evaluating both the product and the process: what has been achieved so far, and the way it has been achieved as well as how the methods keep evolving. Information on those two aspects reveals social processes.

• Ensure collection of sex disaggregated data

This is basic to any gender evaluation. All data gathered should at the very least identify the sex of the respondent. Other basic data about the respondents that may prove relevant to the evaluation include: age, religion, ethnicity, nationality, marital status, occupation.

• Identify female informants

Gender evaluations should draw out the experiences and input of female respondents/stakeholders.

• Interrogate gender roles

The instruments used should address the gender and ICT issues of the initiative or project, and must probe into broader gender issues. For example, in assessing the impact of an ICT training initiative, it is not only important to look into what the trainees have learned but also how they have applied their knowledge in their work or organisation. In order to assess this, it is essential to probe into the gender roles within the trainees’ organisations and look at how they are able (or unable) to practise their newly-acquired skills.

• Be context sensitive

Group dynamics, subject matter, gender, class, caste, age, race, language, culture, rural/urban issues, etc. greatly influence how effectively and inclusively information is gathered.

• Emphasise qualitative data

To get a complete picture of the social transformation issues and gender issues in a project or initiative require more than numbers and statistics. Stories, perceptions, observations and opinions are valuable. They give the human dimension behind the statistics – a crucial part to understanding collected data.

• Practical considerations

The following are important practical considerations in planning your data gathering strategies:

• Affordability – what is the cost of obtaining the information in relation to its contribution to the evaluation?
• Time and Timing – how much time does it take to gather the information? are there forthcoming activities that will provide opportunities to hold evaluation activities?
• Frequency – the number of times mid-project assessments and monitoring will be held

Other data gathering methods are available online. Try to take a look at the following resources:

• Utilisation-Focused Checklist: Evaluation Design

http://www.wmich.edu/evalctr/checklists/ufechecklist.htm#7

• Guidelines for the Integration of Gender Issues into the Design, Monitoring and Evaluation of ILO Programmes and Projects

http://www.ilo.org/public/english/bureau/program/eval/guides/gender/index.htm

• Gender issues in design, monitoring and evaluation

http://www.ilo.org/public/english/bureau/program/eval/guides/gender/issues4.htm#n4 http://www.ilo.org/public/english/bureau/program/eval/guides/gender/issues4.htm