Showing posts with label Methodology. Show all posts
Showing posts with label Methodology. Show all posts

Tuesday, May 22, 2018

On the Difference Between Method and Methodology




"What is the difference between methods and methodology in research?" This question was very recently asked in LinkedIn's "Research, Methodology, and Statistics in the Social Sciences" Group, which currently has 137,052 members. A number of answers were given, including one from me. I've chosen to share mine below, preferring to start with Methodology: 

Methodology is the study or theory or interrogation of a range of Methods of scientific/scholarly research in terms of their relative/respective appropriateness to solving specified types or sets of research problems. Methods, on the other hand, are the actual techniques (not 'tools', which are a different kettle of fish) of data collection and analysis, respectively, that we use in particular studies. 


In principle, the Method(s) we use in a given study is/are determined to be fit for the (stated) purpose -- the problem statement -- in terms of the Methodology or Methodologies they are couched in and, moreover, broadly accepted by the Gatekeepers (our peers and revered precursors) in our respective disciplines. 

It is common knowledge that, nowadays, Methodologies and Methods cut across various disciplines and sub-disciplines.

Tuesday, March 08, 2016

CSO 302: Qualitative Research Methods ~ CAT Questions for Jan-Apr 2016 Semester ~ UoN

UNIVERSITY OF NAIROBI
DEPARTMENT OF SOCIOLOGY
CSO 302: QUALITATIVE RESEARCH METHODS
SEMESTER: January-April 2016 [Regular and Module II]


CAT QUESTIONS [Released: 3:15 p.m., March 8th 2016]


You have four questions divided equally into Sections I and II. Answer only one question from each section, for a total of two.
Click Here to Access the CAT Questions

DEADLINES:  
1. Regular + Module II Day Class: Friday, April 1st, 2016, strictly by 11:00 a.m.

2. Module II Evening Class: Saturday, April 2nd, 2016, strictly by 11:00 a.m.

Sunday, October 18, 2015

Replicability: A Key Standard of Scientific Method Now Found to be Under Existential Threat

Nine key standards, or principles, of scientific research can be gleaned from texts on research method. As I see them, these are:
2. Replicability
3. Control
4. Measurability
5. Validity
6. Reliability (Pending)
7. Objectivity (Pending)
8. Ethics
9. Representativeness.

I will say something about each of the nine standards in later posts. What I want to briefly talk about today is replicability. It turns out that, highly valued as it is, replicability is easier 'said' in conversations about methodology -- defined as the science or theory of method -- than 'done' in actual empirical research. This is what we pick from a study captured in Jessica Firger's account, the link to which I give in the References section below.

First, though, let us recapitulate the accepted meaning and procedures of replicability. Then, second, we will turn to the new study to see how its findings shake the foundations of the scientist's faith (now 'blind faith'?) in the utility and robustness of replicabiliy as a verification process.

First, then, about the centrality of replication to scientific method and to science's "theory of evidence": As a concept of methodology, replicability calls for precision in measurement and accuracy of procedure in all scientific endeavors. That is to say, it calls for the doing of any research project in such a way as to ensure its step-by-step, beginning-to-end, repeatability or 'reproducibility' -- which is what replicability means -- in any subsequent study, by any competent researcher, intended to attest to the veracity of its discoveries or claims. To repeat, it should be repeatable in its exact original form, and should yield the very same results. Thus:
“The research should be repeatable. Any other competent researcher should be able to take your problem and , collecting data under the same circumstances and within the identical parameters as you have observed, achieve results comparable (sic) to those you have been able to secure” (Leedy, 1980: 46; see also Seale, 2004). 
That emphasis on "the very same results", as opposed to 'comparable' results, highlights the ultimate basis for the confirmation or validation of scientific claims. However, the confounding truth, which subverts such a neatly articulated metric, is that several important factors may be suggested as underpinning the likelihood of not achieving such results in the real world which social science, including/particularly sociology, routinely deals with. This is the real world of conscious, querying, skeptical (and often over-researched) human beings unsure of the risks, or real value, of availing themselves and telling everything that researchers wish to be told. And that's even before one factors in variability, for various reasons, in the ways respected scientists implement even widely agreed-upon procedures -- or the procedures that should, in the name of reason and perhaps common sense, be so agreed -- for data capture, processing, analysis and synthesis.

But why all this fuss about replicability or reproducibility to which I believe all seasoned interrogators of methodology have been paying attention for years? Firger, I think, captures the underlying sentiment as well as anyone can or has:
"Most rational humans hold some faith in science. We trust that scientists, through their hard-won expertise, are well equipped to conduct studies that provide proof of why things are the way they are, help solve problems and explain mysteries. Much of that trust is built on an implicit belief that research findings are concrete truths—in other words, that if given the same set of parameters, they would be easy to reproduce. This type of replication is essential to science because it validates key discoveries and helps scientists make progress in their fields of research" (Firger, 2015).
So replicability is a standard worth respecting, implementing and defending. But what if it turns out that in practice we cannot, or have not been doing so? What are we to make of our scientific truths, then? That, at least implicitly, is for me the most fundamental, if implicit, question driving the University of Virginia study, as we will now see.

Second, then: what was the purpose of the new study and what does it tell us? We can discern that purpose and related method from this portion of the study's problem statement, as seen in the abstract:
"Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available"  (Open Science Collaboration, 2015)
What about the results? The key finding of the study was that the results of less than 50% of the studies subjected to replication were corroborated; that is, independently validated. That is a huge shortfall, as elaborated below:
"Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects" (Open Source Collaboration, 2015; see also PSA, September 2015).
Firger (2015) notes problem in this regard: that the threat to replicability may be exacerbated by the existential pressure, perceived by scientists seeking recognition and success (presumably via promotion and tenure, or higher levels of funding, or 'all of the above'), to dump rigor, and all its 'yokes' and promise, in favor of building the capacity to "weave a memorable story out of tenuous science" -- and so to achieve such success.

In the end, perfect replicability, certainly of studies involving conscious human subjects, may not be feasible, even when driven purely by the dictates of scientific rigor, ahead of the capacity for time-travel back to the chronological 'moment' at which the original study of interest was conducted. Incidentally, contrary to Drummond's (2009) claim, this is not to say that 'reproducibility' is a more viable posture for the scientist. The attributes he assigns to reproducibility are in fact, precisely, those of replicability, as widely understood by those who routinely use the latter term (see, for example, the clarification offered by the Replicability Research Group, 2015). What Drummond wrongly sees as the weaknesses of replicability are its ideals and strengths, and what he touts as the strengths of reproducibility are its weaknesses. Reproducibility, defined as he so facilely does, would be a recipe for widespread intellectual fraud.

So what is one to do in the meantime? We will just have to make do with such rigors and caveats as we are able to muster and agree upon. But the rationale for replicating scientific work remains as great and as persuasive as it ever was, and its utility for science and human progress just as enormous. All that we have discovered is its disappointingly low success score, certainly in psychology -- which is a human science. We have also found, alas, that the pressure for recognition and success makes the scientist as much an opportunist as the proverbial self-sacrificing servant of truth. We can work with sharp focus on these personal/human 'frailties' of the scientist -- not necessarily of science -- to raise replicability's score. Replication is (natural) science's ultimate audit tool, which we will abandon only at society's and civilization's own great peril.

Still, we will also have to more consciously and more robustly develop other methods of arriving at the truth -- and the natural scientist, in particular, will have to respect with greater humility and sharper awareness of that potentially debilitating soft underbelly, just discovered, of his/her cherished branch of science. The other methods I have in mind are those which social science and the humanities have been grappling with, self-critically (and against the backdrop of harsh and even dismissive criticism by often sanctimonious natural scientists), for decades, and even more than a century, now. Many of these fall under the qualitative label, as opposed to the quantitative, and include versions of such broad and cross-cutting, timeless and a-disciplinary methods as: observation (naturalistic, participant or device-assisted), comparison, deduction, induction, abduction, focus-group conversations, dialectics, hermeneutics, and, particularly a la Foucault (1973: x-xxiv), genealogy and even archaeology.

REFERENCES:

Drummond, Chris (2009) "Replicability is not Reproducibility: Nor is it Good Science." Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, Montreal, Canada

Firger, Jessica (August 28, 2015) "Science's Reproducibility Problem: 100 Psych Studies Were Tested and Only Half Held Up"  Newsweek

Foucault, Michel (1973) The Order of Things: An Archaeology of Human Sciences. New York: Vintage Books.

Leedy, Paul D. (1980) Practical Research: Planning and Design. Second Edition. New York:  Macmillan Publishing Co.,

Open Science Collaboration (2015) "Estimating the Reproducibility of Psychological Science"  Reproducibility Project: Psychology. University of Virginia

PSA - Psychological Science Agenda (September 2015) "Science Paper Shows Low Replicability of Psychology Studies" APA

Replicability Research Group (2015) "Replicability vs Reproducibility." Tel Aviv University, Department of Statistics and Operations Research

Seale, Clive (2004) "Replication/Replicability in Qualitative Research" in The SAGE Encyclopedia of of Social Science Research Methods (Michael S. ewis-Beck, Alan Bryman and Tim Futing Liao, Eds.)

van Rijn, Hedderik and Sabine Scholz (2014) "Replication of Experiment 3 of 'Tracing Attention and the Activation Flow in Spoken Word Planning Using Eye Movements' by A Roelofs (2008,


POSTSCRIPT:

All that I have said above is based on the assumption, not necessarily correct, that:
1. The original study to be subjected to replication provided all the details necessary to do so.
2. Each scientist involved in the replication exercise is/was fully competent to do so, and that the replicating work is itself fully open to replication.

It is also important to observe that the scientist seeking to do replication work may have to come to terms with three potential failure nodes:
1. The failure to replicate a study whose methodological procedures and related tools were not detailed enough to permit faithful or full retracing.
2. The failure to replicate because the researcher did not quite have the requisite competency or capacity -- intellectual, resource, temporal -- to replicate.
3. The failure to replicate because the original work is/was of such a type (partially or totally qualitative, for example) that it is/was not fully, adequately or in any other meaningful way replicable.


END NOTE: Paper updated October 19-20, 2015

Monday, March 19, 2012

Introductory Remarks at the Start of a UNES/UoN-Sponsored Proposal Writing Workshop


BY Prof. Mauri Yambo (Workshop Coordinator), March 19, 2012


1. INTRODUCTION

May I take the opportunity, as the workshop coordinator, to once again welcome you to this five-day workshop; and to thank you for deciding and finding the time to come. It was a good decision, and it is an honour to have you.


2. OVERALL PURPOSE OF THE WORKSHOP

The overall purpose of this workshop, the eighth in a series of proposal writing workshops sponsored by UNES and the office of the DVC (RPE), is to significantly enhance the quality of research and project proposals written by academic and senior technical staff of the University of Nairobi.

It is expected that enhanced proposal writing capability will lead to:

a. A significant increase in the number of research/project grants, and related funding, won worldwide by University of Nairobi staff. As you may know, the university, as a corporate entity, expects a 15% share of the gross annual revenue from these grants;

b. The publication of more scholarly works: refereed journal articles, book chapters and books;

c. Increased participation by academic staff in scholarly and professional workshops and conferences, and an increased and growing number of evidence-based presentations at such gatherings.

A total of 500 staff are to be trained in these workshops. Out of these, it is expected that at least 10% will become proficient/successful proposal writers.


3. DIMENSIONS OF QUALITY

Quality has two broad dimensions: Form and Content.:

a. The Form (Design, Outline, Format or Structure) may be good, but the Content (such as Problem Statement, Gap Identification, Causality Statement, Logic, Premises) bad – that is, weak or incoherent or without a clear or persuasive or any value-proposition, or generally sub-par;

b. Or the Content may be good, but the Form bad (incoherent, disjointed, incomplete or idiosyncratic);

c. Or, as is too often the case, both the Form and the Content may be below the expectations of:

(i) The Funding Agency: Funding agencies tend to place a high premium on the coherence of argumentation and presentation; and, just as important, the promise of rigour: rigorous problem-solving backed by an evidence-based, logical and imaginative turn of mind; or,

(ii) The self-critical proposer: who, it is to be imagined, is always looking to reach higher levels of quality – in terms of conception and execution – but has generally lacked the opportunity and forum, such as are provided in this series of workshops, to openly and ‘safely’ speak heart-to-heart and share experiences and concerns with colleagues and peers across disciplines.


4. FACILITATORS

There are four facilitators in this workshop, as you have heard in the introductions. They are your colleagues. And though there is no denying that the university is endeavouring to realize workshop goals, as above stated, through them, they are your peers, not your teachers The flow of learning will be in both directions – and even in multiple directions. Believe me, I have already seen it in previous workshops.

All in all, the facilitators, as a group, have a big challenge, rooted in a clear set of deliverables – short-term and medium-term.

There is a certain degree of common ground which we, the facilitators, will endeavour to share with you. We have noted that this common ground has expanded in the course of the seven previous workshops, as we have responded to participants’ feedback, reviewed the programme and engaged in self-criticism. Beyond the common ground, however, we urge you to view the content of our presentations as if they emanated from four separate “books” – which it is up to you to digest: accept or disregard or query or synthesize as your critical faculties and/or research needs may direct you to do


5. WORKSHOP STRUCTURE

The workshop has three broad components:

(a) Presentations and discussions led by different facilitators, as indicated in the workshop programme already in participants’ folders;

(b) Group Work involving the drafting of research or project proposals;

(c) Presentation, on the last workshop day, of proposals developed by different groups.


6. CONCERNING GROUP WORK AND CALLS FOR PROPOSALS

As already noted, Group Work will revolve around the preparation of jointly written proposals, and the eventual presentation of at least the first draft to workshop participants. Participants may choose to respond to the call for a research or project proposal. In view of the constraint imposed on proposal writing by the workshop’s short duration and the ad hoc nature of the groups, the presentations will be very cautiously evaluated on the basis of score sheets prepared, just in the last two weeks, by the facilitators. But we urge you to take the format of the score sheets, and the weighting of the various items, very seriously as you carry on your task.

Five “Calls for Proposals” will be available to participants. Each group will be at liberty to choose any of the five calls for the purposes of Friday's presentations. Four of the calls are hypothetical, while one is actual and current, as follows:

(a) The Ogot Foundation: Call for project proposals on Hunger in the Horn of Africa, focusing on any of hunger’s myriad causes in any specified “local area” within the region. Value of the hypothetical call: Kshs 10 million;

(b) Capital Markets Authority of Kenya (CMA): Call for applied finance research proposals covering three broad areas of interest within Kenya’s asset management community: applied finance research, asset management knowledge and related issues, and knowledge dissemination. Value of the hypothetical call: US$15,000;

(c) Youth Advisory Board: Call for project proposals on Youth Community Service Learning Projects focusing on any of the following: environmental responsibility, disaster preparedness, driver safety, financial education, and bridging the achievement gap through higher education. Value of the hypothetical call: US$150,000.

(d) Domestic Violence Research: Call for research proposals designed, inter alia, to enhance community response capacity through direct services and support to victims and their families; and to strengthen governmental, legal and cultural interventions. Value of the hypothetical call: Kshs 5 million.

(e) Swiss Re’s International Resource Award for Sustainable Watershed Management 2013: Call for proposals for the implementation of sustainable watershed management projects. A proven track record in sustainable planning and/or management of water or other resources will be particularly advantageous. This is an actual annual award, which may be granted to one or several projects. Its value for the year 2013 is US$ 150,000. Deadline for the return of initial application forms: 30 April 2012.


7. PROPOSAL FORMATS AND LENGTHS

The Score Sheets alluded to above incorporate proposal formats which you should, at the very least, bounce off your own groups’ consensus-ideas of format, as you draft your full proposals. The respective facilitators will share with you their more detailed conceptions of format. Each workshop day should therefore be both interesting and full of perspectives to interrogate.

The maximum length of a full proposal is typically “not more than 15 pages” There will be at least two sources in your folders which point to the fifteen page limit; and that is before line spacing and font size are specified. At the National Science Foundation (NSF) of the USA, line spacing (whether double or single) is left to the discretion of the proposer, but maximum page limit is set at 15. The Center for Nonprofit Management also sets the maximum page limit at 15. The sample proposal prepared by Prof. Madara Ogot (former MD of UNES), which appears as Handout 2 in your folder, is already some 8 pages long, single-spaced. This is before we include certain essential components of a full proposal.

While it will be up to you to decide on the lengths of your proposals, of course within the above limits, I wish to inform you that each group will be given only ten minutes to present its proposal on Friday, beginning at noon.


8. FINAL WORD

I believe that you will find ways to make your respective teams work – the teams that you will join presently. That is, work closely enough to produce a draft, in five intense days.

Let me now give way to the first facilitator, Dr. Machoki, whose day Monday is, as the programme shows you. My day is tomorrow. Dr. Marete will facilitate on Wednesday. Mr. Matseshe will take you through all matters financial and budgetary on Thursday. He will be back Friday morning with Dr. Machoki, to round off their parts. All the four of us will be with you from the start of group presentations at noon on Friday until the end of the workshop at 4. Thank you.