Up Your Game: How to Identify the Most Convincing Research

As you may know, I have had a lot of success in winning large federal government grants. In retrospect, I think that my background as a political scientist and as a published author gave me unusual advantages with federal government grant applications because so many of them are based on evidence derived from previous government studies. 

Since I had experience doing my own research and creating literature reviews as a political scientist, it was relatively easy for me to pick through federal websites and identify prospective interventions, programs, and practices that would most likely make a difference for the clients of the charities that hired me to work as a grant writer. 

As such, I apply certain internal standards to help me determine whether or not a particular study will present strong evidence in support of the program that I am trying to get funded. I will share some of these standards that I look for in the section below:  

1. Peer-Reviewed Articles: In general, I know to look for articles printed in peer-reviewed journals. This is just a fancy way of saying that an article in the journal will not be published unless a jury of independent researchers determined that it met their field's standards for academic research. In most fields there are the prestigious and the less prestigious journals. I look for ones that have been around a long time and for individual articles in those journals that have been cited by other researchers.  
2. Random Assignment: Next, I am particularly interested in finding studies which use random assignment to establish test and control groups to test whether or not a particular program or intervention works or not. A test group is simply the group of people - or animals, I suppose - who will be treated differently by the experiment. The control group, on the other hand, is supposed to be a group of very similar people who will not be exposed to the treatment variable. What is crucial is that the assignment to either the test or the control group must be random. This is a fundamental assumption behind the statistical methods used to determine whether or not there is a significant difference between the test and control group after the experiment is done. 
3. Quasi-Experimental Studies: Sometimes, it is not possible to simply assign people at random. In these cases, we might create test and control groups by recruiting volunteers or testing the same people before and after they were exposed to the treatment. These studies are called quasi-experimental studies. 
4. Population Similarities: The strongest studies are experimental and quasi-experimental studies that are based on populations similar to the one you are looking to serve. For example, you should not justify an experiment on college students, based on a study that looked at elementary school students. 
5. Multi-Site Studies: You will also be better off in picking larger studies that included multiple sites, meaning that the research was conducted at a number of different college campuses or in different cities around the nation. The problem with single site studies is that the single site might be contaminated statistically with unique people who respond to the intervention in the same way simply because they already have so much in common.
6. Attrition Rates: You need to pay attention to the little details in the study like how many of the participants dropped out of the study before it was done. This is because the people who dropped out will influence the final results of the study. 

Finally, you should be aware of the possibility for "experimenter bias" which is the tendency of the researcher to confirm their own suspicions in their own study. Sometimes this happens on purpose when the researchers fudge their results or manipulate their data to match their own conclusions. Other times there is an error in the study itself, but the research fails to notice this error because the overall study supports their conclusions. Since the results break their way, the researcher sees no need to double (or triple) check their own results. To guard against experimenter bias, I like to find studies where the researcher(s) appear to be sincerely surprised by their own results. To me, this is at least one indication that they are being honest with themselves and their readers.