This page currently being developed
The terms "Best Practice" and "Evidence Based" are getting increased attention in recent years. In general they apply to standards of practice that are researched based, although some technicalities emerge in discussion of these terms and those technicalities matter. "Evidence Based Practices in Psychology (EBPP)" applies to practice that is based upon hard research to produce certain results, given a specific set of circumstances. These are the research terms getting much attention these days. Other terminology that also refers to research informed care in some form has been around a very long time. Empirically Supported Treatments (ESTs) is another term that sometimes appears. Or we might hear broader but less precise terms like "research validated," or "data supported," or "research supported."
The increasing use of research to guide treatment is a very welcome trend, although research claims are increasingly being used deceptively. What follows is intended to help families understand what to look for when encountering research claims or possible absence of research to support a potential providers methods.
- EBPP defined
- Lack of Commitment to Research Validated Treatment
- Claims without documentation
- Misapplication Research
- Research Methods Favor Behavioristic Procedures Leading to Misleading Conclusions
- Guidelines Checklist for Families Seeking Services
- Guidelines Checklist for schools and programs
Technical Definition of Evidence-Based Practice in Psychology (EBPP)
Arguably, the article "Evidence-Based Practice in Psychology" from the APA Presidential Task Force on Evidence-Based Practice in the May-June 2006 issue of The American Psychologist is the definitive statement on the appropriate use of the term "Evidence Based" in psychology. On page 217 halfway down the page on the left column you will see the heading "Definition." This introduces some of the technical aspects of this. In that section in the second column you will see the term Empirically Based Treatments (EBT)" being introduced. We do not hear that term as frequently. We refer you to the article itself to see the difference. Both are research based. "Best Practice" refers to a practice that is determined by some authority that research indicates the designated practice is the practice most likely to produce the desired results, given a specific set of circumstances.
Lack of Commitment to Research Validated Treatment
The first is that too few schools and programs are paying attention to research, evidence, and best practices.
We strongly urge that schools and programs apply the best of research validated procedures to the maximum extent practical. When we say "practical" we do not mean that Evidence Based methods and Best Practices should become strait jackets. There is still room for creativity. Remember, that which is now strongly researched backed was attempted by someone before there was research backing. We do want all schools and programs to take research seriously and incorporate it into their methodology. We want all schools and programs to be aware of Best Practices and Evidenced Based practices that may apply to the people they serve. We want those practices to be applied to those of their clientele who, according to research, should benefit from those practices. This does not mean that they should be applied to everyone in a "one size fits all" manner.
However no program can base every action or choice of procedure on what is research backed. We want to know that schools and programs are aware of research that might apply to what they do, and can readily demonstrate that what they do is appropriately informed by research and that expansion beyond what research tells us is based upon credible and transparent reasoning.
We realize that some programs are working in cutting edge areas where best practices have not been established and the research is quite limited or non-existent. Some programs are working with innovations that appear to have potential but have not yet been adequately researched. We encourage that kind of innovation, but want programs to be transparent about what is research based, what is innovation, and what is simply the specific personality or twist of the program at issue. What is known from research must never be ignored. It should also never stifle creativity.
Claims Without Documentation
Until recently (this being cited on June 9, 2017) we would have said that most schools and program that are making claims about procedures being research or evidence based or about their practices being best practices are not providing us with documentation of those claims when we inquire. In some cases where they decline, we suspect the claims have some basis in fact after all, but we see no excuse for any school or program making such claims without being prepared with a written documentation of the specific basis for those claims. While we now more frequently get documentation upon request, absence of documentation still occurs more frequently than we would like. When a procedure is labeled "Best Practice," that clearly depends upon the judgment of some authority. That may be true of an "Evidence Based" claim, or it may be that provider's interpretation of the definition of "Evidence Based" (which is still the claim of one organization).
When we see "research based" or "evidence based" or "best practice" claims we want to see a written statement documenting the claims including information on how to access the original published reports of the research at issue. In particular, if the claim pertains to "best practices" we want the written statement to include identification of the organization that has determined that the practice at issue is a best practice. When research is cited, we usually expect providers to let us know whether research is published in juried journals, and the degree to which the research is independent or vs. open to question of conflict of interest. The statement should designate the specific conditions which were studied in the research involved (including the population that was studied), the results of the procedure that were documented, and the degree to which application of the procedure in the school or program corresponds to what was studied in the research and /or designation of "best practice."
When a school or program makes reference to research, "evidence based," or "best practice," and does not provide the written statement we describe in the previous paragraph, we believe misrepresentation cannot be ruled out.
Misapplication of Research
Programs are making "best practice" and/or "evidence based" or "research based" claims based upon research that does not quite match the situation in the school or program about which the claim is made.
Too often schools and programs cite Evidence Based practices and/ or Best Practices that are legitimate in certain situations, but are suggesting, falsely, that the citation covers more than than the narrow circumstances in which the citation actually applies.
As the public becomes more aware of the designations, "Evidence Based," and "Best Practices," we expect that the terms will be used in a misleading way, so the public should be aware of the real issues. For example, the "Seven Challenges" substance abuse programming (which we at FamilyLight highly respect) is an Evidence Based program. However our understanding is that the studies that lead to the EBPP designation are consistently based upon a specific population of people receiving other therapeutic intervention at the same time as they are working with the Seven Challenges.That would suggest that the "Evidence Based" designation would really only apply to a population similar to the population studied, including having that particular population receiving other mental health services at the same time.
Schools and programs that claim to be Evidence Based because they use the Seven Challenges only do that legitimately if they are serving a population demographically similar to the population participating in the study(ies) that are the basis for application of Seven Challenges to be an Evidence Based Procedure. We want to see the fact sheet requested above verifying how the offerings of the school or program matched the terms of the study. That means, among other things, that programs relying on Seven Challenges research to promote their programming have an obligation to disclose what other procedures are in use, so that the findings of that research really do apply.
We do not question the validity of claiming the Seven Challenges to be Evidence Based. We are only challenging the right of schools and programs adopting the Seven Challenges to claim that application of the Seven Challenges is Evidence Based in circumstances that are not consistent with circumstances actually studied in the research that supports the claim. When schools and programs advertise use of Seven Challenges as a single methodology and claim it to be an EBPP, we understand a false claim. Please note the following from the Website of the Seven Challenges Organization website (accessed June 8, 2017; Scroll down the linked page to find the information below citing research backing):
Research [regarding effectiveness of the Seven Challenges]
Independent studies funded by The Center for Substance Abuse Treatment in Washington, D.C. and published in peer reviewed journals -- one study at the University of Iowa and the other at the University of Arizona -- have provided evidence that demonstrates the effectiveness of The Seven Challenges as a "co-occurring" program that significantly decreases the substance use of adolescents and greatly improves their overall mental health status. Data also show that the program has been especially effective with the large number of substance abusing youth who have trauma issues.
The published materials for counselors and young people, combined with supporting documents, effective training, ongoing support and monitoring from The Seven Challenges team, have proven to be effective in replicating the successful outcomes noted in the research. The SAMHSA National Registry of Evidence-Based Programs and Practices, (NREPP) gave Seven Challenges a perfect score for "Readiness for Dissemination."
For more detail, please see the response to our inquiry from the Seven Challenges Organization.
This problem is not just about the Seven Challenges. Families making decisions for themselves, who are attracted by research claims, need to pin down these schools and programs on exactly what the research actually supports and determine whether the research is relevant to that family's needs.
Research Methods Favor Behavioristic Procedures Sometimes Leading to Misleading Conclusions
Procedures that are based upon Behaviorism are more easily researched than methods that are not. In reality, research does not support application of procedures based upon behaviorism in preference to other methods unless fairly constructed research has compared long term outcomes between behavioristic and non behavioristic methods. The reason why behavioristic approaches are more likely is simple. Behavioristic methods are based on defining a desired behavior and using immediate rewards (and sometimes punishments for not producing the desired behavior). Clinical records show whether the intended goal was met. We are oversimplifying just a little when we say that the researcher only needs to count up the number of successes and the number of failures, then compile and analyze the data. If the procedure is not behavioristic, the researcher must put
Research on behavioral programs and methods speaks for itself, just as research on any other kind of program or method. But sometimes fallacious inferences are drawn from actual positive research outcomes on behavioral methods leading to claims that this research demonstrates a preference for behavioral programming over other methods. But without comparison studies, this inference is not warranted.
Behavioral methods in treatment are based upon rewarding desired behavior and, in some cases, punishing undesired behavior. Because of this, observing behavior, in some cases recording it quite objectively, is intrinsic to the function of a behavioral approach to treatment. More formalized methods of behavioral treatment require frequent notations of behavioral observations. Turning this data over to a researcher makes research easy. Regardless of the presenting issues, we know that rewards and/or punishments focused on desired/ undesired behaviors tend to bring short term results. By "short term" we mean both that the results appear quickly and that they might be temporary.
By contrast, research on non-behavioral treatment methods requires identification of specific behavioral criteria to indicate the outcome being studied, strictly to accommodate the research, although these behavioral criteria probably have nothing to do with the treatment methods themselves. They are simply added by the researcher so that an objective decision procedure is in place as needed to provide a research result.
Behavioristic programs, using their over-simplified showing short-term gains, will use this kind of thing to claim they are more effective than the non-behavioristic programs. But this is a fallacy two ways: What happens very short term in a the midst of a reward-punishment environment, says virtually nothing about long term gains, and in any case does not give evidence that one approach is better than another unless both are being studied and the results compared.
A good example of this is the rivalry between "Applied Behavioral Analysis" (ABA) and "Floortime" as methods of treating autistic children. ABA is almost purely behavioristic; Floortime involves a trained adult spending time "on the floor" playing with the child being treated, both building a relationship and opening communication channels.
ABA advocates speak of the research behind their method, implying that it is more effective than Floortime. However the research we are aware of supporting ABA is strictly the kind we described above. It is short term analysis of gains in a structured behavioral setting. We know of no research appropriately comparing long term outcomes between ABA and Floortime. Until we have that, we simply do not know which is more effective. We suspect the best choice is eclectic. But programs purporting to use ABA because research demonstrates its superiority over other methods either do not understand their research studies or are deliberately disingenuous.
Our conclusion is that schools and programs need to say less and do more where research and "Best Practice" designations are concerned. We see far too little of schools and programs being genuinely and constructive guided by research and adoption of best practices. We see far too much of clearly fraudulent marketing making spurious claims about their adherence to what has been learned from research.
Guidelines Checklist for Families Seeking Services:
- Inquire of any therapeutic school or program you are considering the degree to which their procedures are research validated. If they cannot cite research validation, you might want to look elsewhere.
- If they do cite research validation, inquire if that research supports a claim of "Evidence Based" or "Best Practice," ask the basis for that. In any case ask for references to the research they base those claims on.
- If they do not cite any research to support their methods, inquire why not.
- Ask whether the their family's situation corresponds to the population on which the research at issue is was based (if the answer is "no" then that research is not relevant)
- Assess the research claims according to the above.
More to be added here
Guidelines Checklist for schools and programs:
- Schools and programs should examine Evidence Based practices and Best Practices as they develop and refine their methods and communicate a clear rationale for any methods that differ from known Evidence Based practice and Best Practices.
- Schools and programs which make promotional claims about their methods being research based or evidence based or a best practice should routinely provide a fact sheet giving specific references to the research on which those claims are based. Those references need to include enough information to permit the reader to know how to access the original research on the Internet or in a well equipped library. In the case of a "best practice" claim, the information sheet should also include identification of the agency or organization giving the best practice designation. In the case of an Evidence Based procedure or practice the basis for that claim should be clearly communicated.
- The fact sheet described in the above bullet should be specific enough that reader can understand the specific procedure being researched, the specific presenting conditions that the research indicates the procedure is effective in addressing (including but not limited to specific demographics of the people used as research subjects), and the specific outcome the research indicates that the procedure yields.
- Research will not be presented as advocating one procedure to be preferable to another unless the research was comparison study between the two procedures.
- Schools and programs will not use terms like "research," evidence," or "best practice" in any manner that would reasonably lead to misunderstanding and or false understanding of what the research at issue actually demonstrated.
Please read our full disclaimer. You are responsible for verifying our information before acting on it.
Go to Seven Challenges
Last update June 16, 2017