Therapeutic care is best when research guides and supports the methods used and the care is delivered by caring people who inspire trust. However we see many improper references to research in the promotional materials of many schools and programs. We want readers to see this page as an extension of our Warnings page. Families planning for stays in residential schools and programs should read both this page and the Warnings page carefully, as both direct attention to traps easy to fall into, if not guided by a truly experienced consultant.
The great value of research guidance notwithstanding, we advise anyone who approaches decision making on where to go for treatment be guided by research claims only if they heed the warnings on this page.
The terms "Best Practice" and "Evidence Based" are getting increased attention in recent years. In general they apply to standards of practice that are researched based, although some technicalities emerge in discussion of these terms and those technicalities matter. "Evidence Based Practices in Psychology (EBPP)" applies to practice that is based upon hard research to produce certain results, given a specific set of circumstances. Other terminology that also refers to research informed care in some form has been around a very long time. Empirically Supported Treatments (ESTs) is another term that sometimes appears. Or we might hear broader but less precise terms like "research validated," or "data supported," or "research supported."
The increasing use of research to guide treatment is a very welcome trend, although research claims are increasingly being used deceptively. What follows is intended to help families understand what to look for when encountering research claims or possible absence of research to support a potential providers methods.
-
-
- EBPP defined
- Lack of Commitment to Research Validated Treatment
- Claims without documentation
- Misapplication of Research
- Research Methods Favor Behavioristic Procedures Leading to Misleading Conclusions
- When not to use procedures that have the strongest research backing
- Checklist for Families Seeking Services
- Checklist for schools and programs
-
Technical Definition of Evidence-Based Practice in Psychology (EBPP)
Arguably, the article "Evidence-Based Practice in Psychology" from the APA Presidential Task Force on Evidence-Based Practice in the May-June 2006 issue of The American Psychologist is the definitive statement on the appropriate use of the term "Evidence Based" in psychology. On page 217 halfway down the page on the left column you will see the heading "Definition." This introduces some of the technical aspects of this. In that section in the second column you will see the term Empirically Based Treatments (EBT)" being introduced. We do not hear that term as frequently. We refer you to the article itself to see the difference. Both are research based. "Best Practice" refers to a practice that is determined by some authority that research indicates the designated practice is the practice most likely to produce the desired results, given a specific set of circumstances.
Lack of Commitment to Research Validated Treatment
Our first concern is that too few schools and programs are paying attention to research, evidence, and best practices.
We strongly urge that schools and programs apply the best of research validated procedures to the maximum extent practical. When we say "practical" we do not mean that Evidence Based methods and Best Practices should become strait jackets. There is still room for creativity. Remember, that which is now strongly researched backed was attempted by someone before there was research backing. We do want all schools and programs to take research seriously and incorporate it into their methodology. We want all schools and programs to be aware of Best Practices and Evidenced Based practices that may apply to the people they serve. We want those practices to be applied to those of their clientele who, according to research, should benefit from those practices. This does not mean that they should be applied to everyone in a "one size fits all" manner.
However no program can base every action or choice of procedure on what is research backed. We want to know that schools and programs are aware of research that might apply to what they do, and can readily demonstrate that what they do is appropriately informed by research and that expansion beyond what research tells us is based upon credible and transparent reasoning.
We realize that some programs are working in cutting edge areas where best practices have not been established and the research is quite limited or non-existent. Some programs are working with innovations that appear to have potential but have not yet been adequately researched. We encourage that kind of innovation, but want programs to be transparent about what is research based, what is innovation, and what is simply the specific personality or twist of the program at issue. What is known from research must never be ignored. It should also never stifle creativity.
If you are considering clinical care for a mental health issue, it is important to inquire the degree to which research informs the care given by that provider of service.
Claims Without Documentation
Until recently (this being cited on June 9, 2017) we would have said that most schools and program that are making claims about procedures being research or evidence based or about their practices being best practices are not providing us with documentation of those claims when we inquire. In some cases where they decline, we suspect the claims have some basis in fact after all, but we see no excuse for any school or program making such claims without being prepared with a written documentation of the specific basis for those claims. While we now more frequently get documentation upon request, absence of documentation still occurs more frequently than we would like. When a procedure is labeled "Best Practice," that clearly depends upon the judgment of some authority. That may be true of an "Evidence Based" claim, or it may be that provider's interpretation of the definition of "Evidence Based" (which is still the claim of one organization).
When we see "research based" or "evidence based" or "best practice" claims we want to see a written statement documenting the claims including information on how to access the original published reports of the research at issue. In particular, if the claim pertains to "best practices" we want the written statement to include identification of the organization that has determined that the practice at issue is a best practice. When research is cited, we usually expect providers to let us know whether research is published in juried journals, and the degree to which the research is independent or vs. open to question of conflict of interest. The statement should designate the specific conditions which were studied in the research involved (including the population that was studied), the results of the procedure that were documented, and the degree to which application of the procedure in the school or program corresponds to what was studied in the research and /or designation of "best practice."
When a school or program makes reference to research, "evidence based," or "best practice," and does not provide the written statement we describe in the previous paragraph, we believe misrepresentation cannot be ruled out.
Misapplication of Research
Programs are making "best practice" and/or "evidence based" or "research based" claims based upon research that does not quite match the situation in the school or program about which the claim is made.
Too often schools and programs cite Evidence Based practices and/ or Best Practices that are legitimate in certain situations, but are suggesting, falsely, that the citation covers more than than the narrow circumstances in which the citation actually applies.
As the public becomes more aware of the designations, "Evidence Based," and "Best Practices," we expect that the terms will be used in a misleading way, so the public should be aware of the real issues. For example, the "Seven Challenges" substance abuse programming (which we at FamilyLight highly respect) is an Evidence Based program. However our understanding is that the studies that lead to the EBPP designation are consistently based upon a specific population of people receiving other therapeutic intervention at the same time as they are working with the Seven Challenges. That would suggest that the "Evidence Based" designation would really only apply to a population similar to the population studied, including having that particular population receiving other mental health services at the same time.
From our discussions with Seven Challenges author and founder Dr. Robert Schwebel, we believe that he and his organization do their very best to ensure that the service delivery in all programs using the Seven Challenges name are accountable to fidelity to the model as researched. We are not attempting to raise an alarm about Seven Challenges programs, at least not on this point. We are attempting to alert readers to one kind of problem that can emerge when schools and programs promote themselves based on research claims.
Schools and programs that claim to be Evidence Based because they use the Seven Challenges only do that legitimately if they are serving a population demographically similar to the population participating in the study(ies) that are the basis for application of Seven Challenges to be an Evidence Based Procedure. We want to see the fact sheet requested above verifying how the offerings of the school or program matched the terms of the study. That means, among other things, that programs relying on Seven Challenges research to promote their programming have an obligation to disclose what other procedures are in use, so that the findings of that research really do apply.
We do not question the validity of claiming the Seven Challenges to be Evidence Based. We are only challenging the right of schools and programs adopting the Seven Challenges to claim that application of the Seven Challenges is Evidence Based in circumstances that are not consistent with circumstances actually studied in the research that supports the claim. If schools and programs advertise use of Seven Challenges as a single methodology and claim it to be an EBPP, we would understand a false claim. Please note the following from the Website of the Seven Challenges Organization website (accessed June 8, 2017; Scroll down the linked page to find the information below citing research backing):
Research [regarding effectiveness of the Seven Challenges]
Independent studies funded by The Center for Substance Abuse Treatment in Washington, D.C. and published in peer reviewed journals -- one study at the University of Iowa and the other at the University of Arizona -- have provided evidence that demonstrates the effectiveness of The Seven Challenges as a "co-occurring" program that significantly decreases the substance use of adolescents and greatly improves their overall mental health status. Data also show that the program has been especially effective with the large number of substance abusing youth who have trauma issues.
The published materials for counselors and young people, combined with supporting documents, effective training, ongoing support and monitoring from The Seven Challenges team, have proven to be effective in replicating the successful outcomes noted in the research. The SAMHSA National Registry of Evidence-Based Programs and Practices, (NREPP) gave Seven Challenges a perfect score for "Readiness for Dissemination."
For more detail, please see the response to our inquiry from the Seven Challenges Organization.
This problem is not just about the Seven Challenges. Families making decisions for themselves, who are attracted by research claims, need to pin down these schools and programs on exactly what the research actually supports and determine whether the research is relevant to that family's needs.
Research Methods Favor Behavioristic Procedures Sometimes Leading to Misleading Conclusions
Procedures that are based upon Behaviorism are more easily researched than methods that are not. In reality, research does not support application of procedures based upon behaviorism in preference to other methods unless fairly constructed research has compared long term outcomes between behavioristic and non behavioristic methods. The reason why behavioristic approaches are more likely to gain research support is simple. Behavioristic methods are based on defining a desired behavior and using immediate rewards (and sometimes punishments for not producing the desired behavior). Basic clinical records show whether the intended goal was met. We are oversimplifying just a little when we say that the researcher only needs to count up the number of successes and the number of failures, then compile and analyze the data. If the procedure is not behavioristic, the researcher must in most cases introduce criteria based on observation to determine if a successful outcome is achieved. It is at best an extra step, and may cloud the issues being researched.
Research on behavioral programs and methods speaks for itself, just as research on any other kind of program or method. But sometimes fallacious inferences are drawn from actual positive research outcomes on behavioral methods leading to claims that this research demonstrates a preference for behavioral programming over other methods. But without comparison studies, this inference is not warranted.
Behavioral methods in treatment are based upon rewarding desired behavior and, in some cases, punishing undesired behavior. Because of this, observing behavior, in some cases recording it quite objectively, is intrinsic to the function of a behavioral approach to treatment. More formalized methods of behavioral treatment require frequent notations of behavioral observations. Turning this data over to a researcher makes research easy. Regardless of the presenting issues, we know that rewards and/or punishments focused on desired/ undesired behaviors tend to bring short term results. By "short term" we mean both that the results appear quickly and that they might be temporary.
By contrast, research on non-behavioral treatment methods requires identification of specific behavioral criteria to indicate the outcome being studied, strictly to accommodate the research, although these behavioral criteria probably have nothing to do with the treatment methods themselves. They are simply added by the researcher so that an objective decision procedure is in place as needed to provide a research result.
Behavioristic programs, using their over-simplified showing short-term gains, will use this kind of thing to claim they are more effective than the non-behavioristic programs. But this is a fallacy two ways: What happens very short term in a the midst of a reward-punishment environment, says virtually nothing about long term gains, and in any case does not give evidence that one approach is better than another unless both are being studied and the results compared.
A good example of this is the rivalry between "Applied Behavioral Analysis" (ABA) and "Floortime" as methods of treating autistic children. ABA is almost purely behavioristic; Floortime involves a trained adult spending time "on the floor" playing with the child being treated, both building a relationship and opening communication channels.
ABA advocates speak of the research behind their method, implying that it is more effective than Floortime. However the research we are aware of supporting ABA is strictly the kind we described above. It is short term analysis of gains in a structured behavioral setting. We know of no research appropriately directly comparing long term outcomes between ABA and Floortime. Until we have that, we simply do not know which is more effective. We suspect the best choice is eclectic. But programs purporting to use ABA because research demonstrates its superiority over other methods either do not understand their research studies or are deliberately disingenuous.
When not to use procedures that have the strongest research backing. When a particular form of treatment is supported as the most effective for some particular diagnosis, it usually reveals a percentage of those who achieve the desired result from the treatment studied. If the percentage is very high or if it is better with treatment under consideration than with available alternatives, then it is very likely that this treatment should always (or very nearly always) the treatment to apply first. However we have yet to see a study which shows 100% success for any given procedure. That suggests that the procedure is not 100% successful. That opens the question of what is next for the people for whom the procedure at issue did not achieve the desired result.
For example, an oncologist treating a person with cancer will start out with the treatment regimen that is most likely to be successful for a patient with the presenting circumstances of the patient at issue. But if that procedure is not effective with the patient being treated, the oncologist will move to a different treatment, choosing that treatment based in part on how the patient reacted to the treatment that did not work.
We do not consistently see a comparable flexibility in mental health services that are promoted in competition with each other. Take the example of Exposure and Response Prevention (EX/RP) as an example of rigidity in mental health that differs from what happens in other areas of medicine. For this example, let us simplify the actual facts of the research. Let us assume that the incontrovertible research result is that this form of therapy is the gold standard treating Obsessive Compulsive Disorder (OCD). Actually some of the research we see and what we hear from some clinicians would tell us that our assumption is the fact of the matter; others suggest some controversy. Our example works if we go with the proposed assumption just for the purposes of this discussion.
When research demonstrates that a specific procedure is the best choice of treatment given a defined set of circumstances, that does not imply that it is is 100% effective. At this writing, we are in discussion with a prospective client family. This is about a young woman in her mid-twenties whose life is been severely disrupted by OCD. She has experienced EX/RP treatment a number of different times including a number of stays in residential treatment. Each time she has maintained significant progress, then returned to her presenting behaviors when discharged from treatment. As the family has approached us, they reported that all clinicians they had consulted with any experience with OCD were insisting upon returning to another round of EX/RP treatment.
What is often cited as the best definition of insanity? Repeating a repetitive past behavior and expecting a different result? A psychologist whose primary expertise is outpatient psychodynamic therapy familiar with the situation, told us that he recommended that she go into psychodynamic therapy to break the cycle. He did not challenge the value of EX/RP in situations like this, but proposed that psychodynamic therapy could prepare her to benefit from EX/RP, although providing that therapy was not an area of his own competence. In our minds that also raised the possibility of finding a therapist or a treatment team that could do both effectively. Getting that done was not an easy task as clinicians or established teams who do that are simply not routinely available, and insurance companies do not like to pay for psychodynamic therapy in any case.
Mental Health clinicians with different skill sets tend to think of themselves as rivals arguing "either or" rather than "both and." People who are skilled in one therapeutic approach tend to be at odds with people using a different method. We see much too little of clinicians in this situation collaborating in the interests of the people they serve. Too often they insist on having clients whose outcomes have not been satisfactory redo the same treatment, rather than considering the possibility that this client is one of the rare people who will not do well with what research tells us is usually the best approach.
This situation is at its worst in treatments for substance abuse and addiction. Sadly, it is almost impossible to find a program that supports a range of methods that can be accessed after person's needs have been properly assessed.
Our conclusion is that schools and programs need to say less and do more where research and "Best Practice" designations are concerned. We see far too little of schools and programs being genuinely and constructive guided by research and adoption of best practices. We see far too much of clearly fraudulent marketing making spurious claims about their adherence to what has been learned from research.
Checklist for Families Seeking Services:
- Inquire of any therapeutic school or program you are considering the degree to which their procedures are research validated. If they cannot cite research validation, you might want to look elsewhere.
- If they do cite research validation, inquire if that research supports a claim of "Evidence Based" or "Best Practice," ask the basis for that. In any case ask for references to the research they base those claims on.
- If they do not cite any research to support their methods, inquire why not.
- Ask whether the their family's situation corresponds to the population on which the research at issue is was based (if the answer is "no" then that research is not relevant)
- Remember that if research says x procedure is successful 70% of the time, and for that reason it is the most successful in effectively treating the problem at issue you need to think about the remaining 30%. We agree that based on that research, it is important in most cases to go to that method first. However if research shows 70% effectiveness it is likely that 30% are likely to need a different method. You need to inquire what the the school or program does when their preferred method does not work.
- Assess the research claims according to the above. If you are not willing to do that, do not allow research claims to affect your decision.
Guidelines Checklist for schools and programs:
- Schools and programs should examine Evidence Based practices and Best Practices as they develop and refine their methods and communicate a clear rationale for any methods that differ from known Evidence Based practice and Best Practices.
- Schools and programs which make promotional claims about their methods being research based or evidence based or a best practice should routinely provide a fact sheet giving specific references to the research on which those claims are based. Those references need to include enough information to permit the reader to know how to access the original research on the Internet or in a well equipped library. In the case of a "Best Practice" claim, the information sheet should also include identification of the agency or organization giving the best practice designation. In the case of an Evidence Based procedure or practice, the basis for that claim should be clearly communicated.
- The fact sheet described in the above bullet should be specific enough that reader can understand the specific procedure being researched, the specific presenting conditions that the research indicates the procedure is effective in addressing (including but not limited to specific demographics of the people used as research subjects), and the specific outcome the research indicates that the procedure yields.
- Research will not be presented as advocating one procedure to be preferable to another unless the research was comparison study between the two procedures.
- Schools and programs will not use terms like "research," "evidence," or "best practice" in any manner that would reasonably lead to misunderstanding and or false understanding of what the research at issue actually demonstrated and how that relates to the school or program being promoted.
Please read our full disclaimer. You are responsible for verifying our information before acting on it.
Go to Seven Challenges
Last update April 19, 2018