[Letter on power analysis in parapsychology]
By J.E. Kennedy
(Original publication: The Journal of Parapsychology, 2003,
Volume 67, pages 406-408)
(copyright was not exlusively transferred to the Journal)
(Also available as pdf)
To The Editor:
Rex Stanford (2003) recently noted "the wisdom of performing a power analysis prior to attempting replications of a study" (p. 18). Statisticians strongly concur with this point (e.g., Utts, 1991). However, power analysis also brings into focus some of the pivotal and problematic issues in psi research.
Power analysis is used to determine the sample size needed to have a reasonable likelihood of obtaining significant results. It is particularly important for interpreting nonsignificant or marginal results. My experience working in medical research for over a decade has been that power analysis is usually expected as part of grant applications and is required in the protocols for studies submitted to FDA to support approval of new products.
Power analysis, as commonly applied in planning studies, is based on the assumption that the effects being investigated are reasonably stable across studies and reasonably independent of the investigator. The likelihood of obtaining significant results is assumed to increase as the sample size increases.
There is currently strong evidence that psi research does not have these properties. The effects in experiments vary markedly among experimenters. Therefore, any power analysis must be experimenter specific. Even more problematic, the evidence for frequent declines and changes in results for a line of research for an experimenter indicates that psi effects are not stable across studies and often seem to change capriciously (Houtkooper, 2002; Kennedy, 2003).
From a more technical perspective, power analyses, like other statistical methods, are based on the fact that the z score is expected to increase with the square root of the sample size. Similarly, the z score divided by the square root of sample size is used to measure effect size and is expected to be unrelated to sample size.
The expected association between sample size and z score or significance level was not found in meta-analyses of random number generator (RNG) studies (Radin & Nelson, 2000) and early ganzfeld studies (Honorton, 1983). In equivalent results, effect size was found to be inversely related to sample size in RNG studies (Steinkamp, Boller, & Bosch, 2002), later ganzfeld studies (Bem & Honorton, 1994) and early card experiments (Nash, 1989; discussed in Kennedy, 1994). These findings are another way of expressing the experimenter differences and declines noted above, and are also consistent with goal-oriented psi experimenter effects (Kennedy, 1994)
Much greater thought needs to be given to the application and interpretation of statistical methods under these circumstances. Trying to use power analysis to plan the sample size for confirmatory studies brings these issues to the forefront.
It appears to me that understanding these problematic properties of psi is the top priority for research. Historically in parapsychology the inconsistent effects have been thought to be due to variations in psychological factors such as attitude and motivation. That is certainly one of the more testable approaches and is a good starting point. However, there has been little effort to explore these realms in depth -- to understand what peoples' actual motivations are relating to psi and why they feel and believe that way.
For example, it appears that the sex differences in attitude toward psi and the occurrence of psi experiences is an area of interest whose time has come. In the last issue of the Journal, I discussed evidence that the extreme skeptics tend to be males who have rational, controlling personalities, and the likelihood that these personality factors are genetically based and have had adaptive value in evolution (Kennedy, 2003). In the same issue, Stanford (2003) commented that gender is likely to be a significant factor in psi research and should always be examined. Palmer and Neppe (2003) reported a study that found the overall association between psi experiences and temporal lobe dysfunction was confounded by greater reports of experiences and symptoms by females. It also can be noted that Watt and Ramakers (2003) reported a study that recruited favorable and skeptical experimenters, which resulted in 6 of the 9 believers being female and 3 of the 5 skeptics being male.
These sex and personality differences raise the likelihood that attitude toward psi is associated with genetically based personality factors. If psi effects are related to attitude and motivation, understanding these deep-seated motivations would seem to be crucial for obtaining and interpreting psi, as well as for understanding the opposition to psi.
Research on attitudes, motivations, and meanings related to psi may provide a foundation for understanding psi that is more replicable and more widely accepted than research attempting to elicit psi. This foundation may provide a more favorable environment for research eliciting psi as well as more productive interactions with other disciplines and with skeptics.
Bem D.J. & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bulletin, 115, 4- 18.
Honorton, C. (1983). Response to Hyman's critique of psi ganzfeld studies. In W. G. Roll, J. Beloff & R. A. White (Eds.), Reseaarch in parapsychology 1982 (pp. 23-26). Metuchen, NJ: Scarecrow Press.
Houtkooper, J.M. (2002). [Letter to the editor]. Journal of Parapsychology, 66, 329-333.
Kennedy, J.E. (1994). Exploring the limits of science and beyond: Research strategy and status. Journal of Parapsychology, 58, 59-77.
Kennedy, J.E. (2003). The capricious, actively evasive, unsustainable nature of psi: A summary and hypotheses. Journal of Parapsychology, 76, 53-74.
NASH, C. (1989). Intra-experiment and intra-subject scoring declines in Extrasensory Perception After Sixty Years. Journal of the Society for Psychical Research, 55, 412-416.
Palmer, J., & Neppe, V.M. (2003). A controlled analysis of subjective paranormal experiences in temporal lobe dysfunction in a neuropsychiatric population. Journal of Parapsychology, 67, 75-97.
Radin, D., & Nelson, R. (2000). Meta-analysis of mind-matter interaction experiments: 1959 to 2000. [Unpublished Manuscript.] Boundary Institute, Los Altos, California and Princeton Engineering Anomalies Research, Princeton University.
Stanford, R.G. (2003). Research strategies for enhancing conceptual development and replicability. Journal of Parapsychology, 67, 15-51.
Steinkamp, F., Boller, E., & Bosch, H. (2002). Experiments examining the possibility of human intention interactions with random number generators: A preliminary meta-analysis [Abstract]. Journal of Parapsychology, 66, 238- 239.
Utts, J. (1991). Replication and meta-analysis in parapsychology. Statistical Science, 6, 363-403.
Watt, C., & Ramakers, P. (2003). Experimenter effects with a remote facilitation of attention focusing task: A study with multiple believer and disbeliever experimenters. Journal of Parapsychology, 67, 99-116.
J. E. Kennedy