The Delphi Technique: Making Sense of Consensus

A peer- brushuped electronic journal. Copy mature is retained by the first or sole author, who grants right of first publication to the practical(a) Assessment, enquiry & Evaluation. Permission is granted to go this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. Volume 12, bend 10, August 2007 ISSN 1531-7714 The Delphi proficiency Making Sense Of Consensus Chia-Chien Hsu, The Ohio subject University & Brian A. Sandford, Oklahoma severalize University The Delphi proficiency is a widely utilise and accepted method for comp either selective culture from respondents deep down their domain of expertise.The technique is aimed as a chemical sort communication swear out which aims to achieve a convergence of tactile sensation on a unique(predicate) real-world issue. The Delphi p measly has been dropd in discordant field of take in such as computer programme planning, needs appraisement, form _or_ system of government determination, and resource utilization to develop a full range of substitutes, research or expose key assumptions, as intumesce as agree savvys on a topic spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by utilize a series of questionnaires delivered using multiple closed circuits to collect information from a table of selected subjects. yield selection, condemnation frames for conducting and completing a theater, the possibility of low answer respects, and unin ten-spottionally guiding feedback from the respondent convocation be atomic bet 18as which should be considered when conniving and implementing a Delphi field of view. The Delphi technique, mainly higher(prenominal)ly-developed by Dalkey and Helmer (1963) at the Rand Corporation in the 1950s, is a widely utilize and accepted method for achieving convergence of whim concerning real-world friendship solicited from experts at botto m certain topic beas.Predicated on the rationale that, ii heads be better than one, or n heads be better than one (Dalkey, 1972, p. 15), the Delphi technique is designed as a group communication execute that aims at conducting detailed examinations and discussions of a specific issue for the purpose of cultivation setting, policy investigation, or predicting the occurrence of emerging events (Ulschak, 1983 Turoff & Hiltz, 1996 Ludwig, 1997). Common panoramas try to localize what is, whereas the Delphi technique attempts to address what could/should be (Miller, 2006).In the literature, Delphi has been applied in discordant fields such as program planning, needs sagaciousness, policy determination, and resource utilization. Delbecq, avant-garde de Ven, and Gustafson (1975) specifically indicate that the Delphi technique great betray be apply for achieving the adjacent objectives 1. To determine or develop a range of attainable program alternatives 2. To explore or ex pose underlying assumptions or information wind to divers(prenominal) judgments 3. To seek out information which whitethorn fuss a consensus on the part of the respondent group 4.To correlate informed judgments on a topic spanning a wide range of disciplines, and 5. To educate the respondent group as to the diverse and interrelated aspects of the topic (p. 11). CHARACTERISTICS OF THE DELPHI TECHNIQUE The Delphi technique is well suited as a means and method for consensus-building by using a series of questionnaires to collect selective information from a board of selected subjects (Dalkey & Helmer, 1963 Dalkey, 1969 Linstone & Turoff, 1975 Lindeman, 1981 Martino, 1983 Young & Jamieson, 2001).Delphi, in contrast to other data gathering and compend techniques, employs multiple iterations designed to working Assessment, research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique develop a consensus of intellection concerning a specific topic. Ludwig (1994) indicates Iterations refer to the feedback process. The process was viewed as a series of rounds in each round both participant worked through a questionnaire which was returned to the researcher who collected, edited, and returned to every participant a landed e sayment of the position of the whole group and the participants aver position.A summation of comments made each participant awargon of the range of opinions and the reasons underlying those opinions (p. 55). More specifically, the feedback process allows and encourages the selected Delphi participants to reassess their initial judgments about the information provided in previous iterations. Thus, in a Delphi probe, the results of previous iterations regarding specific renderments and/or items tail change or be limited by individual panel members in later iterations based on their qualification to look back and assess the comments and feedback provided by the other Delphi pane amounts.Other nonable characteristics inherent wi th using the Delphi technique are the ability to provide anonymity to respondents, a controlled feedback process, and the suitability of a anatomy of statistical epitome techniques to interpret the data (Dalkey, 1972 Ludlow, 1975 Douglas, 1983). These characteristics are designed to starting signal the shortcomings of conventional means of pooling opinions obtained from a group interaction (i. e. , influences of dominant individuals, racket, and group pressure for conformity) (Dalkey, 1972).One of the primary characteristics and advantages of the Delphi process is subject anonymity which dismiss lessen the effects of dominant individuals which often is a concern when using group-based processes used to collect and synthe surface information (Dalkey, 1972). Additionally, the issue of confidentiality is facilitated by geographic airing of the subjects as well as the use of electronic communication such as e-mail to solicit and exchange information.As such, certain downsides ass ociated with group kinetics such as manipulation or coercion to conform or adopt a certain view operate tummy be lessen (Helmer & Rescher, 1959 Oh, 1974 Adams, 2001). Controlled feedback in the Delphi process is designed to reduce the effect of noise. Based upon Dalkey (1972), noise is that communication which occurs in a group process which both distorts the data and deals with group and/or individual interests rather than focusing on problem solving.As a result, the information developed from this mixture of communication in global consists of bow not related to the purposes of the essay. Basically, the controlled feedback process consists of a well organized compend of the anterior iteration intentionally distributed to the subjects which allows each participant an prospect to generate additional insights and more thoroughly clarify 2 the information developed by previous iterations.Through the operation of multiple iterations, subjects are judge to twist more probl em-solving oriented, to offer their opinions more insightfully, and to minimize the effects of noise. Finally, the ability to use statistical abstract techniques is a practice which further reduces the potential of group pressure for conformity (Dalkey, 1972). More specifically, statistical analysis basis break that opinions generated by each subject of a Delphi meditate are well represented in the final iteration because, at the end of the proceeding there may still be a significant turn out in individual opinions Dalkey, 1972, p. 21). That is, each subject would puzzle no pressure, both real or perceived, to conform to another participants receipts that may originate from obedience to social norms, customs, organisational culture, or standing at bottom a profession. The tools of statistical analysis allow for an objective and impartial analysis and summarization of the collected data. THE DELPHI PROCESS Theoretically, the Delphi process can be constantly iterated until consensus is determined to have been achieved.However, Cyphert and Gant (1971), Brooks (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) point out that three iterations are often sufficient to collect the needed information and to reach a consensus in most cocktail dresss. The following discussion, however, provides guidelines for up to four iterations in pitch to assist those who decide to use the Delphi process as a data assembling technique when it is determined that additional iterations beyond three are needed or valuable. Round 1 In the first round, the Delphi process traditionally begins with an open-ended questionnaire.The open-ended questionnaire serves as the cornerstone of soliciting specific information about a content area from the Delphi subjects (Custer, Scarcella, & Stewart, 1999). After receiving subjects responses, investigators need to shift the collected information into a well-structured questionnaire. This questionnaire is used as the survey legal instrument for the second round of data collection. It should be noted that it is both an acceptable and a common modification of the Delphi process format to use a structured questionnaire in Round 1 that is based upon an extensive review of the literature.Kerlinger (1973) noted that the use of a modified Delphi process is appropriate if staple fibre information concerning the target issue is available and usable. Round 2 In the second round, each Delphi participant receives a second questionnaire and is asked to review the items summarized by the investigators based on the information provided in the first round. Accordingly, Delphi panelists may be required to rate or rank-order Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique items to establish preliminary priorities among items.As a result of round two, areas of divergence and agreement are identified (Ludwig, 1994, p. 54-55). In some cases, Delphi panelists are aske d to state the rationale concerning rating priorities among items (Jacobs, 1996). In this round, consensus begins forming and the actual outcomes can be presented among the participants responses (Jacobs, 1996). 3 Rescher (1959), Klee (1972), and Oh (1974) concur that choosing individuals who are simply cognitionable concerning the target issue is not sufficient nor recommended.Considering the necessity of selecting the most qualified individuals, Delbecq, Van de Ven, and Gustafson (1975) specifically state that three groups of people are well qualified to be subjects of a Delphi study. The authors recommend (1) the top attention purpose makers who will utilize the outcomes of the Delphi study (2) the professional staff members together with their support team and (3) the respondents to the Delphi questionnaire whose judgments are creation sought-after(a) (p. 85). Delphi subjects should be highly trained and competent within the specialised area of knowledge related to the targe t issue.Investigators need to closely poke into and seriously consider the qualifications of Delphi subjects. Oh (1974) indicates that choosing appropriate subjects is generally based on the judgment and discretion of the principal investigators. Jones and Twiss (1978) state that the principal investigators of a Delphi study should chance upon and select the most appropriate individuals through a nomination process. Ludwig (1994) similarly states that, solicitation of nominations of well-known and respected individuals from the members within the target groups of experts was recommended (p. 2). ecumenicly, the pool of selecting possible Delphi subjects is likely to use positional leaders (Kaplan, 1971 Ludwig, 1994), to follow a review of authors of publications in the literature (Meyer, 1992 Miller, 2001), and/or to make contacts with those who have firsthand relationships with a particular issue (Jones, 1975 Anderson & Schneider, 1993). The latter basically consists of individu als who are primary stakeholders with heterogeneous interests related to the target issue or research effort.Concerning the appropriate outcome of subjects to involve in a Delphi study, Delbecq, Van de Ven, and Gustafson (1975) recommend that researchers should use the minimally sufficient number of subjects and should seek to verify the results through follow-up explorations. Ludwig (1994) notes that the number of experts used in a Delphi study is generally determined by the number required to constitute a typical pooling of judgments and the information affect capability of the research team (p. 52). However, what constitutes an optimal number of subjects in a Delphi study never reaches a consensus in the literature.Delbecq, Van de Ven, and Gustafson (1975) suggest that ten to fifteen subjects could be sufficient if the background of the Delphi subjects is homogeneous. In contrast, if various propagation groups are involved in a Delphi study, more subjects are expect to Round 3 In the third round, each Delphi panelist receives a questionnaire that includes the items and ratings summarized by the investigators in the previous round and are asked to revise his/her judgments or to allege the reasons for remaining outside the consensus (Pfeiffer, 1968, p. 52). This round gives Delphi panelists an opportunity to make further clarifications of both the information and their judgments of the relative importance of the items. However, compared to the previous round, only a slight increase in the degree of consensus can be expected (Weaver, 1971 Dalkey & Rourke, 1972 Anglin, 1991 Jacobs, 1996). Round 4 In the fourth and often final round, the list of remaining items, their ratings, minority opinions, and items achieving consensus are distributed to the panelists.This round provides a final opportunity for participants to revise their judgments. It should be remembered that the number of Delphi iterations depends strikingly on the degree of consensus sought by the investigators and can vary from three to five (Delbecq, Van de Ven, Gustafson, 1975 Ludwig, 1994). Subject Selection Regarding the selection of subjects for a Delphi study, choosing the appropriate subjects is the most important cadence in the entire process because it directly relates to the quality of the results generated (Judd, 1972 Taylor & Judd, 1989 Jacobs, 1996).Since the Delphi technique focuses on eliciting expert opinions over a short period of time, the selection of Delphi subjects is generally dependent upon the disciplinary areas of expertise required by the specific issue. Regarding any set standards of selecting Delphi subjects, there is, in fact, no exact criterion currently listed in the literature concerning the selection of Delphi participants. That is, throughout the Delphi literature, the definition of Delphi subjects has remained ambiguous (Kaplan, 1971, p. 24).Regarding the criteria used to guide the selection of Delphi subjects, individuals are conside red eligible to be invited to participate in a Delphi study if they have somewhat related backgrounds and experiences concerning the target issue, are capable of contributing helpful inputs, and are willing to revise their initial or previous judgments for the purpose of reaching or attaining consensus (Pill, 1971 Oh, 1974). Helmer and Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique be needed.Witkin and Altschuld (1995) note that the approximate size of a Delphi panel is generally under 50, but more have been employed. Ludwig (1997) documents that, the majority of Delphi studies have used between 15 and 20 respondents (p. 2). In sum, the size of Delphi subjects is variable (Delbecq, Van de Ven, & Gustafson, 1975). If the warning size of a Delphi study is too small, these subjects may not be considered as having provided a representative pooling of judgments regarding the target issue.If the sample size is too large, the drawbacks inherent within the Delphi technique such as potentially low response rates and the pledge of large blocks of time by the respondents and the researcher(s) can be the result. Time Requirements Conducting a Delphi study can be time-consuming. Specifically, when the instrument of a Delphi study consists of a large number of statements, subjects will need to dedicate large blocks of time to complete the questionnaires.Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a minimum of 45 old age for the administration of a Delphi study is necessary. With regard to the time management between iterations, Delbecq et al. (1975) note that giving two weeks for Delphi subjects to respond to each round is encouraged. Ludwig (1994) indicates, a drawback to Delphi was that the questionnaire method may slow the process greatly as some(prenominal) days or weeks may pass between rounds (p. 54).More specifically, since developing the instrument, collection the data, and administering the questionnaire are interconnected between iterations, ensuring Delphi subjects respond to the investigators on time does in many ways either promote or prohibit the ability of the investigators in analyzing the data, developing a new instrument based upon the prior responses, and distributing subsequent questionnaires in a timely fashion. These are challenging aspects of conducting a Delphi study and do require proper planning and management. The use and preponderance of electronic technologies (i. e. e-mail, teleconferencing, etc. ) may facilitate those who are interested in using the Delphi technique. Witkin and Altschuld (1995) note that electronic technology provides an opportunity for individuals to more tardily employ the Delphi process by taking advantages of, (1) the storage, processing, and speed of transmission capabilities of computers (2) the charge of respondent anonymity, and (3) the potential for rapid feedback (p. 204). Data Analysis Regarding data analysis, decision rules must be established to assemble and organize the judgments and insights provided by Delphi subjects.However, the kind and type 4 of criteria to use to both define and determine consensus in a Delphi study is subject to interpretation. Basically, consensus on a topic can be decided if a certain percentage of the votes falls within a prescribed range (Miller, 2006). One criterion recommends that consensus is achieved by having 80 percent of subjects votes fall within two categories on a seven-point plate (Ulschak, 1983). Green (1982) suggests that at least 70 percent of Delphi subjects need to rate three or higher on a four point Likert-type scale and the average has to be at 3. 5 or higher. Scheibe, Skutsch, and Schofer (1975) separate that the use of percentage measures is inadequate. They suggest that a more reliable alternative is to measure the stability of subjects responses in successive iterations. In the Delphi process, data analysis can invol ve both qualitative and quantitative data. Investigators need to deal with qualitative data if classic Delphi studies, which use open-ended questions to solicit subjects opinions, are conducted in the initial iteration.Subsequent iterations are to identify and hopefully achieve the requisite level of consensus as well as any changes of judgments among panelists. The major statistics used in Delphi studies are measures of central tendency (means, median, and elbow room) and level of dispersion (standard deviance and inter-quartile range) in order to present information concerning the collective judgments of respondents (Hasson, Keeney, & McKenna, 2000). Generally, the uses of median and vogue are favored. However, in some cases, as manifested by Murray and Jarman (1987), the mean is as well workable.Witkin (1984) questions the appropriateness of using the mean to measure the subjects responses if scales used in Delphi studies are not delineated at equal intervals. In the literat ure, the use of median score, based on Likert-type scale, is strongly favored (Hill & Fowles, 1975 Eckman, 1983 Jacobs, 1996). As Jacobs (1996) states, considering the anticipated consensus of opinion and the skewed expectation of responses as they were compiled, the median would inherently appear trounce suited to reflect the resultant convergence of opinion (p. 57).The use of mode is in addition suitable when reporting data in the Delphi process. Ludwig (1994) specifically intercommunicate that the Delphi process has a tendency to create convergence, and though this was usually to a single point, there was the possibility of polarization or clustering of the results slightly two or more points. In these instances, the mean or median could be misleading (p. 57). CONSIDERING DELPHI SHORTCOMINGS AND WEAKNESSES Potential of Low Response Rates collect to the multiple feedback processes inherent and integral to the concept and use of the Delphi process, thePractical Assessment, Res earch & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique potential exists for low response rates and seek to maintain robust feedback can be a challenge. In the Delphi technique, poor response rate is magnified fourfold because a maximum of four surveys may be sent to the same panelists (Witkin & Altschuld, 1995, p. 196). If a certain portion of the subjects discontinue their responses during various stages of the Delphi process, the quality of information obtained could be discounted or at least critically scrutinized.As such, Ludwig (1994) specifically addresses subject motivation as the key to the successful capital punishment of a Delphi study and investigators need to play an active role in this area to help ensure as high a response rate as possible. Consumption of Large Blocks of Time The Delphi technique can also be time-consuming and laborious. Unlike other data collection techniques such as the telephone survey and the face-to-face administration, which can be simultaneously conducted by a group of people and can be completed in a short period of time if the sample size is small, the Delphi technique is terative and sequential. As a result, the necessity of taking large block of time to successively complete a Delphi process is inescapable. Ludwig (1994) indicates that, a drawback to Delphi was that the questionnaire method may slow the process greatly as several(prenominal) days or weeks may pass between rounds (p. 54). Optimally speaking, the iteration characteristics of the Delphi process provide the opportunities for investigators and subjects to improve the accuracy of the results.In contrast, the same characteristic also increases the workload of investigators and the amount of time needed to successfully complete the data collection process (Cunliffe, 2002). Potential of Molding Opinions The iteration characteristics of the Delphi technique can potentially enable investigators to mold opinions (Altschuld, 2003). An experiment, cond ucted by Scheibe, Skutsch, and Schofer (1975), indicated that Delphi subjects would rate their responses differently after receiving a distorted feedback.Dalkey and Helmer (1963) also noted that, some leading by the experimenters inevitably resulted from the selection of the information supplied (p. 467). Moreover, Cyphert and Gant (1971) illustrated that a statement in their study was initially rated below average. However, Delphi subjects rated the statement above average after receiving simulated feedback. Therefore, Cyphert and Gant (1971) concluded that the Delphi technique could, be used to mold opinion as well as to collect data (p. 273).Indeed, subtle pressure to conform with group ratings was one of the major drawbacks in the Delphi technique (Witkin & Altschuld, 1995, p. 188). Delphi investigators need to be cognizant, pattern caution, and implement the proper safeguards in dealing with this issue. 5 Potential of Identifying General assertments vs. Specific Topic Relate d Information An assumption concerning Delphi participants is that they are eq in knowledge and experience (Altschuld & Thomas, 1991). However, this assumption might not be justified.More specifically, the expertise of Delphi panelists could be unevenly distributed, especially in the field of high technology (Marchant, 1988 Altschuld & Thomas, 1991). just about panelists may have much more in-depth knowledge of certain topics, whereas other panelists are more knowledgeable about different topics (Altschuld & Thomas, 1991, p. 187). Therefore, subjects who have less in-depth knowledge of certain topics are unable to specify the most important statements which have been identified by those subjects who possess in-depth knowledge concerning the target issue.The outcomes of a Delphi study could be the results of identifying a series of general statements rather than an in-depth exposition of the topic (Altschuld & Thomas, 1991). SUMMARY The Delphi technique provides those involved or i nterested in engaging in research, paygrade, fact-finding, issue exploration, or discovering what is really known or not known about a specific topic a flexible and adaptable tool to gather and see the needed data. Subject selection and the time frames for conducting and completing a Delphi study are two areas which should be considered carefully prior to initiating the study.The additional precautions concerning low response rates, unintentionally guiding feedback, and surveying panelists about their limited knowledge of the topic rather than soliciting their expert judgments should also be built into the design and implementation of the study. The Delphi technique has and will continue to be an important data collection methodological analysis with a wide variety of applications and uses for people who want to gather information from those who are immersed and imbedded in the topic of interest and can provide real-time and real-world knowledge. REFERENCES Adams, S. J. (2001).Pr ojecting the next decade in safety management A Delphi technique study. Professional Safety, 46 (10), 26-29. Altschuld, J. W. (2003). Delphi technique. Lecture, Applied evaluation design. The Ohio subject University. Altschuld, J. W. , & Thomas, P. M. (1991). Considerations in the application of a modified scree test for Delphi survey data. Evaluation Review, 15 (2), 179-188. Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Anderson, D. H. , & Schneider, I. E. (1993). Using the Delphi process to identify significant recreation research-based innovations.Journal of super C and deflection Administration, 11 (1), 25-36. Anglin, G. L. (1991). Instructional technology past, present and future. Englewood, CO Libraries Unlimited Inc. Brooks, K. W. (1979). Delphi technique Expanding applications. North Central Association Quarterly, 54 (3), 377-385. Cunliffe, S. (2002). prevision risks in the touristry industry using the Delphi technique. tour istry, 50 (1), 31-41. Custer, R. L. , Scarcella, J. A. , & Stewart, B. R. (1999). The modified Delphi technique A rotational modification. Journal of Vocational and Technical Education, 15 (2), 1-10.Cyphert, F. R. , & Gant, W. L. (1971). The Delphi technique A case study. Phi Delta Kappan, 52, 272-273. Dalkey, N. C. (1969). An experimental study of group opinion. Futures, 1 (5), 408-426. Dalkey, N. C. (1972). The Delphi method An experimental study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of life Delphi and decision-making (pp. 13-54). Lexington, MA Lexington Books. Dalkey, N. C. , & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9 (3), 458-467.Dalkey, N. C. , & Rourke, D. L. (1972). Experimental assessment of Delphi procedures with group value judgments. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of life Delphi and decision-making (pp. 55-83). Lexington, MA Lexington Books. Delbecq, A. L. , Van de Ven, A. H. , & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, IL Scott, Foresman, and Co. Douglas, D. C. (1983). A comparative study of the effectiveness of decision making processes which utilize the Delphi and leaderless group methodologies.Unpublished doctoral dissertation, The Ohio claim University, Columbus. Eckman, C. A. (1983). Development of an instrument to evaluate intercollegiate athletic coaches A modified Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Green, P. J. (1982, March). The content of a college-level outdoor lead course. story presented at the Conference of the Northwest District Association for the American 6 Alliance for Health, Physical Education, Recreation, and Dance, Spokane, WA. Hasson, F. , Keeney, S. , & McKenna, H. (2000). Research guidelines for the Delphi survey technique.Journal of Advanced Nursin g, 32 (4), 1008-1015. Helmer, O. , & Rescher, N. (1959). On the epistemology of the inexact science. Management Science, 6, 25-53. Hill, K. Q. , & Fowles, J. (1975). The methodological worth of the Delphi forecasting technique. proficient Forecasting and Social Change, 7, 179-192. Jacobs, J. M. (1996). Essential assessment criteria for physical education instructor education programs A Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Jones, C. G. (1975). A Delphi evaluation of agreement between organizations. In H. A. Linstone, & M.Turoff (Eds. ). The Delphi method Techniques and applications (pp. 160-167). Reading, MA Addison-Wesley publishing Company. Jones, H. , & Twiss, B. C. (1978). Forecasting technology for planning decision. London, UK Macmillan Press Ltd. Judd, R. C. (1972). Use of Delphi methods in higher education. expert Forecasting and Social Change, 4 (2), 173-186. Kaplan, L. M. (1971). The use of the Delphi method in organizat ional communication A case study. Unpublished masters thesis, The Ohio State University, Columbus. Kerlinger, F. N. (1973). Foundations of behavioral research.New York Holt, Rinehart, and Winston, Inc. Klee, A. J. (1972). The utilization of expert opinion in decision-making. AICHE Journal, 18 (6), 1107-1115. Lindeman, C. A. (1981). Priorities within the health care system A Delphi survey. Kansas City, MO American Nurses Association. Linstone, H. A. , & Turoff, M. (1975). Introduction. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method Techniques and applications (pp. 3-12). Reading, MA Addison-Wesley Publishing Company. Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H. A. Linstone, & M. Turoff (Eds. ).The Delphi method Techniques and applications (pp. 102-123). Reading, MA Addison-Wesley Publishing Company. Ludwig, B. G. (1994). Internationalizing Extension An exploration of the characteristics evident in a state university Extension system that achieves i nternationalization. Unpublished doctoral dissertation, The Ohio State University, Columbus. Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Ludwig, B. (1997). Predicting the future Have you considered using the Delphi methodology? Journal of Extension, 35 (5), 1-4. Retrieved November 6, 2005 from http//www. oe. org/joe/1997october/tt2. html Marchant, E. W. (1988). Methodological problems associated with the use of the Delphi technique Some comments. Fire Technology, 24 (1), 59-62. Martino, J. P. (1983). Technological forecasting for decision making. New York North-Holland. Meyer, J. H. (1992). Rethinking the lookout station of colleges whose roots have been in agriculture. Davis, CA University of California. Miller, G. (2001). The development of indicators for sustainable tourism Results of a Delphi survey of tourism researchers. Tourism Management, 22, 351-362. Miller, L. E. (2006, October).Determining what could/should be The Delphi te chnique and its application. Paper presented at the meeting of the 2006 annual meeting of the Mid-Western Educational Research Association, Columbus, Ohio. Murray, W. F. , & Jarman, B. O. (1987). Predicting future trends in adult fitness using the Delphi approach. Research Quarterly for exercising and Sport, 58 (2), 124-131. Oh, K. H. (1974). Forecasting through hierarchical Delphi. Unpublished doctoral dissertation, The Ohio State University, Columbus. Pill, J. (1971). The Delphi method Substance, context, a critique and an annotated bibliography.Socio-Economic Planning Science, 5, 57-71. 7 Scheibe, M. , Skutsch, M. , & Schofer, J. (1975). Experiments in Delphi methodology. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method Techniques and applications (pp. 262-287). Reading, MA Addison-Wesley Publishing Company. Taylor, R. E. , & Judd, L. L. (1989). Delphi method applied to tourism. In S. Witt, & L. Moutinho, (Eds. ). Tourism marketing and management handbook. New York Pren tice Hall. Turoff, M. , & Hiltz, S. R. (1996). Computer based Delphi process. In M. Adler, & E. Ziglio (Eds. ).Gazing into the oracle The Delphi method and its application to social policy and public health (pp. 56-88). London, UK Jessica Kingsley Publishers. Ulschak, F. L. (1983). Human resource development The theory and practice of need assessment. Reston, VA Reston Publishing Company, Inc. Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52 (5), 267-273. Witkin, B. R. (1984). Assessing needs in educational and social programs. San Francisco, CA Jossey-Bass Publishers. Witkin, B. R. , & Altschuld, J. W. (1995). Planning and conducting needs assessment A practical guide.Thousand Oaks, CA rational Publications, Inc. Young, S. J. , & Jamieson, L. M. (2001). Delivery methodology of the Delphi A comparison of two approaches. Journal of Park and Recreation Administration, 19 (1), 42-58. Citation Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique Ma king Sense of Consensus. Practical Assessment Research & Evaluation, 12(10). Available online http//pareonline. net/getvn. asp? v=12&n=10 Editors Note Another paper on the Delphi Technique that appeared in Practical Assessment Research & Evaluation is Yousuf, Muhammad Imran (2007).Using Experts Opinions through Delphi Technique. Practical Assessment Research & Evaluation, 12(4). Available online http//pareonline. net/getvn. asp? v=12&n=4 . Authors Chia-Chien Hsu Post-doctoral Studies The Ohio State University 393 Schrock Road Worthington, OH 43085 Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Tel (614) 885-0763 E-mail hsu. 127 at osu. edu Brian A. Sandford Assistant Professor 214 Willard Hall occupational Education Oklahoma State University Stillwater, Oklahoma 74074 405-744-3461 brian. sandford at okstate. edu 8