Description of the Related Prior Art Many polling methods and techniques presently exist to obtain information about individual attitudes, beliefs or knowledge representative of a population. Scientifically accepted methods involve drawing a random sample of some well-defined population, such as, for example, all adult U. S. citizens, and then either interviewing the sample members, in-person or by telephone, or having them complete a written questionnaire, possibly on-line. The sample need not be a"simple"random sample, but may be designed to incorporate elements of clustering or stratification. The sample must include some form of random sample to provide at least an indication of adequately representing the target population.

These types of conventional polls, when well enough designed and executed, can claim representative samples, but provide no information and involve no deliberation. They simply report existing opinions, many of which are typically ill-informed and ill-considered. They do only a little, and even that little

only inadvertently, to engage or educate their respondents, who generally constitute only a tiny fraction of the population.

Current on-line polling methods share the above limitations in addition to typically relying on wholly volunteer or self-selected samples. Because they are not random, these samples cannot be relied on to accurately represent the population from which the volunteers are drawn. Large numbers of respondents in the samples does not improve sample accuracy because a nonrandom sample of any size affords no assurance of representativeness. Among many other possible biases, the volunteer samples in current on-line polls typically consist very disproportionately of people who are especially interested in and knowledgeable about the topic.

Another type of polls, known as"informative polls"measure respondents' attitudes, beliefs, or knowledge in the same way as ordinary polls, but then attempt to increase the quality of their respondents'thinking by providing additional information during the course of the poll. For example, the poll may provide factual information, competing arguments, or instruction in logic, or some forum for discussion, either before or in the course of re-measuring their attitudes, beliefs, or knowledge. One known poll included in the informative polls type poll is known as DELIBERATIVE POLLS (g).

DELIBERATIVE POLLSS, for example, expose their participants to a variety of"deliberative experiences,"comprised largely of balanced information about some policy or electoral choice (s) and the opportunity to think about and discuss those choice (s). These polls first measure the sample members'attitudes, beliefs, knowledge, and other characteristics by either interview or self-completion questionnaire. Then information and the opportunity for thought and discussion are provided, and the sample members'attitudes, etc. are again measured using the same questionnaire, typically augmented by a few additional questions. The changes in beliefs and attitudes between the two administrations of the same questionnaire suggest what the distribution of beliefs and attitudes in the whole public would be if the whole public were to have similar exposure to information and opportunity to think about and discuss the issues.

Informative polls, including DELIBERATIVE POLLS are thus a cross between a survey (polling) and an experimental design. Some informative polls

feature"quasi control groups"in which certain samples include members that have completed the same questionnaire at least once (at the same time as the participants are completing it for the second time) but have not been exposed to any deliberative experiences. The contrasts between the participant sample and these quasi control groups help establish that any changes in attitudes, beliefs, or knowledge are ascribable to the package of deliberative experiences rather than adventitious factors such as major news events.

While the above described informative polls may claim representative samples, typically isolated pieces of information are presented to the sampled group, and no real deliberation is involved. The informative poll generally lacks both discussion and time for reflection to form an opinion result improved by additional information.

In addition, these types of informative polls usually bring representative samples to a single site for face-to-face deliberations. These types of deliberations have serious limitations, including: a) High Cost. Are very costly. For populations spread over a large territory, the transportation and housing costs are substantial. The housing costs mount as the number of days of on-site deliberation increases. The fuller the deliberation, the more expensive the deliberative poll. b) Time Constraints. The deliberations are often limited to weekends or other short intervals, since few people can leave their families, jobs, or other responsibilities for more than a few days, and the few who could would hardly be a representative sample. c) Representative Sampling. The deliberations are limited for reasons of expense and logistics to samples numbering in the hundreds. The poll therefore cannot directly engage the broader population in similar deliberations. d) Informative Materials. The additional information provided to the sample is limited to balanced information consisting of specially prepared "briefing materials"that are internally balanced. There is usually no way of monitoring the participants'consumption of other, individually partisan but collectively balanced materials that might also be provided.

The present invention is intended to use on-line deliberations to obtain data about the informed opinions of representative samples, while at the same

time engaging and educating larger numbers of population members via similar on-line deliberations and discovering the criteria that differentiates more informed and thoughtful opinions.

SUMMARY OF THE INVENTION It is an object of the present invention to overcome the drawbacks of the prior art discussed above.

Briefly stated, there is provided according to the present invention an indication of relevant information impacting a change in opinion or conclusion for a given issue that is derived using questionnaires while providing pertinent discussion materials and forums. A randomly selected group from a population is polled to collect an range of opinions on a given topic, and then selectively exposed to material pertinent to the topic to improve group topic knowledge.

Another group composed of volunteers from the population is also initially polled and selectively exposed to topic-pertinent materials. Both groups discuss or deliberate about the given topic, and are subsequently polled. Changes in opinions or conclusions are correlated to the selection of material to which the group was exposed. By obtaining a correlation to topical materials, an impact of particular items in the materials can be measured for specific responses by either group. A poller can obtain more precise and informed opinions on a given subject, and understand the factors which influence the opinions.

The present invention provides a method for engaging large numbers of volunteer users, such as Internet users, in on-line deliberations about policy and electoral choices and at the same time using data from these users to determine the effects of information, thought, discussion, and other deliberative experiences on their policy or electoral preferences. The method accommodates typical on-line volunteer samples but randomly assigns its volunteer participants to measurement conditions involving one or more questionnaires or interviews and to treatment conditions involving the presence or absence of particular deliberative experiences. These deliberative experiences preferably include, but are not limited to, reading briefing materials, discussing the issue in small groups, and/or watching video presentations.

The present invention, therefore, provides a valuable scientific use for the volunteer samples characteristic of on-line polls. The self-selected nature of these samples minimizes their scientific value for polling, but the present invention uses the samples to obtain experimental field data to determine causal effect information. The value of the results is increased by the fact that volunteers are given random assignments for deliberation. The absence of random sampling undercuts inferences from the participants to the whole population, but the presence of random assignment facilitates inferences about information that has produced any changes in attitudes, beliefs, or knowledge among the participants. In short, the present invention uses volunteer samples to achieve sound empirical results rather than less reliable results derived in polls according to the prior art.

The present invention therefore complements polls based solely on deliberative experiences in two ways. First, the presence of random assignment makes the method of the present invention a stronger experimental design and thus permits firmer inferences regarding the effect of additional information. The stronger inferences help to validate the package of deliberative experiences that produces any observed changes in attitudes, beliefs, or knowledge. Second, the availability of large numbers of participants and the ability to assign them to numerous treatments involving the presence or absence of several, perhaps many, deliberative experiences permits an assessment of the elements of the additional information package that are effective, and a quantification of effects.

The on-line RM component permits much cheaper yet much lengthier and thus fuller deliberation. The OCG component involves a larger portion of the population in parallel deliberations with the same materials.

BRIEF DESCRIPTION OF THE DRAWINGS Other features and advantages of the present invention will become apparent from the following description of the invention which refers to the accompanying drawings.

Fig. 1 is a diagram of an example hardware arrangement of a system constructed in accordance with the principles of the present invention;

Fig. 2 is flow diagram of the operation of the system according to a first embodiment of the present invention; Fig. 3 is a flow diagram of specifics of operation of the system according to a second embodiment of the present invention; and Fig. 4 is diagram of the operation of the system according to a third embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION Referring now to Fig. 1, an example of a hardware arrangement constructed in accordance with the principles of the present invention is shown.

Referring to Fig. 1, system 10 is preferably comprised of at least one information processor 12, at least one participant terminal 14, at least one expert terminal 16 and at least one moderator terminal 18 each of which are coupled to communication network 20. Communication network 20 is preferably a global public communication network such as the Internet. Information processor 12 and terminals 14-18 can be any devices capable of sending and receiving data across communication network 20, for example mainframe computers, personal computers, laptop computers, personal digital assistance (PDA) or Internet access device such as Web TV. In addition, terminals 14-18 are preferably equipped with a standard web browser, such as MICROSOFT INTERNET EXPLORER, NETSCAPE COMMUNICATOR, and the like to facilitate on-line interactivity as described above. Of course different types of communication applications can be used, as necessitated by the users'interactivity requirements. Information processor 12 and terminals 14-18 are coupled to communication network 20 using any known data communication networking technology.

Information processors 12 include a database equipped with sufficient storage to store questionnaires, survey data, complied results and the like and also preferably act as the web server for communicating hypertext mark-up language (HTML), Java applets, Active-X control programs and the like to terminals 14-18. Information processors 12 are arranged with components, for example those shown in Fig. 2, suitable to the expected operating environment of information processor 12. For example, the central processing unit (s), network

interface (s), memory and storage capacities are arranged to accommodate the expected demand.

It is contemplated that the method and functions described above can be implemented using any known programming language, for example"C++", visual basic, object-oriented programming techniques and the like.

Participant terminals 14 are preferably used by participants to support their on-line activity as described above. Similarly, expert terminals 16 and moderator terminals 18 are used to support the respective functions of the experts and moderators as described above.

A method according to the present invention relies on deliberation, meaning learning and reflection aided by discussion. The method applies to any given population, meaning any well-defined set of individuals, including but not restricted to national, regional, local, and other public groups and the memberships of affinity groups, corporations or corporate divisions, trade associations, and other organizations.

Referring now to Fig. 2, a system according to the present invention, referred to as Deliberative Referendum (DR), has two interacting and complementary components. A first component, referred to as a Representative Microcosm (RM), is chosen through a random sample 32, including sampling conducted on-line. A second component, referred to as an Open Consultation Group (OCG), is a different sample consisting of volunteers 46 of the population, which may be randomly assigned to different deliberative experiences as defined below. In both components, the deliberations and measurement of opinions are preferably conducted on-line.

The components are complementary and interconnected, in addition to being novel. Together, they serve a combination of goals not readily attained otherwise. The RM results shed light on what the whole population would think about the issue with the benefit of greater thought and information. The OCG provides assistance to the population from members outside of the RM to urge the population to learn and think more about the issue. When members of the OCG are randomly assigned to deliberative experiences, it is possible to reveal the dynamics of opinion change from learning and thinking.

The inventive polling method analyzes changes in attitudes, beliefs, and

knowledge among volunteer participants, particularly on-line participants.

Initially, a volunteer sample is recruited and divided into random subsets. The method of the present invention is explained by way of a simple example using only two possible measurements and four possible treatments, consisting of the presence or absence of exposure to two deliberative experiences, briefing materials and small group discussions. It should be noted, however, that the invention is not limited solely to this measurement and exposure arrangement and can be expanded to include many possible measurements with many possible treatments, involving combinations of many deliberative experiences.

Initially, at"time 1", a population random sample 32 and a subset of randomly chosen volunteers 46 complete questionnaires 34 and 48, respectively, which are designed to measure their attitudes, beliefs, and knowledge relating to a particular topic or topics (as well as their sociodemographic and other characteristics). In the case of on-line users such as Internet users, questionnaire3 34 and 48 are provided as on-line questionnaires. On-line questionnaires 34 and 48 can be provided, for example, using Internet browser software and Internet web page servers as are known in the art and written in the hypertext mark-up language (HTML), java, common gateway interface-bin (CGI-BIN), etc.

Subsequently, at"time 2,"all participants, whether they completed the "time 1"questionnaire or not, are asked to complete second questionnaires 40 and 54, which are either the same or an augmented version of the time 1 questionnaires 34 and 48. In the interval between time 1 and time 2, random subsets of the participants are exposed to treatments consisting of the presence or absence of the two deliberative experiences, briefings 36,52 and deliberations 38, 50. In this example, the set of four possible treatments are thus (briefing, deliberation), (briefing, no deliberation), (no briefing, deliberation) and (no briefing, no deliberation). The materials for briefings 36,52 (if present) are presumed to precede deliberations 38,50 (if present), but the sequence could be varied, in which case (briefing, deliberation) and (deliberation, briefing) would be different treatments.

The time 2 questionnaires 40 and 54 are administered immediately after the experiences of deliberations 38,50, or approximately so in the cases where the random subsets are not actually exposed to any deliberative experiences, such

as numbers of sample 32 and volunteers 46 who receive a treatment of (no briefing, no deliberation).

It should be noted that, in addition to questionnaires 34 and 48 themselves, materials for briefings 36,52 will normally (but need not) be presented on-line, in the form of multimedia presentations, text or audio information, etc. The small group discussions, similarly, will normally (but need not) occur via electronic mail, chat rooms, etc. Of course, the deliberative experiences can be conveyed in a more traditional manner such as with printed textual materials and/or live assembled groups.

By varying the treatment (exposure to deliberative experiences) from no deliberation at all to the whole package of available deliberative experiences, the present invention permits strong inferences to be made. The effects, within the numbers of sample 32 and volunteers 46, of both individual deliberative experiences and the whole package of deliberative experiences on the opinions and conclusions reached by the various groups can be determined.

The questionnaire data can be analyzed to ascertain the effects of deliberative experiences and previous measurement in correlations 42 and 56 (in the time 2 questionnaire or interview) on attitudes, beliefs, or knowledge (or changes therein) by any appropriate known statistical technique. For example, correlations 42 and 56 can include regression models with attitudes, beliefs, or knowledge (or changes therein) as dependent variables and at least a subset of the deliberative experiences and previous measurement among the explanatory variables. It is presumed that one of ordinary skill in the art would be able to determine appropriate statistical models and estimators.

As an alternative embodiment, the method of the present invention is used in conjunction with informative polls, such as DELIBERATIVE POLLSO, conducted with random samples on the same subject. In this connection, the method of the present invention can be used to reinforce the inference that something about the package of deliberative experiences is indeed causing any changes in attitudes, beliefs or knowledge in the informative polls and to determine which among the deliberative experiences is having how much effect.

It is the random assignment of participants to treatment and measurement groups that makes these inferences possible. It should be noted, however, that the present

invention need not be implemented in conjunction with an informative poll.

An example of an on-line poll using the method of the present invention and an on-line informative poll, is as follows. First, an advisory board or other body responsible for selecting polling topics determines a group of candidate topics and then chooses among them at least partly on the basis of volunteered users'opinions. Next, the random sample for the informative poll is drawn, off-line, and Internet access provided for its members. The sample members complete an initial questionnaire, either as part of an off-line interview when initially contacted or on-line immediately following the provision of Internet access, or both. Information about the topic and opportunities for discussing the topic are then made available to sample members through such synchronous and asynchronous on-line techniques as Internet web pages, electronic mail, chat rooms, message boards, Internet forums and the like. (The Internet web pages are the on-line analogue of the"briefing materials"in off-line informative polls.) Finally, the sample members complete a questionnaire containing, but not necessarily confined to, the same questions as the questionnaire they completed at the beginning. A preferred method for this example follows.

The Representative Microcosm (RM) The members of the Representative Microcosm (RM) are a random sample of a given population, as defined above. The sample members are engaged in on-line (Internet) deliberations about issues affecting collective decisions. The method gathers data from these samples to estimate the distribution of more informed and thoughtful opinion. The measurement of opinion, like the deliberation, is preferably conducted on-line.

More precisely, the RM component of the DR proceeds in the following way with reference to Figs. 3 and 4: Step One: A random sample 72 of the population is drawn, either on-or off-line.

All sample members are asked to complete an on-line questionnaire concerning the collective decision at issue, and other demographic and attitudinal items.

They are then randomly divided into small groups 74, preferably of

approximately 8 to 12 participants.

Step Two: The small group members introduce themselves to one another in a small group meeting 76. Depending on the circumstances, they may or may not be encouraged to give their real names, professions, and background. The small group members also"meet"their on-line moderator, who will explain that the moderator's role is to encourage but not shape dialogue.

Step Three: Immediately after the round of introductions, or at approximately the same time, the small group members are provided with background briefing information 78 regarding the first of one or more issues. This information is preferably comprised of internally balanced briefing materials laying out the arguments for and against each of a set of major proposals for dealing with the issue, a"library"of collectively balanced links to other Internet sites, or both.

The internally balanced briefing materials may be designed as hyperlinked web pages, which may include video, graphics, streaming audio or audio/video, or any combination thereof. The links to other sites may be individually partisan, so long as they are collectively balanced.

Step Four : The small group members discuss this issue among themselves in special Internet dialogues, a separate dialogue for each group. This discussion 80 may occur synchronously, in scheduled chat rooms, or asynchronously, on moderated message boards. When the participants are already relatively expert about the area of discussion, the moderator may begin the discussion by asking whether or not the background materials adequately cover the issues. Otherwise, the moderator begins the discussion by asking the small group members what they see as the advantages and disadvantages of the first proposal regarding this first issue.

As the discussion unfolds, the number of contributions regarding each advantage or disadvantage are counted, and the moderator makes further

interventions as necessary to ensure that all the major considerations are adequately discussed. The moderator similarly intervenes as necessary to ensure that all the major advantages and disadvantages of all the major proposals are adequately discussed in turn, although the flow of the discussion may well naturally cover the ground without much prompting. At the end of the period allotted to this step, the moderator inquires as to which key questions about this first issue would the group members want to ask a panel of experts. This sequence is illustrated in deliberation step 62 of Fig. 3.

Step Five: At the end of Step Four, the questions from all the small groups are collected, sorted by topic, and posted in an expert dialogue 82 for review and discussion by pre-selected experts. The expert discussion lasts through the end of Step Six, described below. Participants will visit this expert dialogue on a read-only basis. In addition, selected questions from the Open Consultation Group (OCG) component of the process are also inserted into this stage and posed to experts and made available to the respondents (as described below). The expert review step is illustrated in expert review 64 in Fig. 3.

Steps Three, Four, and Five are repeated for each issue in turn as illustrated by respective reference designators 84,86,88,90,94 and 96. The expert dialogues are independent for each issue and continue even after the small groups have turned to subsequent issues, until Step Six, described below.

Step Six: The small groups discuss the expert dialogues on all the issues in discussion 92, in the order in which they were raised (so that the expert dialogues on later-raised issues have more time to evolve). The discussion may address the questions of how well the experts responded to the questions, what if any important new aspects of the issues emerged in the expert dialogues, and above all what the group members'individual opinions now are. Review step 66 of Fig. 3 shows this step.

Step Seven: After these final small group discussions, the participants are surveyed again in poll 98. They are asked everything from the pre-deliberation survey except for those sociodemographic and other questions unlikely to exhibit change, plus some additional questions about their experience of the process.

Deliberation step 68 in Fig. 3 illustrates the reformulation of opinions or conclusions based on results obtained in review step 66. The results are then analyzed, and a report drafted for either the media, the sponsor, or both, depending on the nature and sponsorship of the particular DR. The results from DRs sponsored by organizations may be used internally, externally, or both, in part or in whole.

Outlines for Discussion The moderators are provided with an outline, consistent with the organization of any briefing materials, of the issues, proposals, and arguments to be covered. They are instructed to begin with a designated issue (which need not be same issue in every group) and to ensure that all the issues, proposals, and arguments in the outline receive adequate attention.

The Role of the Moderators Small-group moderators intervene to ensure that all major arguments about all the major proposals for dealing with all the issues slated for discussion are voiced and considered, that the discussion stays on-topic and civil, and, so far as possible, that all participants take an active part. Among other things they may e-mail participants who have not done much if any posting on their small group's message board, encouraging them to do more.

Timing The exact duration of each step will vary from DR to DR, depending on the budget, the urgency of the results, and the number and complexity of issues included. Steps Four, Five, and any repetitions thereof can be expected to take longest, followed by Step Three and any repetitions thereof. The whole process will normally take anywhere from a few days to a few months (and occasionally,

when called for, still longer).

Incentives for Participation People drawn into the sample of the RM may be offered incentives (a) to participate and (b) to participate seriously and actively. Incentives of the first kind preferably include a flat-amount honorarium or a free, equal chance in a participants-only lottery in which the prize (s), in cash, goods, or services, would be of considerably greater value than the honorarium. Incentives of the second kind preferably gear either the honorarium or the lottery chance to the extent of the participant's activity in pursuing links to the briefing materials and other information and in contributing to the message boards. The software implementing the functions of the present invention counts both sorts of activity.

At approximately the same time as the sample for the on-line informative poll is being drawn, a separate sample of volunteer users, preferably recruited on-line, agree to participate in an experiment based on the method of the present invention. Any number of on-line users can be involved the experiment, subject only to the capacities of the computer processors involved. The volunteers are randomly divided into subsets or"treatment groups,"then exposed or not exposed to one or more measurements various combinations of deliberative experiences, as described above. The set of deliberative experiences to which various random subsets of the volunteer sample are exposed is the same as the set to which all the random sample members are exposed in the informative poll, and in both cases the deliberative experiences will extend over approximately the same period. An initial questionnaire, completed by some random subset of this volunteer sample, is the same as that completed by the random sample at the beginning of the informative poll. At the end of the experiment, all the volunteer sample members complete the same questionnaire as that completed by the by the random sample at the end of the informative poll. Both questionnaires are completed at the same time by the relevant members of both samples. A description of a preferred realization of this example follows.

The Open Consultation Group (OCG ! The OCG parallels and draws on the RM. The same briefing materials

and questionnaires are used for both. The OCG generates questions that are incorporated in the agenda of expert review 64, together with questions and answers considered by the RM (as noted above) and uses a distillation of issues raised in the RM as one of its deliberative experiences.

The biggest single difference between the groups, or components, is that whereas the members of the RM are recruited by random sampling, those of the OCG are self-selected. This openness to volunteers makes the OCG sample unrepresentative in the same way as those of most ordinary Internet polls.

However, the openness also allows a much larger number of population members than in the RM to share the benefits of deliberation-to learn, think, and talk about the issue and to become more engaged in the political system or sponsoring organization. As compared to the RM, then, the OCG engages a much larger number of population members. Compared to ordinary Internet polls, the OCG involves deliberation not generally available in on-line forums.

In addition, the members of the OCG may be randomly assigned to different deliberative experiences. In this random-assignment version, the OCG participants become the subjects of a true social science experiment designed to verify that elements of the deliberative process produce the changes of opinion in the RM. The experimental, or empirical, nature of this technique can also assist in estimating the weight that certain elements of the process provide in reaching a result.

This random-assignment version of the OCG may be illustrated by a simple example involving only two possible measurements and four possible treatments, consisting of the exposure or absence of exposure to two"deliberative experiences,"namely the RM briefing materials and small group discussions paralleling the RM's.

The OCG begins with a random subset of its volunteers completing an on-line questionnaire designed to measure their attitudes, beliefs, and knowledge relating to a particular topic or topics (as well as their sociodemographic and other characteristics). Subsequently, all the volunteers complete the same on-line questionnaire, possibly augmented by additional questions. In the interval between these two measurements, random subsets of the volunteers are exposed to treatments consisting of the presence or absence of the two deliberative

experiences. In this example, the four treatments are thus (briefing, discussion), (briefing, no discussion), (no briefing, discussion) and (no briefing, no discussion). The second questionnaire is administered immediately after the deliberative experiences, or approximately so in the cases of the random subsets not actually exposed to any deliberative experiences, such as those for whom the treatment is (no briefing, no discussion) in this example.

In this example, the briefing materials, if present, are presumed to precede the discussion, if present, but the sequence could be varied, in which case (briefing, discussion) and (discussion, briefing) would be different treatments.

Note also that the OCG is not limited solely to this measurement and exposure arrangement, and can be expanded to include many possible measurements with many possible treatments, involving combinations of many deliberative experiences.

By varying the treatment (exposure to deliberative experiences) from no deliberation at all to the whole package of available deliberative experiences, the present invention permits strong inferences about the effects, within the volunteer sample, of both individual deliberative experiences and the whole package of deliberative experiences.

The questionnaire data can be analyzed to ascertain the effects of deliberative experiences and previous measurement (in the initial questionnaire) on attitudes, beliefs, or knowledge (or changes therein) by any appropriate known statistical technique, most notably including regression models with attitudes, beliefs, or knowledge (or changes therein) as dependent variables and at least a subset of the deliberative experiences and previous measurement among the explanatory variables. It is presumed that one of ordinary skill in the art would be able to determine appropriate statistical models and estimators.

The OCG thus complements the RM. It is a way of engaging large numbers of volunteer population members in on-line deliberations. Because the sample is not representative, the results are not reported in that light. To learn what a more informed and thoughtful population would think about the issue, there are the results of the RM; to get more of the population informing themselves and thinking about the issue, there is the OCG.

In the random-assignment version, the OCG also complements the RM in

two further ways. First, the presence of random assignment makes it a stronger experimental design and thus permits firmer inferences that it is indeed something about the package of deliberative experiences that is producing any observed changes in attitudes, beliefs, or knowledge. In terms common to social science analysis, the RM is particularly high on external validity (representation of the population), the OCG particularly high on internal validity (certainty about what is responsible for the observed changes of opinion). Second, the availability of large numbers of participants and the ability to assign them to numerous treatments involving the presence or absence of several, perhaps many, deliberative experiences permits some assessment of which elements of the package are having what effects.

The random-assignment OCG thus provides a valuable scientific use for the volunteer samples characteristic of on-line polls. The self-selected nature of these samples minimizes their scientific value for polling, but the OCG uses them instead for a true experiment, for which they are indeed useful, given random assignment.

The above-described process according to an aspect of the method of the present invention in combination with an on-line informative poll permits a stronger inference regarding the impact of items in the package of deliberative experiences to be made. The stronger inference permits a correlation between items in the package and changes in attitudes, beliefs, or knowledge in addition in the informative poll. In addition, an assessment of the impact of the various items can be made to help quantify their effect. The present invention also actively involves a great many more individuals in the deliberative process than could be done by an informative poll alone. In addition, the present invention puts the self-selected on-line polls, common on the Internet, to legitimate scientific use by refashioning them into empirical data gathering activities.