Extra copies of the report may be purchased by remitting the indicated purchase price to the Treasurer.

The Prometheus Society Membership Committee Report is copyrighted by the Prometheus Society. This report nor any portion thereof shall be republished in any form without the express permission of the Society as indicated in writing by the President.

By previous agreement of the members of this committee (including all officers of the Prometheus Society) with Darryl Miyaguchi (also one of our committee members), Darryl will have rights to publish this report on his website with all other rights and privileges being retained exclusively by the Prometheus Society.

This material is presented for the membership of the Prometheus Society in determining whether to accept the recommendation resulting from the deliberations of the 1998/99 Membership Committee deliberations.

Neither the Prometheus Society nor the Membership Committee warrant this material beyond its intended use. We do not maintain that there are no errors in this document. It is simply the best that we could do within the limitations of time and resources that were available to us.

Prometheus web site: http://prometheussociety,org

Report on-line: http://prometheussociety.org/mcrept/memb_comm_rept.html

For the User identification and Password to restricted areas of the web site where the Membership Committee Report is available, you may contact the Web Site Coordinator Fredrik Ullén.

Figure 1: Difficulty of Compromised Mega Problems 17

Figure 2: Mega vs. SAT Score Correlation 22

Figure 3: Equipercentile Equating of Mega and SAT 22

Figure 4: Correlation of Mega vs. Other Test's Score Pairs 23

Figure 5: Equipercentile Equating of Mega and GRE 24

Figure 6: Equipercentile Equating of Mega and CTMM 24

Figure 7: Mega48 IRT Test Scoring 25

Figure 8: Distribution of Mega Test Raw Scores for Sixth Norming 26

Figure 9: Mega IQ-Scaled Distribution (actual, predicted, general population) and filter 27

Figure 10: Mega IQ-Scaled Distribution (actual and predicted) -- log scale 28

Figure 11: Correlation of Score Pairs of Mega27 and Mega48 31

Figure 12: Mega27 IRT Test Norming 31

Figure 13: Mega (48-item) Test Scoring -- Traditional vs. Maximum Likelihood 32

Figure 14: Mega27 Test Scoring -- Traditional vs. Maximum Likelihood 32

Figure 15: Titan vs. Mega (48-item) Correlation of Score Pairs 34

Figure 16: Titan vs. Mega Equipercentile Equating 34

Figure 17: LAIT vs. Mega (48-item) Correlation of Score Pairs 35

Figure 18: SAT (Verbal Plus Mathematical Parts) Frequency data 40

Figure 19: Population Distributions for the SAT (general, actual, predicted) 40

Figure 20: SAT Actual and Predicted Distributions --log scale 41

Figure 21: SAT Discrimination Capabilities (Test1) 41

Figure 22: SAT Discrimination Capabilities (Test2) 41

Figure 23: GRE Equipercentile Equating with SAT for reported Score Pairs on Mega 48

Figure 24: GRE Correlation with MAT for 1341 Score Pairs 49

Figure 25: Extent of Data for CMT 54

Figure IX.1: A Normal Distribution 69

Figure X.1: Illustration for Confidence Interval Determination 76

Figure X.2: Difficulty Profile, p_n(C_K), for Problem #11 on the Mega 81

Some Available Psychometric Instruments List 15

Selectivity by IQ 20

Correlations of IQ Tests with Mega 23

Mega "Verbal" vs. "Non-verbal" Factor Analysis 28

LAIT "Verbal," "Spatial," and "Number" Factor Analysis 36

LAIT Rotated "Fluid" and "Crystalized" Factors 36

SAT Coaching Improvement Table 38

SAT High Range Data Distribution (1984) 42

SAT High Range Data Distribution (1984 - 1989) 42

RAPM General Population Percentile by Age Group 45

RAPM General Population Percentile by Age Group -- Extended to 4 sigma 45

RAPM Norms for Various Occupation Groups -- mostly UK 46

RAPM Untimed Smooth Summary Norms for USA 46

GRE Percentiles for Filtered Population 49

WAIS-R Regression Equations for the Full Scale IQ 51

Standardization Sample for Stanford Binet 53

The purposes of the deliberations of the 1998/99 Prometheus Society Membership Committee were several. One purpose was to address concerns about the leakage of information over the Internet on tests accepted for qualification for entry to the Society. It was also to investigate the possibility of including a broader scope of tests of cognitive ability while maintaining the 99.997 percentile (1-in-30,000) of the general population Prometheus Society selection level criterion. Another purpose was to analyze the current entry criteria on accepted tests to determine whether the 1-in-30,000 criterion is being maintained by all. Some of these issues were identified by Kevin Langdon in "Admission Standards" (Gift of Fire, Issue 99, 7, September 1998).

As outlined by the chairman Fred Vaughan in "The Membership Committee and Its Charter" (Gift of Fire, Issue 100, 6, October 1998), it has been our objective to have a recommendation to the general membership of the Prometheus Society by the deadline for publication to Gift of Fire issue #102 (submission deadline January 9, 1999) with balloting to take place in issue #104. It has also been our intent from the outset to produce a report that will be available to members and nonmembers who wish to scrutinize the membership entry requirements of the Prometheus Society; we hope thereby to eliminate disputes -- or at least to provide data to make such debates more meaningful.

The academic literature on psychological testing or psychometrics is now huge. No concerted attempt was made to make a comprehensive review of this literature, but see, for example, the following recent works to get a flavor of this field: Benbow & Stanley (1996), Van der Linden (1996), Nunnally & Bernstein (1994), Murphy (1997), Janda (1998), Fischer & Molenaar (1995), Kline (1993 and 1998), Crocker & Algina (1986).

We do not claim that our results are indisputable nor that there are no flaws or oversights in the analyses presented here. We present this as a start in what must be a continuous process of maintaining the integrity of our entry criteria. This report attempts to address concerns such as that expressed by James Harbeck in his brief note entitled, "Questions Concerning the Membership

Committee", (Gift of Fire, Issue 83, March 1997). The membership requires more than a recommendation -- they require information in order to know whether to support that recommendation. We think we have provided that data.

The President and Membership Officer are constitutionally installed members of the Prometheus Society Membership Committee as described in section III of the Prometheus Society constitution duplicated in section III of this report below.

We appreciate our Membership Officer, Robert Dick's constructive participation which was unfortunately limited by serious medical problems. Robert has asked that we print the following statement.

"I have been a Constitutionally mandated member of the Membership Committee. In that capacity I have supplied member score data sanitized so the names cannot be identified. Due to personal illness, among other reasons, that is about all of my contribution. Accordingly I cannot claim for myself the honor of being an author of the Committee report. My hat is off to the many expert and dedicated members who deserve both the honor and the responsibility for the report."

The role and authority of the Membership Committee as well as the President and chairman in their respective capacities on this committee are defined in the constitution of the Prometheus Society as follows:

The committee performed its business primarily over the Internet using e-mail messages that were routed only to other members of the Membership Committee except as authorized specifically in writing by the chairman. (This was felt to be particularly important because we would be discussing topics that could compromise the tests accepted for qualification to the Society.) Web site files were also used but if they pertained to the Membership Committee exclusively, they were either password protected or their URLs were not disclosed outside of the committee except as authorized by the chairman. Individual Membership Committee members have interacted among themselves at their own discretion, but only information routed to all Membership Committee members was considered for inclusion in this final report including the recommendation to the general membership for balloting. Individual members or Membership Committee splinter groups defined by the chairman to perform specific tasks, have reported their findings for discussion by the entire committee. Discussions of specific problems and whether or not they were to be considered compromised based on answers circulated and the specifics of where such data is available typically involved only a subset of the committee.

Each step (agenda item) in the deliberation process was documented by the chairman or his designee using materials generated by the Membership Committee and the results were routed for comment and consensus. No item was closed out until every member of the Membership Committee had been given a reasonable opportunity to review and comment upon it. This required a 24-hour minimum per item to accommodate our world wide Membership Committee membership. Membership Committee members checked their e-mail regularly and responded to those items for which they had specific interest or concern. (They were encouraged to notify the chairman if they would be out of contact for more than 24 hours. An effort was made to keep consensual actions from occurring on weekends.) Requested delays prior to concluding a Membership Committee decision were honored without exception. Requests for delays were requested to be accompanied by specific rationale and/or the data that the requester wished the rest of the committee to consider.

Decisions identified as being made by the chairman (other than the appointments to the committee), have been consensus positions wherever possible. The chairman acted primarily as a focal point of that consensus to reduce chaos. Procedures were subject to modification as we went along but the procedures documented here were essentially the procedures that we followed throughout.

Specific positions argued and quotations of individuals during the deliberation of the Membership Committee will remain confidential. Detailed rationale for all recommendations of the committee are provided in this final report signed by all committee members. A pledge of confidentiality for the discussions in deliberation was a prerequisite for continued appointment to this committee. It was decided that a single consensus position would be incorporated into this report if such a consensus could be obtained. If more than a single individual shared a position counter to the consensus, that position is summarized in the report as well subject only to the desires of those sharing the position.

Intellectual rights to publication of material generated as a part of the deliberations of this committee belong to the individual or individuals who generated the material, but publication must be approved by the committee as expressed in writing by the chairman to assure the following: 1) All individuals who contributed to the material to be so published shall be cited if they so desire and 2) No data contained in the material to be published shall compromise Prometheus Society entry criteria.

Agreement to these operating conditions has been a prerequisite for continued appointment to this committee. Concurrence with these conditions is tacit by a member's not having notified the chairman of a wish to resign appointment.

The following recommendation of this membership Committee has been printed in issue #102 of the Gift of Fire (submittal deadline 9 January, 1999) which was mailed out together with a hardcopy of this report to all hardcopy members of the Prometheus Society and hardcopy subscribers of record to the Gift of Fire. On-line members and subscribers have been notified of availability of the report on-line at <http://prometheus.wwwh.com/subscribers/MCReport.html>.

5.1 Statement of Recommendation

We on the Membership Committee are proud to present to the Prometheus Society our proposal for revised entry requirements to the Society. We aver that it is our considered opinion that this recommendation, if adopted by the membership, will be in the best interest of this Society and its members. Our recommendation is as follows:

Entry into the Prometheus Society based on a Mega or Titan score shall no longer be allowed after the date of issuance of the issue of Gift of Fire in which acceptance of this recommendation is indicated to have been ratified by the membership. Anyone having secured a raw score of 36 on either of these tests dated before that date shall be entitled to rights and privileges of the Society.

Anyone with a score of 164 or greater on the LAIT scored before December 31, 1993 shall be entitled to rights and privileges of the Society.

Anyone with a score of 1560 on the "old" SAT (taken before April 1, 1995) shall be entitled to rights and privileges of the Society.

Anyone with a score of 1610 on the "old" GRE (taken before October 1, 1981) shall be entitled to rights and privileges of the Society.

Anyone with a score of 98 on the MAT shall be entitled to rights and privileges of the Society.

Anyone with a raw score of 88 on the Cattell Culture Fair III (A+B) obtained at an age of 16 years of age or older shall be entitled to rights and privileges of the Society.

Anyone with a score of 160 on the WAIS-R obtained at an age of 16 years or older shall be entitled to rights and privileges of the Society.

Anyone with a score of 21 on the Mega27 shall be entitled to rights and privileges of the Society, if a validated accompanying score on an accepted test for demonstrating a 1-in-1,000 cognitive ability according to that test is provided to the Membership Officer along with proof of the mega27 score.

And, for a trial period of one year:

5.2 Rejection of the recommendation

If our recommendation is rejected by a majority of voters, the Prometheus Society will retain the entry requirements established by vote in 1997.

We have accepted the following outstanding items that we recommend for further action.

6.1 Obtain written agreement with Ron Hoeflin on Mega27

Firm up agreement in principle with Ron Hoeflin on scoring procedures and application processing for the Mega27 test. Also obtain written specifications of how profile data is to be handled by Membership Officer.

6.2 Evaluate Titan test

We have agreement in principle with Ron Hoeflin to obtain data for 500 individuals who have taken the Titan test. We must perform analyses similar to those which gave rise to the Mega27 for the Titan to avoid compromised problems. Also solidify norming for the Titan.

6.3 Consider the relationship of age and intelligence

There are a couple aspects of IQ variations with age that must be considered in some depth with regard to our entry requirements:

1. whether to allow test results for individuals under 16 years of age and

2. whether to consider an age profile (particularly applicable to those over 30 years of age) for intelligence criteria.

6.4 WAIS subtest qualification possibilities

6.5 Evaluate results of one-year trial period of use of ThinkFast to qualify applicants for Prometheus.

After one year, the following data will be used to determine whether to retain the test permanently, extend the trial period or discontinue this test as an entry requirement to the Society:

1. numbers of applicants to the Society who use this ThinkfastTM test criterion,

2. accompanying scores on standard tests of applicants who use this ThinkfastTM test criterion,

3. additional statistics available on high scores of ThinkfastTM participants,

4. our increased understanding of ThinkfastTM as a chronometric/psychometric instrument.

6.6 Investigate tests in other languages, including translations of English tests.

The following are the dates and accomplishments that we had originally scheduled. We were considerably off-schedule from time-to-time but it did nonetheless draw us back to reality. We feel that we have accomplished the pressing tasks that were before us.

8.1 Operating Definition of Prometheus Society Entry Conditions

Entry criteria for membership in the Prometheus Society are based on verifiable claims of a particularly high level of intelligence. A 99.997 percentile or 1-in-30,000 of the general population has been maintained as the goal; the accuracy with which we have been able to meet that goal in the past and intend to maintain it in the future are discussed as a part of the analyses of this report.

With regard to the question, "What is the intelligence that should be assessed at this level?" we have been somewhat reticent to assert an answer. In other words we have waffled somewhat on whether, it is a fluid intelligence factor (spatial/abstract reasoning) or a crystallized intelligence factor (accumulated knowledge and verbal skills). A consensus of the Membership Committee believes it should be the former. Hedley St. John-Wilson gives the evidence for a general factor in his article, "The Scientific Evidence Behind 'General Intelligence' Tests" (Gift of Fire, Issue 95, January 1998). However, the Membership Committee is quite divided on whether the "fluid intelligence factor" is a single or many biologically-based capabilities. It is also divided on the ability of individual tests to effectively discriminate between a single general and a combination of many specific mental capabilities. The articles, "What is this thing called 'g' or Gee, what is this thing called?" (Gift of Fire, Issue 80, November 1996) and "What Intelligence is...isn't...is too!" (Gift of Fire, Issue 82, February 1997) by Robert Low, Ronald Penner's "Gee, Maybe There's More to 'g'"(Gift of Fire, Issue 82, February 1997) and "Discussion of the Central Limit Theorem as Applied Specifically to Overall Intelligence" (Gift of Fire, Issue 82, February 1997) by Fred Vaughan all address this debate. These conceptual and philosophical disputes involve medical, anatomical, psychological and genetic expertise which have not been adequately represented on our team. See for example, Fredrik Ullen's article, "The Multiple Biological Correlates of g", (Gift of Fire, Issue 100, October 1998), David Roscoes "Group IQ Tests" (Gift of Fire, Issue 81, January 1997) and Fred Britton's "Is There a Physical Substrate to Intelligence", (Gift of Fire, Issue 83, March, 1997).

We, therefore, have decided to restrict our assessment of testing capabilities to the statistical validity of accepted psychometric instruments to correlate well with other accepted instruments and to discriminate individuals at the 1-in-30,000 level. We recognize that individuals selected via different tests may differ in their thinking abilities accordingly, but each will have satisfied the ostensible requirement of being in the top 1-in-30,000 of the general population with respect to cognitive abilities measured by one of these tests. It is generally agreed that the general intelligence factor ("g") will influence performance across the spectrum of cognitive abilities measured by such tests and will result in at least a moderate g loading (~0.5 - 0.6) for an accepted test.

The Membership Committee is also in agreement that "1-in-30,000" rather than "4 sigma" is our target since the former claims nothing with regard to the distribution of the population as assessed by the test, restricting its emphasis to the rarity of individuals in this category.

The issue of age restrictions for entry to the Prometheus Society has been discussed and it has seemed reasonable to us at this time not to accept individuals under the age of 16 years, although we are somewhat split with regard to the age limit for scores that can be allowed. We think this issue should be addressed at a later time when there is more time to fully evaluate the data -- we have taken such an action item. Our current recommendation of the 16 years of age limitation derives in part from of our concern with regard to what might otherwise give rise to restrictions on subject matter in the journal. It is also related to concerns that too early testing has been shown in many cases to significantly overestimate intelligence. See for example Michael Colgate's article, "P's and Q's of Intelligence" (Gift of Fire, Issue 97, July 1998) in which he presents cogent arguments suggesting that another aspect of intelligence which he calls "precociousness" that applies exclusively to younger children may render rather unrealistic IQ scores on tests taken at a young age. Also Sare presents arguments and predictions that discount Stanford-Binet scores for younger individuals. (See <http://www.brain.com/bboard/read/iq-archive2/2351>.)

If the recommendations of this Society are approved, we will begin a new era with new members joining based on a diverse spectrum of psychometric instruments, but each with credentials establishing him or her at the 1-in-30,000 level of capabilities as measured by a particular instrument. Many of these tests (in fact most standardized tests of mental ability) make no claims for being able to discriminate intelligence beyond 150 IQ. Our acceptance is based on frequency data indicating that the rarity of 1-in-30,000 is attained independent of the particular intelligence claims made for that distinction. We have opted in all cases to base our recommendations on as solid a factual foundation in available data as possible and not on the claims of developers and/or distributors -- nor yet the detractors -- of these instruments.

It is of particular interest that in Joseph Matarazzo's book, Wechsler's Measurement and Appraisal of Adult Intelligence, (5th ed., 1972), he attributes lowered ceilings to intentional acts based on presumptions by the test developers themselves of a lack of utility of intelligence above the 150 IQ level.

If our recommendations are accepted, the Mega27 test (a subset of the Mega test defined by this Membership Committee for which the developer has agreed to provide scoring capabilities) will be the only existing tie to former testing methodologies and Prometheus Society entry qualification criteria. We also have an agreement in principle with the developer of the Titan test in which he has expressed willingness to provide data with which we may perform analyses similar to what has been done to obtain the Mega27 test. We have taken an action item to perform such analyses so that hopefully a version of the Titan test can be reinstated among our recommended tests. The elimination of formerly accepted tests has not been intentional in the sense of discrediting former methodologies and entry criteria but rather a requirement imposed by compromises that have occurred to these previously accepted tests. We are hopeful that we will be able to provide similar capabilities in the future.

8.2 Definition of the Scope of Membership Committee Evaluations

In its recommendation, the Membership Committee has acted to maintain the integrity of the Prometheus Society entry criteria and enable continued enrollment into the Society to anyone whose credentials can be verified as meeting those criteria.

The Prometheus Society will be forced to reevaluate the specifics of its entry criteria whenever new information emerges and is made available to the Society concerning any of the following:

1. New tests become available (or are presented to our attention) for which normative data is available to justify claims of their being able to effectively discriminate at the 1-in-30,000 level,
2. New normative data has become available for existing tests which supports discrimination at the 1-in-30,000 level,
3. An accepted test is retired for any reason,
4. The integrity of an accepted test is found to have been compromised either by having been incorrectly normed in the light of new evidence or answers to its problems having been too widely distributed.

A specific task before the Membership Committee was to determine whether any of the changes identified above have occurred with respect to accepted tests, which would necessitate entry criteria changes at this time. We think there have been and have acted to assure that we have maintained reasonable means for entry to the Society.

To warrant that a test or methodology satisfies membership criteria the Membership Committee has felt it appropriate to perform analyses to verify the following:

ACT

American School Intelligence Test-High School Battery

American School Intelligence Test-Primary Battery

Analysis of Learning Potential-Advanced I Battery

Analysis of Learning Potential-Advanced II Battery

Arthur Point Scale of Performance Test

BAS -- British Abilities Scale

Black Intelligence Test of Cultural Homogeneity (BITCH)

California Short-Form Test of Mental Maturity

Cattell Culture Fair Intelligence Test-Scale 2&3

Chicago Non-Verbal Examination

Cognitive Abilities Test Form 5 1993

Counter Intelligence Test-Chitlings

Detroit General Intelligence Exam-Form A

Full-Range Picture Vocabulary Test

GMA -- Graduate Management Assessment (UK)

Goodenough/Harris Drawing Test

GRE

Henmon/Nelson Test of Mental Ability

Henmon/Nelson Test of Mental Ability-College Level-Rev Ed

Hiskey/Nebraska Test of Learning Aptitude

Kuhlmann/Anderson Tests-8th Ed

Langdon Adult Intelligence Test (LAIT) Retired

Learning Efficiency Test-II (LET-II) 1992

Leiter International Performance Scale

Lorge/Thorndike Intelligence Tests

LSAT

MAT

MCAT

Mega Test

Oregon Academic Ranking Test

Otis/Lennon Mental Ability Test-Advanced Level

Peabody Picture Vocabulary Test 3rd Ed Form IIIA (PPVT-IIIA) 1997

Pintner/Cunningham Primary Test-Rev

Pressey Classification & Verifying Tests

PSR (Psychological Stimulus Response)

Quick Test

Raven Advanced Progressive Matrices -- Sets I & II

Ross Test of Higher Cognitive Processes

Slossen Full-Range Intelligence Test (S-FRIT) 1993

Slossen Intelligence Test (SIT-R)-Rev Ed 1990

SAT

SRA Pictorial Reasoning Test

SRA Primary Mental Abilities (PMA)

Standard Progressive Matrices

Stanford/Binet Intelligence Scale-4th Ed

Stanford Ohwaki/Kohs Block Design Intelligence Test for the Blind

System of Multicultural Pluralistic Assessment (SOMPA)

Test of Cognitive Skills 2nd Ed (TCS/2) 1992

Test of Nonverbal Intelligence 3rd Ed (TONI-3) 1997

ThinkFast (Chronometric)

Titan Test

Wechsler Adult Intelligence Scale-Rev (WAIS-R)

Wechsler Adult Intelligence Scale 3rd Ed (WAIS-III) 1997

Wide Range Intelligence & Personality Test (WRIPT)

Woodcock-Johnson Psycho-Educational Battery-Rev (WJ-R) 1989/90

8.3 Review of historical entry criteria

8.3.1 Assessment of Current Prometheus Membership Intelligence Credentials

The Prometheus Society was founded in 1982. It's initial constituency had all been members of the former Xenophon Society which had entry requirements of 1-in-10,000 of the general population, an IQ of about 160. Notwithstanding many of these initial members were qualified at the 1-in-30,000 level and beyond according to accepted psychometric instruments. The initial entry requirement, once Prometheus had been established, was set at the 1-in-30,000 level which was incorporated into the Prometheus Society constitution.

There are currently 67 members of the Prometheus Society that are in good standing. There are upwards of 150 to 200 who have been members at one time or another.

By checking our current roster and that which was first published in issue #2 of Gift of Fire (July 1984) shortly after the Society was formed, it has been determined that there are no more than 8 currently active members who could have been admitted under the Xenophon cut-off of 1-in-10,000. That assumes that no other Xenophon members who weren't active in July 1984 have since joined using their prior Xenophon membership as entry qualification. We believe that to be the case.

Within the constraint of 1-in-30,000, the specifics of membership criteria have changed over the years with various tests and acceptance levels having been used that reflected that requirement. However, according to the Membership Officer's records, the current Prometheus Society average IQ according to LAIT, Mega, and Titan test normings (using each test independently or using all the data) is about 167. This is what would be expected statistically for a society with a 1-in-30,000 cutoff.

Using data derived from the Membership Officer's records, the following further characterizations can be made: The average and median for the current and former members taking the LAIT in the 1978-79 time frame is the same as for the 1992-93 time frame (around 166-167). The average and median for the current and former members taking the Mega test in the 1984-85 time frame is about 1 point lower than for the members taking the Mega in the 1994-98 time frame (Mega average and median are around 37-38 in 1984-85, around 38-39 in 1994-98, and around 38-39 for 1998 alone). Differences over the years do not seem to be statistically significant. For the Mega test calculations, this result did not include scores below the current Prometheus Society cutoff of 36 and thus a conclusive result would require a comprehensive review of Dr. Hoeflin's scoring data for the respective years for which the average is a raw score of 35.5.

8.3.2 Review of compromise and erosion threats and discussion of the appropriate reactions

A major reason for the current Membership Committee's urgency is the concern with regard to rumors that there have been significant compromises to our entry criteria tests. This aspect of our deliberations has been a priority and we feel that we have obtained a good understanding of the threats and the actualities of compromises over the Internet and via other media. Our recommendations reflect that understanding.

Compromises to the Mega:

There have been several different types of answer distribution problems on the Mega test. The numbers of, and difficulty index associated with, problems that have been leaked in various categories from easiest to hardest are captured in the graphic below. Also shown are the means whereby each problem has been compromised.

Although the graphic contains only two compromised problems at the highest difficulty, some feel that at least three of the spatial/numerical problems have published solutions in Martin Gardner's books and/or other puzzle classics (references needed). Ron Hoeflin denies that they originated at that source.

The five hardest problems that have been leaked are the ones that would result in the most significant impact on the Prometheus Society. In order to get a score of 36 (a current entry criterion) on the Mega test by cheating one would need to have gotten these five correct plus 31 others. Someone who can solve 31 extremely difficult problems without cheating would probably be able to solve the 10 easiest leaked problems on his or her own without cheating. Thus, even with the leaked answers, the best someone would be able to do would be to turn a legitimate score of 31 into a with-cheating score of 36. By the 6th norming of the Mega Test, a score of 31 corresponds to an IQ of 158. So the impact on the Prometheus Society of leakage of these problems is not felt to be extremely significant at this time.

The existence of on-line integer sequence solvers compromises all integer sequence problems -- if they are not already solvable someone will solve them so we feel that we should include them in the list of compromised problems.

A point that may need some reconsideration in the future is that several of the problems in the Mega test appear to be easily solved/checked with computers. At least 7 of the non-verbal problems could be solved with a fairly simple computer program. A professional programmer might perhaps attack even more problems this way. This raises the question of whether we are unduly slanting our criteria to computer professionals. For a counter argument you might refer to "Sweetness and Stinging from the Honeycomb Series" (Gift of Fire, Issue 101, November/ December 1998).

We are faced with the question: Should the Mega test be retired in order to keep out dedicated cheaters with IQs in the 158 - 163 range? At a minimum several precautionary moves can and should be implemented to ameliorate the problem as, for example, eliminating test questions that are known to have been compromised. The Mega test is still a valuable instrument in that, largely because of Darryl Miyaguchi's web site, many new people are becoming familiar with the High IQ Societies and taking this high range test. The Prometheus Society continues to receive an appreciable number of new applications which may derive in part from this cause. This seems to us to offset some of the negative aspects associated with the possibility that the Society might thereby accept a few individuals at this time who are only marginally qualified for membership because they may have found a leak before we have. However, we must take care of the problems of which we are aware and address the possibility of continued erosion of the test with continued vigilant surveillance.

Notice that a certain amount of trust is involved even if the Mega test answers are not available on the Internet. In particular, there is no way to verify that a test-taker worked independently. Also, the "leaked" answers available on the Internet are not exactly easily available. One can not just use a search engine to search on "Mega test" to obtain the answers. The answers to the hardest five of the leaked problems are available separately on unrelated sites which would therefore require some ingenuity and persistence.

However, it is our consensus opinion at this point that we cannot warrant all 48 questions of the Mega test for qualification to the Prometheus Society. As you will see further on, we have defined a subset of questions which would constitute a test in its own right that we have shown to be able to discriminate at the 1-in-30,000 level. This test (the Mega27) eliminates all known compromises to the test as well as a few of the very simplest problems that may certainly be compromised in the near future and which have added little of value as we have demonstrated in discriminating at to the Prometheus Society's desired cutoff level and beyond.

The entire Mega test will probably need to be retired in the not too distant future. Alternative high range tests may be available before that time comes.

Compromises to the Titan:

The Titan is a newer test and appears on face value to be a more difficult test than the Mega which may have protected it somewhat from those individuals on the Internet who have concentrated on "cracking" the Mega.

However, the number sequence problems are compromised in the same way as for the Mega, and lacking the item data that we have on the Mega, we have been unable to come up with a method for estimating scores if we exclude the sequences. Therefore, unless and until we obtain norming data for the Titan, we feel that we must remove the Titan from our recommendation of approved tests for qualification for the time being.

Notice that we have obtained agreement in principle with Dr. Hoeflin whereby he will provide us with data for 500 examinees with which we can perform analyses similar to those that gave rise to the Mega27. We have accepted an action item to perform those analyses and report back to the membership on our conclusions.

8.3.3 Surveys of capabilities and comparisons of various segments of the population and psychometric instruments

There are a considerable number of summaries and reviews that have been published in High IQ journals and elsewhere which review relative ranges of coverage of different tests and expected intelligence of various segments of the population. However, these summaries are not all in agreement and typically do not include data at the level of our entry requirement. So we have used them primarily for orientation and guidance.

Greg Grove, psychometrician of the Triple Nine Society (TNS), published data that relates percentile rankings of various segments of the population, relating them to Mega test raw scores in his article, "IQ/Percentile Ready Reckoner" (VIDYA, Issue 177, July/August 1998). A problem with this review from the Membership Committee's perspective is that since it was prepared with TNS in mind, it only goes up to the 99.9th percentile.

There is also a survey of numbers of participants in various High IQ groups by percentiles presented by Guy Fogleman in "An Amateur Statistical Analysis of a Hi-IQ Society Membership Trend" (Gift of Fire, Issue 97, 16 - 17, July 1998).

Kjeld Hvatum provides a table of IQ percentiles versus scores on various psychometric instruments including the Mega in his "Letter to Ron Hoeflin" (In-Genius, Vol. 15, August 1990) that shows comparative raw scores at percentiles up to and beyond the Prometheus Society cutoff level. This table is provided below as reference only. It has not been validated by the Membership Committee and is not a part of our recommendation per se. But it represents of the kind of digested information that is available which has led us to investigate some of these tests in more depth and place less emphasis on others.

WAIS CLASSIFICATION,  %ile in the general population
descriptions,         |         standard deviation
High-IQ societies,    |         |    IQ SD-15 - WAIS, WISC
v = "here and down"   |         |    |   SD=16 - Binet, CTMM, Otis-Lennon
                      |         |    |   |
PROFOUND RETARD.---v  .13e-8    |    00 -07  IQ SD-23.7 - Cattell (Verbal)
SEVERE RETARD.-----v  .29e-4    |    25  20  |   SAT Verbal
MODERATE RETARD.---v  .0031     |    40  36  |   |   GRE Verbal
MILD RETARD.-------v  .13       |    55  52  |   |   |   Miller Analogies
BORDERLINE RETARD.-v 2.3        |    70  68  |   |   |   |   SAT Verbal+Math
DULL-NORMAL--------v 9.1        |    80  79  |   |   |   |   |   Mega Test
AVERAGE------------v25.0        |    90  89  |   |   |   |   |   |
general pop. ave.---50.0       0.00 100 100 100 340  |   |   |   1
high sch. grad ave.-60.0       0.25 104 104 106 370  |   |  790
                    70.0       0.53 108 108 112 410  |   |  860
BRIGHT-NORMAL------v75.0       0.68 110 111 116 430  |   |  910  2
                    80.0       0.83 112 113 120 450 420  |  940
college grad ave.---84.1       1.00 115 116 124 470 440 38  980  3
                    90.0       1.29 119 120 130 500 470 43 1040
SUPERIOR-----------v9l.O       1.33 120 121 132 510 480 44 1060  4
                    93.0       1.47 121 122 135 530 500 47 1100  5
Ph.D. & M.D. ave.---95.0       1.63 125 126 139 550 530 52 1150  6
                    97.0       1.87 128 130 145 580 580 60 1190  8
VERY SUPERIOR------v97.8       2.00 130 132 147 590 600 65 1220  9
Mensa,  Camelopard-v98.0       2.06 131 133 149 600 610 66 1230 10
Intertel, TOPS-----v99.0       2.33 135 137 155 640 670 74 1310 14
NMSQT Semifin.-----v99.5       2.57 139 141 161 670 710 81 1360 17
                    99.7       2.74 141 144 165 690 730 84 1390 19
                    99.8       2.88 143 146 168 710 740 86 1420 21
ISPE,TNS,Min,Cinci-v99.9       3.09 146 149 173 730 760 89 1450 24
                    99.95      3.29 149 153 178 750 780 91 1480 27
                    99.97      3.43 151 155 182 760 790 92 1500 28
                    99.98      3.54 153 157 184 770 800 93 1510 30
                    99.99      3.73 156 159 188 780     94 1530 32
                    99.995     3.90 158 162 192 790     95 1540 34
Prometheus,4 Sig.--v99.997     4.02 160 164 195 800     96 1550 36
                    99.998     4.10 162 166 197         97 1560 37
Geniuses of Dstng.-v99.999     4.27 164 168 201         98 1570 39
                    99.9995    4.42 166 171 205            1580 40
                    99.9997    4.53 168 172 207                 41
                    99.9998    4.61 169 174 209            1590 42
Mega, One-in-a-Mil-v99.9999    4.75 171 178 212            1600 43
                    99.99995   4.89 173 178 216
                    99.99997   5.00 175 180 218                 44
                    99.99998   5.07 176 181 220
                    99.99999   5.20 178 183 223                 45
                    99.999995  5.33 180 185 226
                    99.999997  5.42 181 187 228                 46
                    99.999998  5.50 182 188 230
                    99.999999  5.61 184 190 233                 47
                    99.9999995 5.73 186 192 236
                    99.9999997 5.82 187 193 238                 48
                    99.9999998 5.88 188 194 239
                    99.9999999 6.00 190 196 242

* Kjeld Hvatum's  "Letter to Ron Hoeflin" and Ron's response, In-Genius, # 15, August 1990

8.4 Review of norming analyses of currently accepted tests

In view of continued criticisms of tests that have been accepted for entry to the Prometheus Society, it has seemed prudent to review the norming analyses of these tests to assess whether in our view they warrant continued use for application to the Prometheus Society and to provide data for a more meaningful debate on related issues. We have attempted to understand the rationale for approaches and to determine its legitimacy to the best of our abilities. We have also presented the arguments that have been levied against these instruments.

We believe that we have been fair in our assessments.

8.4.1 Mega test

We feel that Ron Hoeflin's Mega test may represent the best one can reasonably expect in terms of establishing a credible 1-in-30,000 cutoff on a high-level test of mental performance abilities because of the dearth of available information on other tests at the high level of our cutoff criterion with which to norm and calibrate a high range test. A general statement that can be made about the Mega is that the predictive value of a fairly small number of Mega problems is quite amazing as can

be seen in a subsequent section of this report where the "short Form" of the Mega (the Mega27) is discussed. We have considered negatives that have been pointed out with regard to the Mega test over the years and have attempted to capture those criticisms in a separate section below. Notwithstanding such criticism, we have concluded that the setting of 1-in-30,000 cutoff at a score of 36 on the sixth norming of the Mega Test is quite credible. That is, of course if one could discount the possibility of compromised answers on the Internet and elsewhere as described in section 8.3.2 above. These conclusions are based on the following analyses.

Review of the Mega Test sixth norming:

The Mega Test sixth norming is based on a weighted average of several tests for which paired raw scores are available. The norming is, however, heavily biased towards the SAT since that test provides the largest number of score pairs for the norming. According to the sixth norming, a comparison of the average of the standard tests vs. the combination of standard tests plus SAT scores agree well to somewhat beyond the 1-in-30,000 cutoff with which we are particularly concerned. Notice, however, that in accepting Dr. Hoeflin's sixth norming, we feel that we should also accept SAT and other test scores used in the norming at the associated level for admissions to the Prometheus Society. To do otherwise would be inconsistent since SAT raw score data (in particular) was used explicitly to norm the Mega.

Test Equating: Methods and Practices, by Michael Kolen and Robert Brennan and Test Equating, edited by Paul Holland and Donald Rubin who are with Educational Testing Service (ETS) both discuss "Equipercentile Equating" under the general heading of common ways to find equivalent scores on two different tests. The meaning of this approach is obvious from the name. (It is worth noting that these references refer to "equipercentile equating" rather than equating based on equivalent standard deviations.) This is the technique we have sometimes referred to as "score pairing."

We have spent considerable time discussing the legitimacy of this method and believe the approach itself to be valid. Since it is the approach used by Dr. Hoeflin in norming his Mega test that was previously accepted for entry to the Society, it seemed essential that we understand the rationale for the method. (In general it seems more advisable for the committee to merely evaluate norming data rather than attempting to re-do it.) Bill McGaugh has conducted an independent study to be published in GoF Issue 102 as "(Bill we need a title)" describing the application of this methodology to athletic capabilities of world class decathlon participants in which he shows the method to work effectively in that arena as well. Some (even some of us) have considered this approach nontraditional and somewhat controversial as the "side-by-side" score-pairing by which it is sometimes referred, as though the technique were exclusively used by Hoeflin for establishing the 1-in-30,000 cutoff using the 220 SAT-Mega score pairs. See for example Roger Carlson's article, "The Mega Test" (Test Critiques, Volume VIII, 1991). Evidently, however, it is an accepted method used routinely by the ETS. The following is a plausibility-based argument that the committee used in understanding this equipercentile equating method :

If one assumes that raw scores on the Mega and the SAT are monotonically related to mental ability, i. e., that a higher raw score on either test correlates with higher mental ability, then there is some function z₁(n) that relates raw scores on the Mega to standard intelligence scores z and there is some function z₂(m) that relates raw scores on the SAT to standard scores z, where z = (IQ-100)/16. It is plausible to assume that the joint probability distribution of z₁ and z₂ is just the bivariate normal distribution p(z₁,z₂,r) for some correlation r. This function is symmetric in z₁ and z₂. Thus, for any random sample for which raw scores exist for both the SAT and Mega, if we have n scores with z₁ > 4, then we would expect n scores with z₂ > 4. These would not generally be the same n individuals in each case. Thus, if we know the 1-in-30,000 cutoff on the SAT (raw score=1560), and if there are N people in the sample of people taking both the SAT and the Mega scoring at this level or higher on the SAT, then counting down the highest N Mega scores from the sample would give a reasonable estimate of the 4-sigma cutoff on the Mega (raw score=36). Ron Hoeflin showed that, if you do this for several different cutoffs, then the resulting Mega normalization is linear over a range of scores including 36. This linearity feature seems to be standard on IQ tests over their range of applicability.

We are aware that there are difficulties in this argument (e.g., with respect to self-reporting of SAT scores, nonrandomness of sampling, small sample sizes, and mathematically allowed but "unphysical" test scores associated with ceiling effects). Roger Carlson has pointed out several of these problems in his review, "The Mega Test" appearing in Test Critiques, Volume VIII, 1991. We believe that the arguments could be tightened up in the future, but that the use of the data shown in the equipercentile equating plot does not raise any immediate "red flags" in these regards for determining the Prometheus Society cutoff score on the Mega. The correlation data does seem to reveal some reticence on the part of participants to claim SAT scores below 1150 which has probably reduced the correlation coefficient (r=0.495) that is shown by the trend line in figure 2 significantly and is very likely responsible for some of the nonlinearities in figure 2 especially the bending at the low end.

Figure 2: Mega vs. SAT Score Correlation Figure 3: Equipercentile Equating of Mega and SAT

In addition to correlations with the SAT raw scores, data is available from which correlations have been made against eight other intelligence tests. Plots of score pairs are provided for the LAIT, Cattell, CTMM and WAIS in figure 4. (The data for the figure are available at the URL http://www.eskimo.com/~miyaguch/megadata/megacorr2.html.) These correlations "r" and the number of raw score pairs (N) that they are based upon are included in the following table:

 

This table and the corresponding figure are considerably at variance with the table of "actual correlations" presented by Langdon in the article, "Mensa Tests and Other Standard Tests" (Gift of Fire, Issue 81, January 1997) in response to Greg Scott's article "For Acceptance of Mensa Supervised Tests" (Gift of Fire, Issue 99, September 1998) that cites the data above, which is available to the general public at: <http://www.eskimo.com/~miyaguch/megadata/megacorr.html>. The table above has been verified and corresponds to currently available data. If Langdon's data does indeed support the correlations as he indicates (and we have no basis for disagreeing) then it must be concluded at a minimum that there is considerable variation in such measurements with respect to the Mega. Interpretations of his table can not be made without inspecting the data (which is currently not available to us).

Some of these other tests for which correlations are available including the GRE (presumably on the earlier version for which raw scores sometimes exceeded 1600) and CTMM support equipercentile equating up to or near the 1-in-30,000 cutoff as shown in figures 5 and 6 below. The GRE score of 1610 would seem to be a comparative score to the Mega raw score of 36. This is quite compatible with statements to the effect that ETS had provided data indicating a score of 1620 corresponded to the 4-sigma level as reported by Paul Maxim in his article "Renorming Ron Hoeflin's Mega Test", (Gift of Fire, Issue 79, 8 - 12, October 1996.)

A rather amazing fact is that for the CTMM a 1-in-30,000 cutoff is indicated at a CTMM score of only 155, but of course there is insufficient data to confirm