User's Guide
Contents
1. Confidence for incorrect items High Medium Low Confid HIG 1 Over confident 2 High confidence 3 Highly Accurate In H Judgement Judgement Judgement correct items ME 4 Inappropriate 5 Moderate confidence 6 Moderately Accurate D confidence Judgement Judgement Moderate Misjudgement LO 7 Inappropriate 8 Inappropriate 9 Under confident W confidence High confidence Judgement Misjudgement Misjudgement V20150126 2010 2015 Team Focus Limited 18 These 9 patterns are described more fully below 1 Over confident Judgement Pattern These are people who have a very high confidence in their judgements even when they are wrong They do not appear to know when they have insufficient information or when they are making incorrect assumptions If this translates to everyday life it suggests that they may act with confidence even when they are unsure This could mean that they have a bold and entrepreneurial style When this is working well they may be full of confidence and initiative However this is most likely to work well when they are on familiar ground It raises the question as to whether they could sometimes be over confident especially in new situations Perhaps they need to learn to recognise those times when more information is required before they make a judgement If they do not learn to do this it would be worth exploring further the degree to which they see themselves as high risk takers2. and whether to others this is sometimes seen as reckless 2 High confidence Judgement Pattern These are people who have a high confidence in their judgements However they temper this to some degree when the situation is less clear cut at which times they still show a moderate level of confidence If this translates to everyday life it suggests that they recognise when the ground they are on is less certain but that this only affects their confidence to a moderate degree As a style this could mean that they have the confidence to take initiatives which is likely to work well when they are on familiar ground It raises the question as to whether they could sometimes be over confident especially in new situations Perhaps they need to learn to recognise those times when more information is required before they make a judgement If they do not learn to do this it would be worth exploring further the degree to which they see themselves as high risk takers and whether to others this is sometimes seen as reckless 3 Highly Accurate Judgement Pattern These are people who have been accurate in their judgements They appear to know when they have sufficient information and when they are justifiably unsure If this translates to everyday life it suggests that this awareness would help the person to recognise when they needed to inquire further before making a decision 4 Inappropriate confidence Moderate Misjudgement Pattern These are p
3. decision making linked to specialist knowledge held in the memory A feature of the Decision Analysis Test is the progressive build up of information which it could be argued would favour those with a better memory in as much as such a person would not need to keep checking information or refreshing their memory prior to solving a particular problem In many testing scenarios this is seen as a design problem but this is built in to the Decision Analysis test experience where more information is added during the assessment because it mimics a known attribute of those who show sound clinical reasoning namely that they possess and use a good memory Development of the Decision Analysis Test None of the above influences have defined the structure for dealing with thinking and decision making but all of them deal to some degree with two main dimensions complexity of information and ambiguity of rules or processes Complexity can be viewed in terms of the increasing amount of information coming from an increasing range of sources in an increasingly disorganised way Recognising remembering tracking and identifying become increasingly difficult V20150126 2010 2015 Team Focus Limited no Independent of the complexity is the processing of it which has been classified as follows 1 Knowledge and Analysis based on known information and tangible elements recognising classifying etc This means dealing with the What a
4. jobs The DAT is designed to fill some of this gap by simulating aspects of real life more closely Specifically it focuses less on logic and deduction and more on inference and judgements made on the basis of incomplete or ambiguous information V20150126 2010 2015 Team Focus Limited 5 The DAT also differs from traditional ability tests by not being defined in terms of the stimulus material presented i e verbal numerical abstract Instead it is defined in terms of the cognitive processes involved As a simulation the test is not based on any single theory although there are several sources which have influenced the design The most important are 1 Diagnosis and Fault Identification 2 Stratified Systems Thinking Elliott Jaques 1976 3 Managerial Decision Making which has made extensive use of Boolean Logic and Bayesian Inference to model the uncertainties of real word decision making 4 Models of medical or clinical reasoning Each one of these describes a hierarchy of analysis and decision making all of which overlap to some degree A brief description of each is given below Diagnosis and Fault Identification 1 Any system c f medical diagnosis mechanical production systems a factory an economy etc presents itself as having certain symptoms Symptoms are defined as departures from normal functioning which are of concern to one or other people who normally benefit from the normal functioning of the
5. missing In the DAT a confidence level was introduced to explore the relationship between confidence and judgement which in other areas is called self efficacy with the potential implications for an approach to risk It does this by identifying people who show an 1 an appropriate level of confidence high when they get the question right and low when they get the question wrong These are potentially better decision makers since they may recognise when they do not have the information or the understanding from which to proceed 2 an inappropriate level of confidence high when they get the question wrong and low when they get the question right These are potentially bad decision makers since they misjudge their information or level of understanding 3 a high overall level of confidence regardless of whether they are right or wrong These are potentially good decision makers until they venture into unfamiliar territory at which point they fail to recognise their inadequate grasp of the situation 4 a low overall level of confidence regardless of whether they are right or wrong These are potentially cautious decision makers who fail to take even calculated risks Potentially the difference between the confidence level on items right versus items wrong is a better measure of judgement in the real world than how good a score is on a test For analysis purposes we can identify 9 types of decision makers as illustrated below
6. system in question e g the person themselves in the medical context or people who benefit from a machine or a factory and so on For this purpose there must be some implicit concept of normal functioning or the range of functioning considered normal and evidence which agrees or conflicts with this concept There is therefore a need for data evidence and facts 2 The second stage is for a diagnosis to be made of the cause of the presumed malfunctioning This is in essence a guess at all the known system abnormalities which could conceivably give rise to the symptoms This requires patterns to be identified and links to be made V20150126 2010 2015 Team Focus Limited 26 3 The next stage involves differential diagnosis This is the area that clinicians view as being central to the art of diagnosis It involves setting specific tests for each of the candidate diagnoses in order to eliminate hopefully all but one The tests can be observational e g looking for a symptom or other expression of the inner malfunctioning of the system which has not yet been observed but which could be observed easily for example asking the patient a question checking a product etc Alternatively an invasive observation might take place i e making an observation which cannot be made without getting inside the person or machine Then at a more complex level the test would involve conducting an experimental observa
7. A teamfocus PROFILING FOR SUCCESS Decision Analysis Test DAT User s Guide CR Team Focus Limited T 44 0 1628 637338 Heritage House 13 Bridge Street E teamfocus teamfocus co uk teamfocus Maidenhead Berkshire SL6 8LR W wwwteamfocus co uk PROFILING FOR SUCCESS 2015 Profiling for Success www profilingforsuccess com No part of this material may be reproduced stored or transmitted in any form or by means electronic mechanical photocopying recording or otherwise without prior written permission of the publishers This material may not be photocopied even within the terms of the Copyright Licensing Agency Ltd The Profiling for Success series is published by Team Focus Limited Heritage House 13 Bridge Street Maidenhead Berkshire SL6 8LR England tel 44 0 1628 637338 V20150126 2010 2015 Team Focus Limited zo Contents Section One General introduction Section Two Rationale for the Decision Analysis Test Section Three DAT development Section Four DAT administration Section Five Conducting a review session Section Six Technical data Section Seven Normative data References Appendix 1 Product Synopsis Appendix 2 Sample reports Appendix 3 Online Price list V20150126 2010 2015 Team Focus Limited 7 V20150126 2010 2015 Team Focus Limited SAS Section One General introduction Profiling for Success PfS is the brand name for a whole suit
8. NTHESIS AND Box 4 APPLICATION P Box 3 If A is red B is blue C is R Extends the known yellow D intensifies E O by combining negates red and blue make C information making purple blue and yellow make E new connections If A is red and B is blue and green yellow and red make S and inferences and C intensifies the colour then orange then what is the S reaches logical what is AC difference between ACD and conclusions ADC D Answer deep bright red l Jacques Levels 1 2 Answer ADC has a redder M and3 hue than ACD EVALUATION AND JUDGEMENT Box 5 Box 6 Creates new O connections and If 1 means long 2 means N concepts deals If 1 means long 2 means water and 3 means snake with missing and water and 3 means snake what code do you need to ambiguous content what does communicate the idea of a fast and makes best 1 2 3 mean flowing river Hegements Answer a river Answer a code for speed Jacques Levels 1 V20150126 2010 2015 Team Focus Limited Relating Thinking to Decision Making As ambiguity increases it is harder to make sound decisions A person needs to identify the gaps and judge whether to fill the gap extrapolate try to identify a pattern and come to some conclusion A person who does not recognise the ambiguity or who jumps to conclusions on the basis of too little information without recognising they have too little information could blindly make decisions without realising what is unclear or
9. cess involves assessing the likelinood of occurrence or a range of syndromes in relation to a variety of candidate diagnoses Secondly there is actuarial information known as epidemiological information in a medical context V20150126 2010 2015 Team Focus Limited a7 ee This is to do with the known interrelationships between specific variables in the patient population at large for example the knowledge that X percentage of men in Britain who smoked 30 cigarettes a day will have a heart attack before a certain age Similar to epidemiological information is outcome information i e the knowledge that if men who have had a heart attack have stents implanted this will add Y years of added life expectancy Probabilistic information of this kind is used in the differential diagnosis process to establish both the likelihood of particular causes and the suitability of different remedial measures Stratified Systems Thinking This is a comprehensive theory of cognitive functioning and the aspect most relevant to the Decision Analysis test is the way the theory classifies thinking into different levels as follows Level 1 using simple rules anchored in tangible results or outputs Level 2 adapting rules to meet an objective Level 3 extrapolating rules and making new connections within know boundaries Level 4 using alternative approaches and evaluating according to known outcomes Level 5 making connections between previously unr
10. e of tests and questionnaires designed to bring new ideas to the field of psychometrics The Decision Analysis Test DAT is one such example Whilst psychometrics has a long history of assessing cognitive abilities using Verbal Numerical and Abstract reasoning tests we recognise that the capabilities of the human brain are far more diverse In fact in real life we are often faced with a vast amount of information which can be both complex incomplete and ambiguous and the DAT has been developed to explore and assess these additional capabilities 1 1 A Rationale for Ambiguous Problem Solving The Decision Analysis Test DAT is designed to measure areas of cognitive functioning that are important in real life Any analysis of what most managers and professionals have to do soon shows how they are subjected to vast amounts of information US figures suggesting up to 34GB of data every day Bohn and Short 2009 and that generally it is of variable quality Thus in real life people find that they have to make judgements based on incomplete ambiguous and sometimes contradictory information By way of contrast an analysis of the traditional ability test format shows that they generally present all the information needed in one place and the boundaries and rules are usually clear And whilst the research does show that these skills are important they do not assess the full range of analytic and thinking modes that are required for complex
11. elated material and hence creating new rules or areas of knowledge The upper levels of this approach to thinking match real life especially where it is usual for people to make decisions with incomplete information a common feature of medical diagnosis V20150126 2010 2015 Team Focus Limited 8 Managerial Decision Making The aspect of this vast subject that relates to the Decision Analysis Test is the way in which Bayesian inference has been used to model the uncertainties of real life decision making For example traditional Boolean logic deals with clear information which can be true or false or at least there is enough information so as to leave no room for uncertainty For example there can be clear evidence that there is oil on a person s driveway If the only place where oil is used is in the car then the person may conclude that the oil has come from the car or is in some way linked with the car However when making a decision about what to do there are many more uncertainties such as Why is it there Is it a cause for concern What should be done Can you afford to wait and see what happens next Bayesian Inference has been used to model such uncertainty by combining common sense knowledge and observational evidence Models of Clinical Reasoning The models that most influence the Decision Analysis Test are based on hypothetico deductive knowledge driven or script models that depend on logical
12. eople who are prone to make misjudgements They show very high confidence in their judgements when they are wrong but moderate confidence when they are right If this translates to everyday life it suggests that they may sometimes act with confidence inappropriately However this is a slightly unusual pattern and it would be worth exploring to what degree they understood the task they were given perhaps there are issues about understanding the purpose of the test familiarity with the language or whether they got the confidence scale the wrong way round If none of these apply it would be worth exploring further the degree to which they see themselves as being prone to misjudge situations and perhaps being over confident in the wrong situations V20150126 2010 2015 Team Focus Limited to 5 Moderate confidence Judgement Pattern These are people who have a moderate level confidence in their judgements Like most people they recognise that things are not always clear cut However they do not appear to differentiate between situations where they are making accurate judgements and others where they are not If this translates to everyday life it suggests that they make decisions based on the information they have without any particular tendency towards self doubt or over confidence As a style this could mean that they have the confidence to take initiatives or to seek further information to the same degree as most people Perhaps the
13. g situations 8 Inappropriate confidence Misjudgement Pattern These are people who are prone to making misjudgements They show moderate confidence in their judgements when they are wrong but low confidence when they are right If this translates to everyday life it suggests that they may not always act with appropriate confidence However this is an unusual pattern and it would be worth exploring to what degree they understood the task they were given perhaps there are issues about understanding the purpose of the test familiarity with the language or whether they got the confidence scale the wrong way round If none of these apply it would be worth exploring further the degree to which they see themselves as being prone to misjudge situations 9 Under confident Judgement Pattern These are people who have a very low confidence in their judgements They appear to doubt the information they have and the judgements they make If this translates to everyday life it suggests that they may either act with very little confidence or even fail to act at all Perhaps they are extremely risk averse This may be appropriate in some situations but they may benefit from learning to recognise situations where they are capable of making sound judgements and to learn to act with greater confidence V20150126 2010 2015 Team Focus Limited SAN References Bohn R E and Short J E 2009 How Much Information Report on American Consumers Calif
14. nd applying logic and rules 2 Synthesis and Application based on recognising patterns of information making meaningful links generating hypotheses that fit the evidence etc This means dealing with the How and recognising underlying causes and problems 3 Evaluation and Judgement based on recognising the incompleteness of the information and the potential ambiguities but nevertheless recognising probabilities and making best judgements By extracting these elements we have produced an item writing framework which has been summarised in the table below There are six boxes in this framework and we have created a simple illustration in each box to help clarify what is required to cover the framework see next page V20150126 2010 2015 Team Focus Limited 10 Overview of Decision Analysis Test Item Writing Structure sps INFORMATION COMPLEXITY DIMENSION SIMPLE COMPLEX 1 2 pages limited sources 2 4 pages Multi source Content is tangible or Content is conceptual or concrete abstract ANALYSIS AND Box 2 KNOWLEDGE Box 1 A grows at a rate of 1 cm a Extracts clear year information follows A is taller than B who is taller B grows at a rate of 1 2 cm a known rules and than C year M reaches logical C grows at a rate of 2 cms a conclusions Who is the tallest year T Jacques Levels 1 Answer A If Ais 90 cms in 1961 and B is A and2 93 cms in 1962 when will they L SYS
15. ornia University of San Diego Jaques E 1976 A General Theory of Bureaucracy London Heinemann Jacques E and Kathryn Cason 1994 Human Capability a study of individual Potential and its application Gower Robertson and Kandola R S 1982 Work Sample Tests Validity Adverse Impact and Employee Reactions Journal of Occupational Psychology 55 171 183 V20150126 2010 2015 Team Focus Limited 15 We hope you have enjoyed exploring this free introductory version of the Decision Analysis test DAT User s Guide We would be delighted to get your feedback and to discuss your thoughts and reactions In addition we hope that you have got a flavour of what the DAT is all about We realise that not all the features that make this approach innovative and practical have been explained but you will get some idea by referring to the Table of Contents and the full version will give you more detail If you are interested in the complete version normally provided to our affiliated partners and clients as part of our training courses or in discussing your interest further we welcome you to get in touch by filling in the V20150126 2010 2015 Team Focus Limited 46
16. tion in which something was done to the system and the output recorded For example giving a particular medication to see if it reduced blood pressure might confirm one of the suspected causes of an illness or putting oil in a particular part of the machine may fix the problem Other tests might involve doing an analysis of a patient s blood or checking temperature variations in a machine In a piece of electrical equipment one might test voltages at particular points or data output at some node This means working with levels of ambiguity hypothesising testing and or identifying what is missing before a diagnosis can be made 4 The next stage though not clearly differentiable from the former is to attempt remedial measures and see if they cure the problem This usually only happens after the differential diagnosis process has reduced the candidate diagnoses down to only one or two with a particularly high probability being assigned to one If the attempted remedy works this gives final confirmation more or less to the selected diagnosis There are one or two different types of evidence and information used throughout the above steps The first has to do with symptom clusters or fault clusters in the case of machines i e the idea that a given underlying cause disorder is associated with a syndrome So if a person shows all the symptoms in a known syndrome the probability of him suffering from the disorder in question is high The pro
17. y need to learn to question a little more and to recognise those times when more information is required before they make a judgement 6 Moderately Accurate Judgement Pattern These are people who have been accurate in their judgements Perhaps they could develop greater confidence in the judgements they do make but their confidence is in the right direction higher when they are right and lower when they are wrong They appear to know when they have sufficient information and when they are justifiably unsure If this translates to everyday life it suggests that this awareness would help the person to recognise when they needed to inquire further before making a decision 7 Inappropriate confidence High Misjudgement Pattern These are people who are prone to make misjudgements They show very high confidence in their judgements when they are wrong but low confidence when they are right If this translates to everyday life it suggests that they may act with confidence inappropriately However this is an unusual pattern and it would be worth exploring to what degree they understood the task they were given perhaps there are issues about understanding the purpose of the test familiarity with the language or whether they got the confidence scale the wrong way round If none of these apply it would be worth exploring further the degree to which they see themselves as being prone to misjudge situations and perhaps being over confident in the wron
Download Pdf Manuals
Related Search