Assessment Center FAQs
Contents
1. 2 0 0 neti 7 How do score PROMIS instruments ccc entered 7 Have the PROMIS instruments been validated cece eee ened 8 Ls PROMI S in the public domain cycles A ea sete id vi ed 8 Who owns data from a PROMIS instrument 0 0 cece eens 8 What is the difference between short fOrmS c cece ttrt nunn n ntir nn Ennn EEE EEEE EEEE EAEAN EEEE 9 Last Updated 10 20 2014 1 Is one short form better than the Other ccccccccceeceeeee eects eeeeeeeeseeeeeeeeeeeeeeeteeeeaeeeeaetengeanen yes 9 Where can learn more about a given domain 0 ccc eens 9 When should use the pediatric or adult instruments for young adults cece cece cece eerren 9 What is an adequate response rate in health related quality of life research ccceeeee eee eee 10 Does PROMIS have proxy M ASUIES cece eee nner teens 10 Are PROMIS instruments available in languages other than EngliSh cccceeeeeeene eens eaten ees 10 What is the difference between versions of PROMIS instruments cceeeeeee eee eee ee teens ea eaee 10 What are the licensing details for using PROMIS it MS ccceee cece eee eee e eee eee ae neta een ea ees 10 What is the minimum change that represents a clinically meaningful difference ccc 11 Have any meetings been held with regulatory bodies FDA and scientific advisors csee eens 11 What is the difference between t2. a participant to type in text due to their lack of keyboard Additionally if the registration fields are being utilized be sure to pilot test how those screens appear and function on a touchscreen as they utilize a different template multiple items per screen How long does it take to complete an instrument The length of time it takes a participant to complete an instrument during data collection depends of factors such as number of items functional ability of the participant type of administration mode etc For example individuals with motor impairments or low literacy will most likely require more time to complete an assessment In general when a self or proxy reported instrument is administered on a computer with one item appearing per screen it is estimated that it will take approximately10 second to complete each item More complex items e g two items on a screen longer text Toolbox item involving card sorting will usually require more time per item Assessment Center includes a item template Auto Advance which enables the system to navigate to the next item immediately after a response is given for the previous item Utilizing this template will reduce the amount of time required to complete an assessment Last Updated 10 20 2014 6 I was previously able to see my accrual numbers on the Overview page within the Administration tab but now I cannot Why At this time the accrual dashboard is able to handle calcu
3. assessmentcenter net enter your user ID and click Forgot Password in the upper right corner This will email you your password If your email address has changed or you do not remember your user ID please contact help assessmentcenter net What are the minimum hardware operating system browser software needs for conducting a study in Assessment Center Operating System Windows XP or Higher Windows Vista or Higher Screen resolution RA 1024 X 768 1024 X 768 or higher Internet Browser Participant 800 X 600 Mozilla Firefox 4 0 Mozilla Firefox 4 0 Connection Speed DSL Broadband 750 Broadband 3 Mbps down Kbps down and 128Kbps up and 384Kbps up or higher or higher Misc Software jPad is only supported for participant administration only A researcher cannot currently set up and preview a study on an iPad My colleague has registered in Assessment Center and want to add her to my study team but do not see her on the User List Why To complete registration a user must log into Assessment Center at least once If this step is not completed they will not appear in the User List on the Study Team page Do I have to use a PROMIS instrument if want to use Assessment Center No You may use Assessment Center to create and administer your own instruments exclusively Do have to use Assessment Center if want to use a PROMIS instrument No You are welcome to use a PROMIS measure in other a
4. participant data From the Administration tab s default page Study Overview enter the participant s login and click Find Create Login Navigate to the Participant Data page Click on the appropriate instrument name on the left navigation pane Complete data entry Note that if you write over data the original data will not be stored anywhere I have several studies in Assessment Center and I enrolled a participant into a wrong study Can participant responses be transferred from one study to another No we cannot transfer a participant s responses from one study to another study We would suggest exporting the data from Study 1 and manually entering it in Study 2 Then in Study 1 change the participant s status to Consent Test When it is time for data analysis select only those participants whose consent status equals 1 Yes Can I change items after have launched my study To maintain consistency among items and instruments you may not edit items after launching launched my study but forgot the URL can I retrieve it On the Launch page of your study there will be a hyperlink to your study URL Additionally you can view the URL on the Basic Set up page Can participants use a touchscreen to complete an assessment Assessment Center is compatible with touchscreen computers The mouse adapts to your finger instead of the mouse hardware input device However touchscreen computers are not ideal for items that require
5. Assessment Center FAQs General Assessment Conter 4 unigesdt eee ie tone ein yk See ee ieee ote a ca 3 What iis Assessment Center ic enean cen Ea Aa A AAA al aaa shoe ed haya ie aes 3 Why should use Assessment Center cc nna 3 Who is using Assessment Center 0 cece nee eee nee nents 3 How long will Assessment Center be around cece neta 3 What security measures are in place for online data CAPtUre cccceceeee cece ee eeeeeaeeateneeeeeneeateas 3 Who has access and who owns data collected in Assessment Center cccceseeeee eset eeeeeeeeeeee eae 3 How do register in Assessment Center ccc cee eee eee eee enna eee eee e nate nanan etna 4 What will you do with the email address provide when register in Assessment Center 4 How do retrieve a lost PASSWOI urrunt rrur rrer ErSEEEENEEERSEERSEEERSENE REENEN 4 What are the minimum hardware operating system browser software Needs ccceceeeee nent ea ees 4 Why do not see my colleague on the User LiSt 0c cece nena 4 Do have to use a PROMIS instrument if want to use Assessment Center cceceeeeeee erent eed 4 Do have to use Assessment Center if want to use a PROMIS instrument cceeeeeee eee ee ee 4 Can combine PROMIS with other instruments cece eee eee eee teeter teeters 5 My colleague is using Assessment Center Where can find his her instruMents
6. RT models What statistical information is required to administer a custom CAT To administer a custom CAT items must have calibrations i e slope thresholds on the Item Statistics page and the instrument must have CAT parameters to access click on Customize hyperlink then click on IRT parameters button How do CATs compare to fixed instruments e g short forms CAT would be expected to produce a more precise measurement with fewer items This is the main justification for using CAT instead of short forms Another advantage is that CAT produces better measurement at the ends of the continuum i e the extremes very high or very low of your continuum Typically short forms do best measuring the middle of the continuum e g those with medium levels of fatigue rather than extreme fatigue or extreme lack of fatigue Should I use a CAT instrument or a short form As the research has shown thus far CAT is usually better and never worse than the same length short form However if the groups to be compared are large e g more than 75 or so participants in a group short forms especially the longer 8 item ones will almost always be as good as CAT What are the disadvantages of CAT and reasons not to adopt it There may be some resistance to the idea that different participants are taking different items There is even more resistance to the fact that individual participants will take different items across time points M
7. Stone A Rothrock N Reeve B Yount S Amtmann D Bode R Buysse D J Choi S W Cook K F DeVellis R DeWalt D Fries J F Gershon R Hahn E Pilkonis P Revicki D Rose M Weinfurt K amp Hays R D on behalf of the PROMIS Cooperative Group 2010 Initial item banks and first wave testing of the Patient Reported Outcomes Measurement Information System PROMIS network 2005 2008 Journal of Clinical Epidemiology 63 11 1179 94 2 Liu H H Cella D Gershon R Shen J Morales L S Riley W amp Hays R D 2010 Representativeness of the PROMIS Internet panel Journal of Clinical Epidemiology 63 11 1169 78 3 Rothrock N E Hays R D Spritzer K Yount S E Riley W and Cella D 2010 Relative to the general US population chronic diseases are associated with poorer health related quality of life as measured by the Patient Reported Outcomes Measurement Information System PROMIS Journal of Clinical Epidemiology 63 11 1195 1204 Finally the www nihpromis org website hosts a current list of PROMIS publications that can address an instrument s validity for a specific PROMIS instrument within a specific population for a specific use Is PROMIS in the public domain PROMIS is available to you and all for use without charge PROMIS is not however in the public domain PROMIS was supported by grants to institutions and while all grant terms differ slightly from one anot
8. Study Preview function to see the full assessment What item formatting options are available in Assessment Center When you create an item you have the ability to underline or italicize text By default all item text uses bold font To add underlining use the code lt U gt before the text you would like underlined and lt U gt at the end Similarly to italicize use the code lt I gt before and lt I gt after the desired text For example an item entered like this on the Item Detail screen To what degree did you have trouble lt U gt starting lt U gt things because of your fatigue would appear like this on the data collection screen as To what degree did you have trouble starting things because of your fatigue You may also add a line break to appear on data collction screens or add a hyperlink to an item If you want to force a line break you can include the code lt BR gt to force text to start on a new line For example Context In the past 7 days Stem have been bothered by lt BR gt a lack of energy This would appear on the data collection screen as In the past 7 days have been bothered by a lack of energy To add a hyperlink first enter the code lt a href followed by the website s URL in single quotation marks Please note you will need to include the entire URL including the hypertext transfer protocol http or https where applicable the colon and the forward slashes Next enter a left an
9. a PROMIS instrument to another instrument that produces score s for similar domains e g CES D measures depression the relevant short form from the Profile here depression is a good choice The Global scale provides a general understanding of overall functioning If a respondent has a low score for physical health it isn t clear if this is fueled by poor physical functioning high fatigue or intrusive pain However if you need a brief fast summary providing information about how a person is doing overall the Global scale is a good choice Last Updated 10 20 2014 11 Are cut score available for PROMIS instruments To translate cut scores from other instruments to the PROMIS metric you may use PROsetta Stone linking tables PROsetta methodology places two instruments on the same metric allowing the creation of a cross walk table which matches each possible score on a legacy instrument to a T score of the comparable PROMIS instrument Linking tables along with a full report may be found on the PROsetta website www prosettastone org What is the PROMI S HAQ The HAQ Health Assessment Questionnaire in improved form was incorporated into the PROMIS Physical Function item bank The PROMIS Physical Function SF 20a is the successor of the PROMIS HAQ so the PROMIS Physical Function SF 20a is recommended to be used in instances where the PROMIS HAQ or HAQ DI would have been used except in studies in progress where baseline meas
10. ar instruments measuring abilities attitudes or other variables IRT is an array of mathematical models that describe how an examinee responds to an item probabilistically as a function of the trait level of the examinee and the characteristics of the item e g difficulty and discriminating power In IRT the examinee s score is defined on the latent unobservable trait rather than on the test as is the case in classical test theory Therefore the IRT trait score is considered independent of the particular set of items whereas the true score in classical test theory is bound to the particular set of items What is a computerized adaptive test CAT Computerized adaptive testing CAT is a form of computer based testing that adapts to the examinee s trait level For this reason it has also been called tailored testing Questions are selected based on a respondent s previous answers For example if an examinee indicates having difficulty climbing a flight a stairs the next selected question will be appropriate for someone at that level of Last Updated 10 20 2014 12 functioning The next question may be about climbing up a single step or curb If someone can climb a flight of stairs with ease the next question may be about climbing several flights of stairs Can I add CAT parameters for Partial Credit model Right now Assessment Center only supports a Graded Response Model Future releases of Assessment Center will enable other I
11. brated independent of version 1 0 and is a larger 44 items superior item bank Details on the specific differences between instrument versions can be found in scoring manuals available on the Assessment Center homepage What are the licensing details for using PROMIS items There are currently no licensing fees for PROMIS See the PROMIS Terms and Conditions available on the Assessment Center homepage www assessmentcenter net TandC aspx Last Updated 10 20 2014 10 What is the minimum change on a PROMIS instrument that represents a clinically meaningful difference This question is related to an area of active research in the PROMIS network namely the determination of the minimally important difference or MID for a PROMIS instrument A manuscript in the Journal of Clinical Epidemiology outlines the process for MIDs for PROMIS measures and estimates the MIDs for six PROMIS Cancer scales Yost K J Eton D T Garcia S F amp Cella D 2011 Minimally important differences were estimated for six PROMIS Cancer scales in advanced stage cancer patients J ournal of Clinical Epidemiology 64 5 507 16 As described in that manuscript the MID is a tool to enhance the interpretability of patient reported outcomes and is often defined as the the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate in the absence of troublesome side effects and excessive co
12. cceceeeeee ees 5 cannot find an instrument created in another Assessment Center study Why cceeeeeeeee es 5 Can review how an item looks for participants cece eee eterna 5 Why do I see response scores when preview My study 0c cee eee eects 6 How do remove participant PINs that use in Preview Study cceeeee cece ee eee eee eee eee ed 6 How do update a participant s reSPONSE ccc neat 6 Can participant responses be transferred from one study to another ccccceceseeeseeeeeeeeaeeateneeees 6 Can change items after have launched my study cc cceee cece e eee eee eee eee ee teeters 6 launched my study but forgot the URL can retrieve it 0 6 Can participants use a touchscreen to complete an assessment cecceeeee ects ee eee eat eeteeeeaeeatenennes 6 How long does it take to complete an instruMeNnt 0 cece eee tied 6 Why am not able to see my accrual numbers on the Overview page cccceeeee eee eeee eee ee eaeae 7 How can find a participant s registration c eee eee ened 7 How can request download data nein ne raed 7 received a message Request timed out when requested a data export why cc cece ees 7 PROMIS lNSERGIMGNES ya Ee dadbe sens Ea OE gadanemsie eed deena aniaes 7 Where can I find a PROMIS instrument 0c eerie 7 What do the PROMIS instruments look like
13. cores are also available in an individual report The PROMIS CAT Questionnaire Summary Report is a single time point report which includes a participant s standard score along with information on how that score compares with other people in the general population If age and gender were collected during registration the participant s score is compared to other individuals in his her age group and of his her gender The second portion of the report contains a graph displaying Last Updated 10 20 2014 13 scores with standard error bars You may complete a demonstration of CAT and receive a sample report here http www nihpromis org software demonstration How should one determine whether to administer a long vs short CAT It is known that longer tests produce more precise estimates However there are diminishing returns after 10 or so items In other words adding more items will only produce small benefits The decision of how many items to include in a CAT instrument depends on the goals of the assessment and concerns regarding response burden There are no rules for what is best in all situations What are robust approaches for CAT item selection Most algorithms select an item based on participant s provisional theta trait level Every time an item is responded to the CAT algorithm updates its estimate of theta Each time the estimate is updated the algorithm goes into the bank and selects the item that provides the mo
14. create study specific websites for capturing participant data securely Studies can include measures within the Assessment Center library as well as custom instruments created or entered by the researcher PROMIS instruments short forms CATs profiles are a central feature of the instrument library within Assessment Center Any PROMIS measure can be downloaded for administration on paper or be included in an online study Detailed statistical information about PROMIS items and instruments is available for review Assessment Center enables customization of item or instruments e g format randomization skip patterns real time scoring of CATs storage of protected health information in a separate secure database automated accrual reports real time data export graphing of select individual CAT or Profile scores and ability to capture endorsement of online consent forms among many other features Why should I use Assessment Center Assessment Center is unique in online survey administration tools as it enables the administration of computerized adaptive tests CATs It also enables easy access to PROMIS instruments including CATs short forms and profiles It allows secure collection of protected health information as well Who is using Assessment Center Assessment Center is a publicly available application Anyone with access to the internet who has registered is free to use Assessment Center How long will Assessment Center be aro
15. dministration platforms Last Updated 10 20 2014 4 Can I combine PROMIS with other instruments Yes Within Assessment Center you can opt to include a PROMIS measure as well as another existing measure or draft your own items My colleague is using Assessment Center Where can I find his her instruments If your colleague would like to share an instrument with you ask to be added to his her study team Then on the Add Instruments page select your colleague s study name to view all instruments that were created in that study You can add those instruments to your own study If you would like to make your instrument available to any Assessment Center user contact help assessmentcenter net I want to add an instrument I created in Assessment Center to another study I have created but I cannot find it when I do a search An instrument must have the status of Administered or Public in order to be added to another study Status is automatically updated to Administered once a study is launched Alternatively an instrument status can manually be changed on the Instrument Properties page Can I review how an item looks for participants Templates indicate how an item will appear on the participant s screen You can view images of these templates in the Assessment Center User Manual You can also select a template for a given instrument and utilize the Preview hyperlink to see how it will look to a participant You can also use the
16. gle bracket then retype the URL without the single quotation marks Finally enter the code lt A gt As the syntax of this code is more complex than others presented it is important to preview the item to ensure if appears as expected For example Stem For more information please access the PROMIS website at lt A href http nihpromis org gt http nihpromis org lt A gt This would appear on the participant s screen as For more information please access the PROMIS website at http nihpromis org Last Updated 10 20 2014 5 Why do I see response scores when I preview my study Under Preview tab Preview Study item response scores is enabled only on Preview but not after a study has been launched You can select No if you do not want to view the scores on Preview This is meant to help you check accurate response scores for custom instruments How do I remove participant PI Ns that I use in Preview Study so that they are not included on my data export Go to Administration tab of your study enter the login PIN click on find create login button and then change the Consent to Test The Consent field in export will show Consent 4 You should select only Consent 1 participants prior to analyzing data Codes are shown in the Data Dictionary A participant just told me that s he changed his her mind and wants me to update his response How do I do that Team members with the role of Data Entry Administrator can enter or change
17. he PROMIS Global scale and a PROMIS profile instrument 11 Are cut score available for PROMIS instruments cceceee cette eee eee ee eee eens 12 Whatiissthe PROMIS HAQ iorno aaea ee Sethu digitata sade vai en Ab aealnw bata yee ae Tay Gov ine AE AR 12 PROMUS NGEWORR iSro oiea os wet ery occ erie tee Sas oe eaves een eee es ete 12 How can get started learning about PROMIS ccc cece cece nee 12 How can I connect with the PROMIS effort cece cece eee eee eee eens 12 Psychometric Issues amp Computerized Adaptive Testing CAT ccecce 12 What is item response theory IRT sssssssssssssrsrrenerrrrrssssrrrrt EEE EEE EE EE EEE ea 12 What is a computerized adaptive test CAT ccicc cece rE EEE EE EEE EE Ener 12 Can add CAT parameters for Partial Credit MOdEI cccecceeeeee ects eee eet ea tent eee ea teateateaennneategs 13 What statistical information is required to administer a custom CAT 0 cece cece eect ee eee eta ea eee 13 How do CATs compare to fixed instruments e g Short fOrmS cceee eect eee eee ee eee teens 13 Should use a CAT instrument or a Short fOrM 0 nerd 13 What are the disadvantages of CAT and reasons not to adopt it cee eeeeeee eee ee ee eee eaten ees 13 How does CAT handle trial items which need examination cccceeee cece teen ee eee eee eee ata 13 What types of instant CAT score reports exist in Assessment Cen
18. her they all make it clear that grant as opposed to contract work products belong to the institutions and their investigators not the US government or the public at large The US Department of Health and Human Services has the right to freely use what it has sponsored for its purposes but the institutional and investigator copyrights are consolidated by the PROMIS Health Organization From the beginning the goal of PROMIS was to make these tools as widely and freely available as possible In order to maintain the integrity and quality of the item banks item wordings translations and their calibrations we do maintain copyright protection on behalf of all who have contributed to the project Who owns data from a PROMIS instrument The study team owns all data collected using PROMIS measures The PROMIS network makes no claim to any data collected using PROMIS measures There is a statement in the PROMIS Terms Last Updated 10 20 2014 8 and Conditions that encourages researchers to provide a summary report to allow the PROMIS network to know about how the measures are being used in various study paradigms Users of PROMIS tools are strongly encouraged to submit a brief report including sample demographic information clinical data sufficient to define the sample without indicating treatment response data and PROMIS score distributions e g baseline mean and standard deviations or change scores blinded to treatment information This br
19. ief report should be submitted to help assessmentcenter net for internal review None of this submitted information will be published without the written consent and participation of the submitter What is the difference between short forms You will notice that there are several versions of many of the PROMIS short forms e g the original 1 0 short form as well as the 4a 6a and 8a versions The original short forms were constructed by the domain teams The goal was to identify 6 8 items that represented the range of the trait and also represented the content of the item bank Domain experts reviewed short forms to give input on the relevance of each item Each domain group worked independently and those original short forms are 6 10 items long Psychometric properties and clinical input were both used and likely varied in importance across domains Later we developed the PROMIS Profile instruments This is a collection of short forms for 7 PROMIS domains PROMIS constructed high information PROMIS short forms for seven domains Anxiety Depression Sleep Disturbance Fatigue Pain Interference Impact Physical Function and Satisfaction with Participation in Social Roles For each domain 4 item 6 item and 8 items have been selected so that the items are nested overlap e g the 8 item form is the 6 item form plus two additional items The selection of items was in part based on two sets of rankings of items within each domain u
20. ious psychometric studies are undertaken and their results serve as character witnesses that reveal the level and nature of a measures usefulness in different populations and for different purposes With this introduction and clarification it would be valuable to rephrase the PROMIS validation question and ask What evidence is there for the validity of the PROMIS instruments To reiterate the validation of an instrument is an ongoing process that is never completed However substantial qualitative and quantitative evidence has been gathered supports the validity of the PROMIS instruments Below we describe some of this evidence We believe that validity starts at the beginning We share the view that content validity is built into a test from the outset through the choice of appropriate items Anastasi A 1988 Psychological Testing New York Macmillan Publishing Company p 122 127 The content related validity of PROMIS instruments began in our use of patient interviews and review by expert review panels Details about these activities can be found in DeWalt D Rothrock N Yount S Stone A A amp on behalf of the PROMIS cooperative group 2007 Evaluation of item candidates the PROMIS qualitative item review Medical Care 45 5 S 12 21 Additional information about the validity of PROMIS instruments can be found in a triplet of articles published in the Journal of Clinical Epidemiology 1 Cella D Riley W
21. lations for a finite amount of data If you have a large study the accrual dashboard may be overwhelmed You can export your data using the export functions on the Administration tab Contact help assessmentcenter net for assistance I did not collect participants names and I don t know a user s login How can I find his or her registration The Registration Data Export will include all participants with their logins Without collecting name it may be difficult to identify who is who in your study All data is saved with the time and date it was created That may help you identify a given participant The download buttons are grayed out under Administration tab How can request download data Grant yourself the Data Statistics Administrator role on the team hyperlink of your study under Studies tab requested for data export but when I tried to open the file received a message Request timed out Add and grant appropriate roles to the user Desk Help NU to your Study and email the name of your study to helo assessmentcenter net This is most likely due to file size so we will download the data from our end PROMIS Instruments Where can I find a PROMIS instrument Review the Instrument Library document for a complete list of PROMIS instruments publically available in Assessment Center http assessmentcenter net documents InstrumentLibrary pdf These instruments may be utilized in any data collection stud
22. le with some participants who will be young adults ages 18 24 Should I use the pediatric or adult instruments for young adults There is not yet a perfect answer here The pediatric items will likely work in this age range but they haven t yet been tested The pediatric items are targeted towards an 8 18 age range and some of the wording may feel a little young The adult items are appropriate for young adults but then you are using two different measures in one sample The PROMIS network is conducting multiple studies that will link the pediatric and adult scales so that in the future it won t matter which scale is used Review the banks of interest to determine what is best for your study Last Updated 10 20 2014 9 What is an adequate response rate in health related quality of life research Setting an adequate response rate depends upon context In a single one time assessment where you are talking about a percent of a defined sample e g percent of patients from a clinician s practice the percentage is less important than getting a representative sample and sufficient sample size Minimum sample size for one group summary statistics not per subgroup is roughly 40 but preferably 50 In longitudinal studies of relatively healthy people who remain alive through the follow up period 80 is adequate though 90 is better In longitudinal studies with sicker patients 70 tends to be treated as adequate but missing data is
23. lly individual items can be flagged to indicate if they include protected health information making de identification of datasets easier Also registration fields are stored in a separate database from other survey data This allows collection of protected health information to be stored separately from patient reported outcome data Who has access and who owns data collected in Assessment Center Only study team members that have been given access by you can view or export study data All data collected in Assessment Center is owned by you The Assessment Center team and PROMIS network make no claims on your data Last Updated 10 20 2014 3 How do I register in Assessment Center 1 Go to www assessmentcenter net 2 Click on Register New User 3 Fill in all the required fields for registration 4 An email will be sent to you for validation Click on the link you received on your email 5 Log in to Assessment Center using the credentials you created What will you do with the email address provide when I register in Assessment Center Your email address is used by the Assessment Center team to periodically provide you with updates about new measures training opportunities and scheduled maintenance time We also may contact you about research opportunities Your email is not sold or utilized for non Assessment Center purposes How do I retrieve a lost password To retrieve a lost password go the Assessment Center website www
24. ml and the Assessment Center User Manual link in left column of home page or www assessmentcenter net UserManuals aspx have read the Assessment Center User Manual but need more help What should do Consider attending an Assessment Center workshop These workshops provide an introduction to PROMIS item response theory computerized adaptive testing and a hands on tutorial for using Assessment Center Information about upcoming workshops is available at www assessmentcenter net You can also email help assessmentcenter net Last Updated 10 20 2014 14 Can you set up my study for me We hope that you will be able to set up your own study by using the available help resources We are not able to set up a study for you However we are happy to answer questions at help assessmentcenter net Can you review my study before I finalize it We would love to review your study before it is finalized Contact Assessment Center Help a team member by sending an email to helbp assessmentcenter net with your study name Can I still use Assessment Center if do not have internet access There are options available for collecting data via Assessment Center when connecting to the internet is not feasible To learn more about offline data collection requirements please contact us at help assessmentcenter net Is it possible to integrate Assessment Center with an EMR Electronic Medical Record It is currently possible to integrate A
25. multiple versions of instruments i e Item Banks Computerized Adaptive Tests CATs and Short Forms Generally it is recommended that you use the most recent version available which can be identified as the instruments with the highest version number Instruments are changed for various reasons For example the original PROMIS Bank version 1 0 Physical Function included 124 items after being tested in a diverse sample Later during an effort to translate instrument items into multiple languages translation challenges were identified Therefore minor modifications to the English source items were required e g metric equivalents to measurements such as Over 10 pounds 5 kg were added These modifications 19 in total resulted in the creation of a version 1 1 item bank Later version 1 2 was created by eliminating two items due to restrictions in their use In most cases an instrument that has a decimal increase v1 0 to v1 1 or v1 2 retains the same item level parameters as well as instrument reliability and validity In cases where a version number increases by a whole number e g v1 0 to v2 0 the changes to the instrument are more substantial For example the PROMIS Bank v1 0 Satisfaction with Participation in Social Roles is a small item bank comprised of 14 role function items While the PROMIS Bank v2 0 Satisfaction with Social Roles and Activities instrument includes a broader range of item content Version 2 0 was re cali
26. nt Center after the instrument has been completed Scoring instructions for short forms are also available on the Assessment Center homepage under the Scoring Manual hyperlink Last Updated 10 20 2014 7 Have the PROMIS instruments been validated This question implies a simple yes or no answer There are some questions that can be answered appropriately with a positive or negative response Do you have disability insurance Have you ever had surgery Are you pregnant The validity of an outcomes instrument however is not at all like being pregnant An instrument can never be said to be valid in any unqualified sense One might expect that an expertly developed fourth grade math test would be valid for discriminating math proficiency levels among fourth graders Validity however is not portable A test that is valid in a particular population for a particular purpose may not be valid for use in other populations and for other purposes The hypothetical math test described above would not be a valid test of math proficiency of high school students nor would it be a valid test of the writing skills of fourth graders Validation is an ongoing and never completed exploration and documentation of how a measure functions The strength of the validity evidence is judged in the courtroom of scientific opinion In fact the judicial system provides a serviceable metaphor Validation is the process of building a case for a measure Var
27. of medical product treatment At the end of the day the best way to get a read from the FDA on PROMIS is to include it with a submission for consideration What is the difference between the PROMIS Global scale and a PROMIS profile 29 43 57 instrument Why would I use one over the other The PROMIS Global scale and PROMIS Profiles serve different purposes PROMIS Global is a 10 item scale that produces two health scores physical and mental These are amalgamations of multiple constructs For example physical health includes items about physical functioning pain intensity and fatigue The PROMIS Profiles are 29 43 or 57 item batteries that consists of 4 6 or 8 item short forms for depression anxiety fatigue sleep disturbance physical function pain interference and satisfaction with participation in social roles It also has a single pain intensity item Profiles produce scores for each of these domains You should choose the instrument that best fits your specific need The PROMIS Profile instruments are able to provide you a sensitive measure of each specific domain The items included in each profile short form were selected because they provide the most information for the broadest range of people Longer Profiles have more measurement precision than shorter profiles though they require more time to complete If you want to evaluate an assortment of self reported health domains a Profile is a good choice If you are comparing
28. often not randomly missing which causes problems Does PROMIS have proxy measures PROMIS has developed proxy measures for the pediatric instruments These are intended to be completed by a parent for a child ages 5 18 These measures are available in Assessment Center PROMIS does not have proxy measures to be completed by a caregiver for an adult In selecting whether to use the pediatric or pediatric proxy instrument it is important to consider both the population and the domain which you are studying Pediatric self report should be considered the standard for measuring patient reported outcomes among children However circumstances exist when the child is too young cognitively impaired or too ill to complete a PRO instrument While information derived from self report and proxy report are not equivalent it is optimal to assess both the child and the parent since their perspectives may be independently related to healthcare utilization risk factors and quality of care Are PROMIS instruments available in languages other than English If so how can I get them Most PROMIS instruments are available in Spanish Other translations and validation of translations is in process See the NIH PROMIS website for a table of available translations http www nihpromis org measures translations Contact helo assessmentcenter net for more information What is the difference between versions of PROMIS instruments Some PROMIS domains have
29. ost of the resistance comes from a lack of understanding of how responses to different items could be on the same metric However there is a legitimate concern if a short form includes items that have differential item function DIF or some other psychometric issue When the same short form is used at two different time points the error introduced is consistent In other words in repeated measure designs the error is accounted for because participants are their own controls When completing CAT instruments a participant may get 5 items with DIF at the first time point and no items with DIF at the second time point This results in the introduction of an incorrect error This issue is introduced because some item banks vary in how they work with different subpopulations Without the creation of perfect item bank this problem cannot be completely eliminated How does CAT handle trial items which need examination These items often called seeding items are administered but do not count towards persons scores The data collected is used later to calibrate the new items What types of instant CAT score reports exist in Assessment Center CAT scores are reported in two data exports Assessment Data Export and Assessment Score Export When administering a public PROMIS v1 0 CAT instrument of the following domains Anxiety Depression Fatigue Pain Impact Physical Function Sleep Disturbance and Satisfaction with Social Roles s
30. sing two psychometric criteria 1 maximum interval information and 2 CAT simulations Item rankings were similar for both criteria For the maximum interval criterion each item information function was integrated without weighting for the interval from the mean to 2 SDS worse than the mean For the CAT simulations responses to all items in each bank were generated using a random sample of 1 000 simulees drawn separately for each bank centered on 0 5 SD worse than the general population mean Items were rank ordered based on their average administration rank over the simulees Content experts reviewed the items and rankings and made cuts of 4 6 and 8 items Is one short form better than the other In selecting a short form the difference is instrument length The reliability and precision of the short forms within a domain is highly similar If you are working in a sample in which you expected huge variability in a domain area and wanted different subdomains covered you would probably go for the original short form If you are working with a sample in which you wanted the most precise measure you would go for an 8a short form If you had little room for additional measures but really wanted to capture something as a secondary outcome you would go for a 4a short form Where can I learn more about a given domain For more information on a domain please see http nihpromis org science publications I am working with a pediatric samp
31. ssessment Center with the Epic EMR system Once integrated Assessment Center is able to send PRO data back to Epic in the form of an HL7 coded message Epic is then able to display instrument scores as lab results Additionally the alert system in Epic may be used to send alerts to various supportive staff e g clinicians and nursing pools when a patient responds to an instrument in a pre defined way Patients access assessments from the Epic patient portal MyChart There are two options for integrating PROMIS instruments into Epic 1 Epic 2012 includes a library of PROs including a number of PROMIS short forms for adults pediatric patients and parent proxies Prior to use of these instruments a clinician would work with his her local EPIC team to identify what triggers an assessment e g appointment happening within a specific clinic and the content of the assessment Patients would receive emails to complete assessments via MyChart a Advantages cost effective if organization is using Epic 2012 b Disadvantages limited to current content of Epic library c If interested in utilizing PROMIS instruments in this manner reach out to local Epic staff for further information guidance and or training 2 Assessment Center s IT group has worked with Epic to integrate the two systems This allows a clinician to create an assessment in Assessment Center that can include instruments not in the Epic library e g PROMIS CATs other legacy in
32. st a change in the patient s management J aeschke R Singer J Guyatt GH Measurement of health status Ascertaining the minimal clinically important difference Controlled Clinical Trials 1989 10 4 407 415 However for patients with advanced stage disease where palliation is the intent of treatment the MID is patient centered and may have no specific reference to the clinical aspect of the patient s change Wyrwich KW Bullinger M Aaronson N Hays RD Patrick DL Symonds T Estimating clinically significant differences in quality of life outcomes Qual Life Res 2005 14 2 285 95 The recommended MID ranges for the PROMIS Cancer T Scores reported in that paper ranged from 2 5 to 6 0 points depending on the specific short form of interest Additional studies are currently being conducted to confirm these MIDs in other patient populations and to estimate MIDs for other short forms and CATs Have any meetings been held with regulatory bodies FDA and scientific advisors Throughout the PROMIS initiative several meetings have been held with the FDA including the PRO Consortium There are ongoing discussions regarding the acceptability of PROMIS measures and a broader shared regulatory science agenda Laurie Burke MPH PRh Director of Study Endpoint and Label Development at the Center for Drug Evaluation and Research at the FDA stated FDA joins NIH in the goal of ensuring that PROMIS results in meaningful measurement of the impact
33. st IRT information at the estimated trait level Are there any organizations which provide CAT item analysis training See the links below for information on selected online measurement courses http www statistics com dr everett smith http education uic edu programs 27 masters degree program 101 med in measurement evaluation statistics and assessment mesa Additional User Support I got an error message in Assessment Center What should do Some error messages will provide information on how to correct a problem For example if you enter an instrument name that already exists in the system you will be asked to amend the name Other times an error message appears more technical and does not give you a clear indication of what to do In these rare cases copy the error message note where you are in the system what screen what you were trying to do and the date time you received the error Then contact helo assessmentcenter net with this information and we will try and resolve the problem Where do I go to find information on PROMI S and Assessment Center training sessions Information about PROMIS and Assessment Center training workshops is posted at www assessmentcenter net Where can I find detailed instructions on how to use each Assessment Center feature You can find instructions in the Help feature hyperlink in upper right corner of Assessment Center or https www assessmentcenter net ac1 Help index ht
34. struments Assessment Center sends over each survey item along with scores via calls from MyChart Again a clinician would work with his her local Epic team to identify what triggers an assessment and patients would receive emails to complete assessments via MyChart A clinician would need to work with his her local Epic team and Assessment Center s IT team on where to place scored data within Epic a Advantages custom assessments b Disadvantages requires more effort from local Epic team requires subcontract to cover Assessment Center IT effort c If interested in integrating Assessment Center and Epic in this manner you will need to contact Assessment Center s IT Director Maria Varela mvarela northwestern edu Last Updated 10 20 2014 15
35. ter ccccccecsese eee eeeeateneeeennees 13 How should one determine whether to administer a long vs short CAT cceceeeeeeeeeeee teen ees 14 What are robust approaches for CAT item SelECtION cccceete cece etree etree e ete eteee ean eat eee e nent ed 14 Are there any organizations which provide CAT item analysis training ccseeeeee eee eeee eee ene 14 Additional User Supports cise reece pescsccean pecs ovetar teal wh fons ice pees deste caaans seageaie ets 14 got an error message in Assessment Center What Should d0 ccccccececeeee eet eeeeeeat eae eeennnes 14 Where do go to find information on PROMIS and Assessment Center training sessions 14 Where can find detailed instructions on how to use each Assessment Center feature 14 have read the Assessment Center User Manual but need more help What should do 14 Can you set up my study Tor Me ssir wees ees a a e nena RRA ed haga eedewra needa dey ia 15 Can you review my study before finalize it 00 need 15 Can still use Assessment Center if do not have internet ACCESS cece cece ee teen eee eee teens 15 Is it possible to integrate Assessment Center with an EMR Electronic Medical Record 005 15 Last Updated 10 20 2014 2 General Assessment Center What is Assessment Center Assessment Centers is a free online research management tool It enables researchers to
36. und Northwestern University is committed to supporting Assessment Center through 2019 and beyond Should NIH funding be insufficient to support Assessment Center users during this period we will continue to provide services on a cost recovery basis Basic Assessment Center access and support for researchers are currently funded by the NIH through September 2014 including enhanced support for the NIH Toolbox and Neuro QOL through September 2016 We anticipate that some level of NIH funding for all NIH sponsored researchers using Assessment Center will be extended through 2020 As such we recommend that NIH sponsored researchers budget 1 500 per year to cover any such shortfalls Potential long term support costs for other user types e g non NIH funded researchers clinicians and students are available on request What security measures are in place for online data capture Assessment Center takes security very seriously The last section of the Assessment Center User Manual www assessmentcenter net includes information about security including details about the system architecture SSL encryption and server security Additionally within a study Assessment Center utilizes role based permissions Only team members are able to obtain access to a given study Their access is limited based on their assigned role within that study For example only individuals with the appropriate role are able to edit or export participants data Fina
37. urements used the PROMIS HAQ or HAQ DI PROMIS Network How can I get started learning about PROMI S The PROMIS website www nihpromis org and Assessment Center homepage www assessmentcenter net have a lot of information about PROMIS methods and instruments For example e The instruments currently available http www nihpromis org measures availableinstruments e Example instrument questions http www nihpromis org measures SampleQuestions e Tips on selecting a PROMIS instrument http www nihpromis org measures selectinginstrument e The domain framework http www nihpromis org measures domainframework which can be clicked through to review content PROMIS measures e APROMIS Computer Adaptive Test demonstration http www nihpromis org software demonstration Another means of learning about PROMIS is to join the PROMIS community http www nihpromis org researchers oinCommunity and receive our newsletter have a great research idea and am looking for a collaborator How can connect with the PROMIS effort Email helop assessmentcenter net and we can help connect you with the right person in the PROMIS network Psychometric Issues amp Computerized Adaptive Testing CAT What is item response theory I RT In psychometrics item response theory IRT also known as latent trait theory or modern test theory is a paradigm for the design analysis and scoring of tests questionnaires and simil
38. y on Assessment Center These instruments are also available via PDF which can be printed for pen and paper data collection from within Assessment Center In order to access PDF versions of PROMIS instruments you will need to register and log into Assessment Center The PDF versions are available for download on the Resources page which can be accessed by clicking the Resources link in the upper right corner of the main Assessment Center application In addition most PROMIS instruments you add to your study will have a PDF version of the instrument on the Study Content page default page within Instruments tab What do the PROMIS instruments look like The CAT Demo at www assessmentcenter net acl assessments catdemo allows you to select from a number of PROMIS computerized adaptive tests complete them and view a report of your scores This will give you a feel for the type of items included in PROMIS CATs You can also register in Assessment Center and access the PDF versions of PROMIS instruments for review A select number of short forms can be viewed through clicking the Request PDFs of PROMIS Short Forms button on the Assessment Center homepage How do I score PROMIS instruments CATs are scored as they are administered by Assessment Center A theta estimate is calculated after each response Scoring Manuals for short forms are available at https www assessmentcenter net Manuals aspx Short forms and Profiles are scored by Assessme
Download Pdf Manuals
Related Search