AIM Populations and Ecosystems
Contents
1. ZiSslesl l aloala sjel N W A nN oo O N N N N N U N aN N nN N ON N N oo N O a 9 o lw o ld gt olg o gt o w gt w gt w old g o o o w gt o gt o s u U z 30 Primary 2 1 0 5 1 2 2 1 3 1 2 1 1 3 1 Secondary 0 0 1 0 0 0 0 0 0 0 0 1 0 0 Total 2 1 1 5 1 2 2 1 3 1 2 2 1 3 2 slo s 2013 9 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Validity Three lines of evidence support the argument that the assessment is a valid measure of students knowledge of these populations and ecosystem ideas First cognitive interviews with students established that students interpret the items as intended and that they must use their knowledge of content to answer the items correctly Second a content expert individual with a Ph D in biology reviewed the assessment items to ensure content accuracy Third factor analysis indicates that all items on the assessment measure a single dominant trait HRI termed this trait content knowledge about populations and ecosystems Reliability Both classical test and item response theory IRT analyses were conducted on the pilot data and those results were used to select items for the2. student learning Amount of Time Required to Complete the Assessment Although there is no evidence of speededness it is recommended that at least 45 minutes be allowed for completing the assessment 2013 10 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Computing Scores Scores may be computed either as number correct or percent correct Results of an item response theory IRT analysis are shown in Table 5 This table can be used to convert a raw score in terms of number correct to the corresponding scale score Table 5 Assessment Score Conversions Raw Score Scale Score 0 0 1 13 2 21 3 25 4 29 5 32 6 34 7 37 8 39 9 41 10 42 11 44 12 46 13 47 147 497 15 50 16 52 17 54 18 55 19 57 20 58 21 60 22 62 3 64 24 66 25 69 26 71 27 75 28 80 29 87 30 100 T Mean value References American Association for the Advancement of Science 1993 Benchmarks for science literacy New York Oxford University Press National Assessment Governing Board U S Department of Education 2008 Science framework for the 2009 national assessment of educational progress Washington DC U S Government Printing Office National Research Council 1996 National science education standards Washington DC National Academy Press 2013 11 Populations and Ecosy
3. AIM assessments The NAEP Framework was based primarily on the National Science Education Standards National Research Council 1996 and the Benchmarks for Science Literacy American Association for the Advancement of Science 1993 but also reflected developments in science and policy that have taken place since those documents were published HRI specified the assessment domain using a single strand in the NAEP Framework the interdependence of organisms and specific types of interdependence This process had three biologists biology educators unpack the content into series of sub ideas for middle school students These are the ideas that were considered in developing the middle school student assessment The final description of the content domain is shown in Table 1 2013 3 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Table 1 Populations and Ecosystems Content Domain Populations and Ecosystems The interdependence of organisms and specific types of interdependence Sub ideas for students e Organisms interact and are interdependent in various ways including providing food and shelter to one another o An organism is any living thing such as a plant or an animal Organisms are categorized by how they get their food o Organisms depend on other organisms for food and or nutrients o In some interactions both organisms benefit by interacting and are more likely to surviv
4. ecosystem 1 C Food webs illustrate the feeding relationships between producers and consumers and among 1 consumers in an ecosystem D Competitive relationships exist when multiple organisms rely on the same resource s E Mutually beneficial relationships mutualisms exist when organisms interact and both or all are 1 more likely to survive and or reproduce F Co evolution occurs when more than one species have existed together long term influencing 2 changes in each other G The number of organisms and populations an ecosystem can support depends on the biotic 2 resources available and abiotic factors such as quantity of light and water range of temperatures and soil composition H All populations living together and the physical factors with which they interact compose an 1 ecosystem I A population is a group of individuals of the same species that live and breed together in a 3 particular area J Organisms compete for limited resources Populations of organisms can potentially grow 1 unchecked but tend to remain the same size because of limited resources K When organisms compete often only some organisms survive because they are better adapted for 2 acquiring resources in that environment L Population density in an area varies from time to time based on variation in availability of 2 nutrients weather and presence of other organisms M All organisms cause changes in the environment where they live Some of these chan
5. final assessment The assessment has an IRT reliability of 0 82 reliabilities above 0 60 are generally considered acceptable for making judgments about groups higher reliabilities are required for making high stakes decisions about individuals Speededness In the pilot teachers were instructed to give their students 50 minutes or the length of the class period whichever was shorter to complete the test There was no evidence of speededness Using the Assessment The AIM Populations and Ecosystems Student Assessment is available at no cost through an online process to those who agree to the terms of use see the Appendix To complete the terms of use agreement visit http www horizon research com aim instruments Appropriate Use The AIM Populations and Ecosystems Student Assessment yields a score for each individual However the assessment is not valid for making judgments about individuals based on those scores For instance assigning student grades based on scores is not a valid use of the assessment The assessment was not validated for such purposes HRI developed the assessment for use in research contexts involving groups of teachers Appropriate uses with sufficiently large groups of teachers 20 or more include e Measuring the change in group mean from pre workshop to post workshop e Comparing the gains of treatment and control groups and e Researching the relationship between teacher knowledge and other variables e g
6. AIM User Manual Populations and Ecosystems Middle School Student Assessment Overview The AIM Populations and Ecosystems Middle School Student Assessment is a 30 item multiple choice assessment developed for middle grades science students The assessment is based on the Science Framework for the 2009 National Assessment of Educational Progress National Assessment Governing Board 2008 and measures understandings of concepts in related single content area the interdependence of organisms and specific types of interdependence This user manual describes the background development measurement properties and appropriate uses of the assessment User manuals for other AIM assessments may be found at http www horizon research com aim instruments Background Horizon Research Inc HRI developed the AIM Populations and Ecosystems Student Assessment as part of a larger study The project Assessing the Impact of the MSPs K 8 Science AIM was funded by the National Science Foundation under Grant no DUE 0928177 One goal of AIM was to develop instruments that researchers could use to study the theory of action that underlies much professional development for science teachers Briefly the model asserts that changes in teacher knowledge lead to changes in classroom practice mediated by instructional materials and ultimately changes in student learning see Figure 1 Despite the prominent role this model plays in professional development d
7. approval for your study and to follow the necessary requirements for obtaining principal teacher parent and or student permission approval to administer to the assessment s Responsibilities to Teachers and Students Your responsibilities to study participants will largely depend on the details of the IRB and or district school approval of your study In most cases completion of the assessment will be strictly voluntary As such participants should be informed of the voluntary nature of the study Teachers should be assured that if their students data are not anonymous individual identities will be kept strictly confidential i e an individual s score or responses will never be reported in association with his or her name or any other identifying information To encourage a high response rate among teachers it may be helpful to e Clearly explain what the data will be used for and why the data are important for your study e Explain that there are no high stakes consequences associated with completing the assessment and e Offer teachers compensation for time spent outside of the regular school day completing the assessment 2013 12 Populations and Ecosystems Student Assessment Horizon Research Inc Terms of Use Agreement Test Security The AIM Populations and Ecosystems Student Assessment may NOT be shared without prior authorization from HRI Anyone who administers the assessment must agree to e Refrain from using any non r
8. articular area Organisms compete for limited resources Populations of organisms can potentially grow unchecked but tend to remain the same size because of limited resources When organisms compete often only some organisms survive because they are better adapted for acquiring resources in that environment Population density in an area varies from time to time based on variation in availability of nutrients weather and presence of other organisms All organisms cause changes in the environment where they live Some of these changes are detrimental to the organisms or other organisms whereas others are beneficial o The mix of species in a given environment changes in response to a changing environment over time o Ecological succession occurs in areas where there have previously been no organisms e g weathering rock or where prior life has been removed or reduced e g volcanoes fire Pioneer organisms change the environment in ways that make it possible for other organisms that could not previously survive in the environment to live there Organisms successively change the environment making it more suitable for other organisms Some organisms can change their environment to the extent that their offspring can no longer live there This area becomes better suited to other organisms Non native organisms introduced into a new area may outcompete overeat and or replace native organisms resulting in significant environmental changes An
9. choice format recognizing the limitations of such items For instance well constructed open ended items may probe more depth of understanding than multiple choice items but they are more burdensome for both the researcher in terms of scoring costs and the test taker in terms of time required to complete the assessment In addition scoring open ended items requires the training of raters to establish inter rater reliability Development of the Populations and Ecosystems Middle School Student Assessment As described above this development effort was part of a much larger and well funded project which afforded a thorough development process see Figure 2 2013 2 Horizon Research Inc Populations and Ecosystems Middle School Student Assessment User Manual Assessment Development Process Three Ph D s in content area unpack and clarify the science content to be assessed Write multiple choice items Collaborative item editing H Conduct cognitive interviews with teachers students ma y Review by a Ph D in content area y Pilot multiple choice items with teachers students y C Conduct IRT analyses and construct soale gt Figure 2 Clarifying the Content Domain Development began with identifying the target content for the populations and ecosystems assessments We used the 2009 NAEP Framework for direction on the content of the
10. described above Printed Name Signature Date Address Street City State Zip code Phone number including area code Your email address 2013 13 Populations and Ecosystems Student Assessment Horizon Research Inc Terms of Use Agreement
11. e and reproduce o In some interactions one organism will benefit by interacting and is more likely to survive and reproduce while the other is harmed and its survival and or reproduction may be limited Organisms can survive only in environments in which their needs are met o Each type of organism has a specific range of environmental conditions under which it can survive Environmental conditions include but are not limited to temperature moisture amount of oxygen nutrient availability and salinity Some interactions are beneficial others are detrimental to the organism and other organisms When the environment changes some plants and animals survive and reproduce others die or move to new locations An organism s environment includes all of the living and non living things that surround and influence the organism Organisms have different traits some traits are better than others for a given environment i e help the organism meet its needs Organisms with traits that are favorable in an environment are more likely to survive and reproduce whereas organisms that lack those traits are less likely to survive and reproduce Organisms including humans often change the environment in which they live through feeding leaving waste and or competing with other organisms Sometimes environments change and no longer provide for the needs of some or all of the organisms that live there Some organisms will be able to survive in the new conditio
12. ecological niche is the role that an organism plays in its environment An organism s niche includes the resources it uses from the environment and how it acquires and uses those resources No two species can occupy the same niche at the same time competitive exclusion principle Competition between two species sometimes leads to increased niche specialization resource partitioning thereby permitting coexistence of multiple species An adaptation is a physical or behavioral trait of an individual that enhances its chances of surviving and reproducing in an environment The development of an adaptation does not occur within the lifetime of an organism but only within populations and across generations Individuals of a species or population have variation in their traits Natural selection refers to the selection by the environment selective pressure of those traits that promote survival and reproduction better than others Natural selection acting on heritable traits results in the survival of organisms that are more suited to a particular environment 2013 4 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Item Development HRI staff drafted items individually then met to edit them collaboratively As the pool of items grew we began recruiting middle school students for telephone cognitive interviews We typically interviewed three students on each item in the pool using the interview pro
13. eleased item in any presentation paper article or other public forum Items are expensive to develop and pilot and we are attempting to keep our item pool secure e Refrain from distributing copies of any non released item to individuals other than participants in your research project e Refrain from using the assessment in original or in copied form to provide test taking practice or to enhance test taking skills e Refrain from using test items actual or similar for discussion or review HRI acknowledges that in some cases school administrators and IRBs may require that the test materials be reviewed prior to granting permission for study participants to take the test Such a review is not considered a violation of this Test Security Policy as long as the other provisions of this policy are not violated Citing AIM Assessments In any writing in which data from HRI s AIM assessments are included the following citation must be used The assessment was developed by the Assessing the Impact of the MSPs K 8 Science AIM project at Horizon Research Inc funded by the National Science Foundation under grant number DUE 0928177 Any opinions findings and conclusions or recommendations expressed herein are those of the authors and do not necessarily reflect the views of the National Science Foundation or Horizon Research Inc By signing below I acknowledge that I have read the user manual and I agree to abide by terms of use
14. esign it has not been studied systematically in part because of a lack of instruments Among other products AIM developed pairs of assessments one for teachers and one for students focused on the same science content areas These pairs of assessments enable the study of relationships between teacher knowledge and student learning in specific science contexts AIM assessments exist for four content areas 1 evolution and diversity of life 2 force and motion Newton s first and second laws 3 populations and ecosystems and 4 properties of and changes in matter For each content area separate pairs of assessments were developed for elementary school and middle school levels Any opinions findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation 2013 1 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Professional Development Theory of Action Improved Teacher Content a Increased Student s a i Improved Classroom ee EA Knowledge for Science Teaching Practice Achievement J E 7 e E a gt Instructional Materials Figure I To enable large scale research HRI set out to create assessments that would be minimally burdensome both for the test taker and the researcher Accordingly HRI opted for a multiple
15. ges are 1 detrimental to the organisms or other organisms whereas others are beneficial N Ecological succession occurs in areas where there have previously been no organisms e g 3 weathering rock or where prior life has been removed or reduced e g volcanoes fire Pioneer organisms change the environment in ways that make it possible for other organisms that could not previously survive in the environment to live there O An ecological niche is the role that an organism plays in its environment An organism s niche 2 includes the resources it uses from the environment and how it acquires and uses those resources P No two species can occupy the same niche at the same time competitive exclusion principle 4 Competition between two species sometimes leads to increased niche specialization resource partitioning thereby permitting coexistence of multiple species Table 4 shows the answer key and content association for each item on the assessment The letter P denotes a primary association with the sub idea being targeted by the item An S denotes a secondary association with a sub idea that is also necessary in order to answer the item correctly but is not the primary idea being assessed 2013 8 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Table 4 Answer Key and Sub Idea Associations Item Sub Idea Ke A B C D E F G H I J K LIM N O P P
16. it that allows it to survive and reproduce D A process in which organisms with genetic traits that are best suited to their environment are most likely to survive and reproduce Figure 4 This item illustrates some features common to all AIM student assessment items It is not included in the AIM assessment but is shown here to illustrate item features This example item may be flawed and is not intended to be used in any assessments As mentioned previously all items are multiple choice All include only four choices and preclude as choices none of the above all of the above or multiple correct answers such as A and B but not C Pilot We selected 40 items to pilot with approximately 614 students of teachers recruited from mailing lists of middle grades teachers across the country The pilot was administered as a paper form by recruited teachers Table 2 Characteristics of the Pilot Test Sample Percent of Students Grade Level 6 grade 34 7 grade 35 gn grade 31 English is primary language Yes 96 No 4 Gender Female 54 Male 46 Race Ethnicity American Indian or Alaskan Native 3 Asian 2 Black or African American 16 Hispanic or Latino 11 Native Hawaiian or Other Pacific Islander 1 White 75 Percentages may add up to more than 100 as students could select multiple categories 2013 6 Populations and Ecosystems Horizon Research Inc Middle School Student A
17. ns some will move to a new environment where their needs are met and some will not survive Two types of organisms may interact with one another in several ways They may be in a producer consumer predator prey or parasite host relationship Or one organism may scavenge or decompose another o Producers including green plants and algae are the primary food source within an ecosystem o Food webs illustrate the feeding relationships between producers and consumers and among consumers in an ecosystem Relationships may be competitive or mutually beneficial Some species have become so adapted to each other that neither could survive without the other Sometimes organisms compete with each other for food and resources Competitive relationships exist when multiple organisms rely on the same resource s Mutually beneficial relationships mutualisms exist when organisms interact and both or all are more likely to survive and or reproduce Co evolution occurs when more than one species have existed together long term influencing changes in each other The number of organisms and populations an ecosystem can support depends on the biotic resources available and abiotic factors such as quantity of light and water range of temperatures and soil composition All populations living together and the physical factors with which they interact compose an ecosystem A population is a group of individuals of the same species that live and breed together in a p
18. self herself if he she needs to If reading the question aloud is too distracting or uncomfortable allow the student to read the question to himself herself It is not necessary to time how long it takes for the student to arrive at an answer but if it takes an especially long time on a question please make a note of it in the comment area of the notes For each item ask Why did you choose that answer probe for words or diagrams they keyed in on as well as their thinking behind the response What did you think of each of the other answer choices Was there an answer choice you were expecting to see but did not What was it Were there any words or diagrams you did not really understand or situations that made the question confusing Is there anything about the question that did not confuse you but that you think might confuse other students Do you have any other comments on the item Figure 3 An example student assessment item resulting from this process is shown in Figure 4 correct answer is D 2013 5 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Populations and Ecosystems Item What is natural selection A An organism selecting the environment that is best adapted for its survival B An organism passing on traits it develops throughout its lifetime in response to its environment C Environmental conditions that cause an organism to develop a new tra
19. ssessment User Manual Measurement Properties of the Assessment Following is a description of the content coverage of the assessment information about the validity and reliability of the assessment and the results of the item response theory IRT analysis Content Coverage Using results from the pilot 30 items were selected for the final form The distribution of items by sub idea is shown in Table 3 The number of items totals to more than 30 because one item may address more than one sub idea There are fewer sub ideas in Table 3 than in the content unpacking see Table 1 as limiting the assessment to a total of 30 items required restricting the coverage of sub ideas In some cases a sub idea may not be represented in the final assessment because it was deemed to be less central than others In other cases items associated with the sub idea did not perform as well as others in the pilot study 2013 7 Populations and Ecosystems Horizon Research Inc Middle School Student Assessment User Manual Table 3 Number of Items Addressing Each Sub Idea Number of Sub Ideas Items A Two types of organisms may interact with one another in several ways They may be in a 2 producer consumer predator prey or parasite host relationship Or one organism may scavenge or decompose another B Producers including green plants and algae are the primary food source within an
20. stems Horizon Research Inc Middle School Student Assessment User Manual Appendix Terms of Use Agreement Populations and Ecosystems Middle School Student Assessment By using the AIM Populations and Ecosystems Student Assessment developed by Horizon Research Inc HRI you agree to abide by the stipulations below concerning use test security test administration and citations Use of the Assessment The Populations and Ecosystems Student assessment may be used to gauge growth in knowledge about a specific content area as a result of an intervention such as professional development curriculum use or mentoring It may also be used to learn about the contribution of teacher knowledge to student knowledge and classroom instruction We ask that you abide generally by the standards put forward in the Standards for Educational and Psychological Testing AERA APA 1999 You may not use the assessment to evaluate individuals Assessment results may not be associated with any high stakes consequence such as tenure pay hiring or grades The assessments were not developed for making decisions judgments about individuals You should also refrain from using these measures to publicly demonstrate teachers ability or lack of ability in science which may adversely affect willingness to participate in future studies IRB and or District School Study Approval It is your responsibility to obtain proper IRB and or the appropriate district school
21. tocol shown in Figure 3 After a round of interviews HRI staff met to discuss students responses and feedback If substantive edits were made to an item we interviewed additional students about the revised version When interviews suggested no further edits were needed we asked a content expert to review all of the items in the pool for content accuracy AIM Student Assessment Items Cognitive Interview Protocol Prologue Script We are developing test questions for middle school students who have been studying populations and ecosystems and we need your help to get the questions just right I realize that you may not have studied some of this yet in school and I don t expect you to get all of the answers right If you get a few wrong it will help me know whether we have written the answer choices well You can ask me to explain any words or situations that may be unfamiliar or confusing but I can t give you the answer to any of the questions until the end of the interview Remember the point is to help us write a good test not to test what you do or don t know You won t get a grade or anything like that on the test Do you have any questions before we get started If at any point in the interview you would like to stop just say so Procedure e Ask student to read aloud and think aloud as they read the questions and answer choices if they are comfortable doing so Remind the student to go back and reread the question to him
Download Pdf Manuals
Related Search