Assessing Student Learning of Oceanography Concepts

USAGE Permission is granted to copy this article for use in teaching and research. Republication, systematic reproduction, or collective redistribution of any portion of this article by photocopy machine, reposting, or other means is permitted only with the approval of The Oceanography Society. Send all correspondence to: info@tos.org or The Oceanography Society, PO Box 1931, Rockville, MD 20849-1931, USA. Oceanography THE OFFICIAL MAGAZINE OF THE OCEANOGRAPHY SOCIETY


Assessing Student Learning of Oceanography Concepts
Ocean scientists are well versed in utilizing specialized methods and instruments to rigorously assess the ocean's physical, chemical, biological, and geological processes and properties.With growing national interest in specifically ocean literacy for citizens of all ages (National Oceanic and Atmospheric Administration, 2013) and K-12 science education in general (Next Generation Science Standards Lead States, 2013), similarly specialized methods and instruments are being developed to assess students' learning of oceanography concepts.The purpose of this commentary is (1) to introduce Oceanography readers, particularly those who teach at the college level, to a few teaching tools, such as learning goals and Bloom's taxonomy (Bloom, 1956), that could be immediately useful to instructors for assessing learning of oceanography concepts, and (2) to raise awareness of the availability of an instrument called the Oceanography Concept Inventory (OCI; Arthurs and Marchitto, 2011;Arthurs et al., 2015), which can help to assess student learning of oceanography concepts.Learning goals provide the foundation for creating or selecting appropriate assessments of learning, and the OCI is a ready-made instrument for assessing higher-order cognitive skills1 beyond the mere recall of factual information.
Rigorously assessing students' learning of oceanography concepts begins with explicitly defining what students are expected to understand as a product of participating in a given course, before the course begins.These expectations are known as learning objectives or learning goals (Simon and Taylor, 2009;Smith and Perkins, 2010), herein referred to as learning goals.Learning outcomes represent the actual outcomes of learning, whereas learning goals represent the desired outcomes of learning.Wiggins and McTighe (2005) view the articulation of the desired outcomes of learning as the first step in careful course design, aimed at promoting deep learning.Each learning goal is written in the form: "By the end of this course, students should be able to…[fill in the blank]." What follows "be able to" is a verb-driven task.The verb has a clear meaning, and the task is specific and directly assessable.A set of learning goals is ideally composed of learning goals that reflect a range in cognitive difficulty.Bloom's taxonomy (Figure 1) offers a useful and well-known framework for crafting a set of learning goals with varying levels of cognitive difficulty (Krathwohl, 2002).Initially vague or general learning goals can be made more specific and directly assessable by using Bloom's taxonomy and FIGURE 1. Bloom's taxonomy of the cognitive domain provides a useful framework for articulating learning goals that range from lower order thinking skills, such as "remember," to higher order thinking skills, such as "create." Figure is from Shabatura (2013) and used with permission from tips.uark.eduat https://tips.uark.edu/using-blooms-taxonomy By Leilani Arthurs the provided guidelines or template for phrasing learning goals.For instructors who are unsure where to start in creating their learning goals, listing the key concepts that are deemed essential to the course can be a helpful preliminary step; then, for each listed key concept, instructors can write one or more learning goals (Table 1 lists examples of key concepts and learning goals).Well-defined and explicit learning goals provide a strong basis for informing assessment of actual student learning (Wiggins and McTighe, 2005;Handelsman et al., 2007).
Any instrument used to assess student learning should be well aligned with the specific learning goals of a course.Course instructors can create such assessment instruments from scratch and/or utilize instruments that have already been created by test developers.Traditional instruments for assessing student learning include tests designed on the basis of an instructor's subject matter expertise, anecdotal experiences, or speculation about what students do and do not understand (Libarkin and Anderson, 2005).In the past three decades, however, discipline-based education research (DBER) scholars have developed an increasing number of research-based assessment instruments such as concept inventories (National Research Council, 2012).In contrast to traditional tests, concept inventories are assessment instruments designed on the basis of systematic research about students' cognitive models and their scientifically inaccurate or incomplete alternate conceptions.These cognitive models and alternate conceptions are related to earlier notions of preconceptions (Novak, 1977) and misconceptions (Helm, 1980).
It is widely recognized that students' minds are not the "blank slates" or "empty vessels" that they were once assumed to be; contemporary research in education, psychology, and developmental behavior shows that students have a wide range of prior knowledge and experiences that shape their less scientifically accurate conceptions (Treagust, 1988).These studentheld conceptions may either help or hinder future learning (National Research Council, 2000).As such, they are central to research-based concept inventories designed to assess student learning.
Concept inventories are composed of multiple-choice items in which research-derived student-held alternate conceptions are used as the answer choices (Libarkin and Geraghty Ward, 2011).DBER scholars in the discipline of physics were the first to develop a concept inventory, the Force Concept Inventory (FCI) (Hestenes et al., 1992).Since its inception, the FCI has been used effectively to measure and compare learning gains and inform instructional strategies in physics courses across the United States (Hake, 1998).Following the development of the FCI, concept inventories for other disciplines were developed.They include, for example, the Geoscience Concept Inventory (Libarkin and Anderson, 2006), the Light and Spectroscopy Concept Inventory (Bardar et al., 2007), the Genetics Concept Assessment (Smith et al., 2008), the Thermochemistry Concept Inventory (Wren and Barbera, 2014), the Enzyme-Substrate Interactions Concept Inventory (Linenberger and Lowery Bretz, 2015), and the Statistical Reasoning in Biology Concept Inventory (Deane et al., 2016).Recently, the Oceanography Concept Inventory, or OCI, was developed specifically for oceanography and marine science (Arthurs and Marchitto, 2011;Arthurs et al., 2015).
The OCI was developed and tested using psychometrically accepted methods for which the issues of validity, reliability, and generalizability are paramount.A mixed-methods approach was employed wherein qualitative methods and classical TABLE 1. Eleven key concepts for an introductory-level oceanography course and their associated learning goals provided the basis for the 23 items that comprise the full set of Oceanography Concept Inventory (OCI) items.Refer to Arthurs and Marchitto (2011) for more extensive discussion on how these key concepts and learning goals were created.Table from Arthurs and Marchitto (2011), with permission from the Geological Society of America

Key Concept
Learning Goal Item No.

Isostatic equilibrium
Explain how isostatic equilibrium accounts for the existence of ocean basins.test theory were used to develop the initial version of the instrument (Arthurs and Marchitto, 2011), and quantitative methods and item response theory were used to evaluate and further refine the instrument (Arthurs et al., 2015).The answer options to all the OCI multiple-choice items are student-held alternate conceptions that occurred with notable frequency during the conceptions-research stage of development.The answer choices are intentionally written to minimize the use of scientific jargon and to incorporate actual verbiage that students used to express their answers to questions about key concepts.Figure 2 shows an example of two multiplechoice items developed for the OCI.
Consistent with other concept inventories, the OCI instructs students to select, among the choices, the single answer that they think is the best or most correct.The original version of the OCI has 23 items, and a second semi-customizable version has 16 items that are selected from the original 23.The OCI has the flexibility to be scored in one of two ways, either with classical one-point-per-item scoring or item-difficulty-weighted scoring.The OCI was developed to assess student learning of 11 key oceanography concepts and their associated learning goals (Table 1).
The OCI instrument can be implemented in a number of ways.The primary use of concept inventories is as a pre-and post-instruction test for which a class's normalized learning gain (<g>) is measured by comparing the its average pre-instruction score (S POST ) with its average post-instruction score (S PRE ), where, as defined by Hake (1998): 100% -(S PRE ) (S POST ) -(S PRE ) <g> = .
For strictly instructional uses, calculating normalized learning gains is a straightforward process that provides a window of insight into overall shifts in learning that potentially occurred during the period of instruction.The calculated learning gains provide instructors with useful feedback that they can use to inform future iterations of their courses, especially when used in conjunction with the results of other assessments during the semester (e.g., homework, quizzes, exams, projects).Instructors and education researchers who wish to use the OCI for research purposes should, of course, complement calculations of <g> with other statistical analyses such as a t-test or analysis of variance.
This type of data collected over different semesters, during which, for example, different instructional strategies are used, can provide a data-driven approach to instructional decisionmaking.Similarly, data collection with a psychometrically tested instrument can provide an additional element of rigor to program reviews or assessments of undergraduate curricula, such as the one recently described by Barrett et al. (2014).Finally, the misconceptions or less scientifically accurate conceptions that comprise the answer choices can be incorporated into constructivist approaches to instruction.Feller (2007), for example, lists 110 misconceptions that he encountered during 40 years of teaching oceanography and advocates for incorporating student misconceptions into classroom instruction.The OCI could be used in such a way by implementing individual multiple-choice items on the OCI as stand-alone ConcepTests for peer instruction (Mazur, 1997), in which students pair up to discuss a multiple-choice question and vote on their answers using, for example, colored cards or an electronic classroom response system (often referred to as "clickers").Use of individual OCI items as stand-alone ConcepTests is recommended only if the OCI is not being used as a pre-and post-instruction test for a given course so as to not "give away" the test items during instruction and maintain the validity of the post-instruction OCI scores.You might be asking yourself, "Is the OCI an instrument that I could use to assess student learning in my oceanography or marine science course?"Implementation of the OCI is most suitable when the conditions of the desired context are similar to those under which the OCI was developed and tested.Using the OCI to assess the learning of oceanography concepts means that the following conditions are met: • The course is an oceanography or marine science course.• The course is an introductory-level college course.• The learning goals for the course are the same or similar to the learning goals that the OCI assesses (Table 1).Assessments of learning are most useful when they are well aligned with the course learning goals, and this holds true for using the OCI.• The course is comprised of Englishspeaking students.The OCI was developed and tested with English-speaking students, and it is currently available only in English.
Finally, potential users of the OCI instrument are asked to read the "Uses and Applications" section in the article titled "The Oceanography Concept Inventory: A Semicustomizable Assessment for Measuring Student Understanding of Oceanography" (Arthurs et al., 2015) to learn more about how to assemble the shorter 16-item version of the OCI and about two different options for scoring the OCI.
As with other concept inventories (e.g., Force Concept Inventory, Genetics Concept Assessment), the full set of OCI test items is not published here in order to maintain its validity, reliability, and value to instructors.However, a complete set of the test items and their answers will be provided upon request.Interested instructors and researchers should contact the author at larthurs2@unl.edu for a copy of the OCI and/or assistance obtaining the 2011 or 2015 papers referenced in this article.
Feedback from OCI users is most welcome and will be used to further develop and refine the instrument to better meet the instructional community's needs.

FIGURE 2 .
FIGURE 2. (a) Item 1 from the Oceanography Concept Inventory (OCI) is shown as an example of one of the 23 OCI items.(b) Item 18 from the OCI is shown as an example of an OCI item that incorporates a pictorial or graphical component.Approximately half of the items are text only, such as Item 1, and the other half incorporate a pictorial or graphical component.