Teaching undergraduate students to read empirical articles: An evaluation and revision of the QALMRI method

Teaching undergraduate students to read empirical articles:

An evaluation and revision of the QALMRI method

Nicholaus P. Brosowsky

, Olga Parshina

, Anthony Locicero

, &

Matthew J.C. Crump

Department of Psychology and Neuroscience, Duke University

Department of Psychology, The Graduate Center of the City University of New York

Department of Psychology, Brooklyn College of the City University of New York

Preprint submitted for publication.

Abstract

Reading and comprehending empirical articles are important skills for stu-

dents to develop, yet many students struggle to identify and connect the

essential information from empirical articles. Here we describe and evaluate

a scaﬀolded approach for teaching undergraduate students to read empiri-

cal articles called the QALMRI method. The QALMRI is a generalizable

instructional tool for teaching students to identify the key conceptual infor-

mation necessary for the comprehension and critical evaluation of empirical

articles. We had students in a ﬁrst-year introductory course and students in

a third-year research methods course read empirical articles and complete

QALMRI outlines throughout the semester. We found that students very

quickly learned to use the QALMRI outline in both upper and lower course

levels, with performance corresponding to traditional written summaries.

However, we also found that students consistently performed poorly on some

items, prompting an update and revision to the QALMRI method to address

these limitations.

Keywords: QALMRI, reading, summarizing, instructional materials, empirical

articles

One important goal for any undergraduate psychology program is to teach students

skills for critically consuming research. The American Psychological Association (APA)

guidelines lists scientiﬁc literacy as its second major learning goal (Halonen et al., 2013),

and surveyed, life sciences faculty rated skills like interpreting data, writing reports, and

critically analyzing research articles among the most important for undergraduate students

Correspondence concerning this article should be addressed to Nicholaus P. Brosowsky, Duke University,

417 Chapel Dr, Durham, NC 27708, USA. E-mail: nicholaus.broso[email protected]

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 2

to learn (Coil, Wenderoth, Cunningham, & Dirks, 2010). In support of these goals many

psychology instructors incorporate primary source readings into their curriculum; for example,

Oldenburg (2005) estimated that 70% of liberal arts college instructors assigned primary

source readings. And although more prevalent in upper level courses, 46% of introductory

psychology instructors assigned primary source readings (Oldenburg, 2005).

Table 1: An outline of the QALMRI method

Questions What was the broad question being asked by this research project?

What was the speciﬁc question being asked by this research project?

Alternatives What was the author’s hypothesis?

What were the alternative hypotheses?

Logic What was the logic of the hypothesis?

i.e., if the hypothesis was true, what should we expect to happen?

Methods What were the methods?

Results What were the important results?

Inferences What inferences about the questions and hypotheses can be

made base on the results?

Adapted from Kosslyn and Rosenberg (2003); See also Brosowsky and Parshina (2017)

for a more detailed description and additional instructional materials

Primary source readings may not typically be introduced with the purpose of teaching

students how to read primary sources. Instead, primary source readings, like empirical

articles, are incorporated into the curriculum as a tool to help students learn related course

content by way of demonstration and/or elaboration. For example, primary source reading

has been used to demonstrate basic statistical and methodological principles (Christopher

& Walter, 2006; Pennington, 1992; Ware, Badura, & Davis, 2002), demonstrate writing

strategies (Price, 1990; Ware et al., 2002), to spark classroom discussions about course

content (Suter & Frank, 1986), promote critical thinking (Chamberlain & Burrough, 1985),

help motivate student engagement with course content (Carkenord, 1994), and as a tool to

assess student learning and performance (Bachiochi et al., 2011). Although these learning

goals do not explicitly include reading and comprehension, it is of course necessary and begs

the question: how do students learn to read empirical articles in the ﬁrst place?

If empirical articles are to be a useful pedagogical tool for learning other course content,

students must have the basic skills required to read those articles. Moreover, becoming

a critical consumer of the scientiﬁc literature is an important step in transitioning from

post-secondary education into the scientiﬁc community–in and of itself an important goal.

Yet there has been considerably less focus on teaching students how to read and critically

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 3

evaluate empirical articles (e.g., Kershaw, Lippman, & Fugate, 2018; Sego & Stuart, 2016).

In the current study, we evaluated a scaﬀolded approach for reading and summarizing

empirical articles known as the “QALMRI” method in both lower- (ﬁrst-year introductory

course; study 1) and upper-level (third-year methods course; study 2) psychology courses.

Reading empirical articles

Empirical articles can be more challenging than other reading materials, like textbooks,

commonly encountered by undergraduate students. Generally speaking, textbooks are written

for students and empirical articles are written for other experts. As a result, textbooks often

provide a great deal of guidance for the reader by exploiting text structures and signaling

techniques to highlight important information (Kardash & Noel, 2000; Sanchez, Lorch, &

Lorch, 2001; Varnhagen & Digdon, 2002); however, empirical articles often contain unfamiliar

terms and explanatory and argumentative text structures written without non-experts in

mind (e.g., Britt, Richter, & Rouet, 2014). Writing style may also inspire reading style.

For example, textbooks that express content as accessible units of facts may inspire a rote

approach to the content, where students assume their task is to memorize details of the

well-written paragraphs. By contrast, empirical articles contain persuasive arguments (Gillen,

2006; Suppe, 1998) with data and references to convince the reader of some claim (Latour,

1987; Van Lacum, Ossevoort, & Goedhart, 2014); and, although reading argumentative prose

might inspire students to the task of critically evaluating the basis for claims in the article,

they might also inappropriately persist in applying reading strategies like rote-memorization

that were previously useful for textbook readings.

Because students are engaging with materials intended for an expert audience, they

often ﬁnd common assignments like summarizing an empirical article (e.g., Anisfeld, 1987;

Gillen, 2006; Karcher, 2001; Levine, 2001; Suter & Frank, 1986) to be quite challenging (e.g.,

Day, 1986; Taylor, 1983). Students struggle identifying and understanding key components

of empirical articles. For instance, they can have diﬃculty understanding the motivation and

argument structure presented in an Introduction (Newell, Beach, Smith, & VanDerHeide,

2011; Van Lacum et al., 2014; Van Lacum, Ossevoort, Buikema, & Goedhart, 2012), the

concepts of experimental design in the methods (Zieﬄer & Garﬁeld, 2009), and the statistical

concepts found in the results (Dasgupta, Anderson, & Pelaez, 2014).

In short, reading empirical articles poses a unique challenge for undergraduate students.

Two aspects relevant to the current study are worth highlighting: First, students must be

able to ﬁnd and identify important conceptual information that may or may not be explicitly

stated. For example, the author may or may not explicitly state their hypotheses, yet the

reader should be able to discern the author’s hypothesis from the introduction of the study.

Second, students must be able to understand the logical connections between important

ideas, both within the context of the individual research article and in the broader context

of experimental design and the scientiﬁc literature. Students should, for example, be able to

identify the author’s research question, the logic of how the experimental design can address

that question, and how the author’s conclusions relate to both.

Scaﬀolded learning, drawing on the metaphor of scaﬀolding in building construction,

describes how students can perform more complex tasks than they would otherwise be

capable with the help and guidance of someone more knowledgeable (Wood, Bruner, & Ross,

1976). Scaﬀolded approaches can make learning more tractable for students by changing the

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 4

task to be more accessible and better aligned with the student’s current abilities (Quintana

et al., 2004; Rogoﬀ, 1990; Vygotsky, 1978). One important feature is that it provides support

and instruction about both how to do the task well and also why it should be done that

way (Hmelo-Silver, 2006; Hmelo-Silver, Duncan, & Chinn, 2007). In the context of reading

empirical articles, scaﬀolded approaches should aim to reduce the complexity of the speciﬁc

task given at the time (e.g., reading a speciﬁc empirical article to learn the course content)

while, at the same time, helping students develop more general reading, comprehension, and

critical evaluation skills.

Two scaﬀolding approaches for reading primary sources have typically been emphasized

(e.g., Kershaw, Lippman, Fugate, 2018). The ﬁrst approach is to adapt an article to be

more easily understood by the student while at the same time maintaining the article’s basic

argument structure and content (e.g., Yarden, 2009). For instance, one could simplify an

article by removing technical jargon and unnecessary information or add text structures

and signalling techniques more common in textbooks like headers, highlighting, ancillary

side-notes, etc. The adaptation approach has been used in biology and the life sciences

and provides some beneﬁt for critical reading (Baram-Tsabari & Yarden, 2005). There are

however, associated costs in time and eﬀort to the instructor in producing adapted articles;

and although adaptations should improve comprehension of the adapted articles, it is not

clear whether adapted materials help student develop more general strategies for tackling

empirical articles.

A second approach is to design instructional materials that scaﬀold reading compre-

hension without altering the original article (Reiser, 2004; Varnhagen & Digdon, 2002).

Supplemental materials can give students the opportunity to engage with complex materials

that would otherwise be beyond their current abilities. For example, supplementing readings

with organizational signaling materials like overviews (Lorch and Lorch, 1996), outlines or

graphical organizers (Easterday, Aleven, & Scheines, 2007; Nussbaum & Schraw, 2007; see,

also Newell et al., 2011; Scheuer, Loll, Pinkwart, & McLaren, 2010) have been shown to

improve comprehension. Instructional materials can include prompts, directing students’

attention toward key features of a text (Hmelo-Silver et al., 2007) or organizational outlines

to teach students explicitly about the structure in the text.

The instructional material approach also scales well to articles in general or speciﬁc.

For example, Van Lacum et al. (2014) provide an outline for the general identiﬁcation of the

rhetorical moves used to present an argument in the Introduction of a scientiﬁc article. Sego

and Stuart (2016) created a set of 22 general, open-ended questions that could apply to a

number of empirical articles (e.g., “What was/were the independent variables?”). Bachiochi

et al. (2011) developed a more targeted approach tailoring a set of questions to a speciﬁc

article. Using these kinds of materials students tend to improve in their ability to dissect

empirical articles (Bachiochi et al., 2011; Sego & Stuart, 2016).

In the current study we evaluated one such scaﬀolded approach known as the

“QALMRI” method. The QALMRI is an instructional tool for teaching students to identify

the key conceptual information necessary for the comprehension and critical evaluation of

empirical articles (Brosowsky & Parshina, 2017; Kosslyn & Rosenberg, 2003). “QALMRI”

is an acronym for Question, Alternatives, Method, Results, and Inferences (Table 1; see

Brosowsky and Parshina, 2017 for a more detailed description and student instructional

materials). It is a general framework to help students identify and draw connections between

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 5

2 3 4 5

QALMRI Session

QALMRI Score

Figure 1 . Results from Study 1 showing participant performance on the QALMRI summaries.

The dashed line shows performance on the ﬁrst QALMRI session (prior to the instruction

intervention) and error bars represent 95% conﬁdence intervals around the mean.

the research questions being asked, how the researcher tried to answer them, and the

implications of the answer. The QALMRI methods diﬀer from some of the earlier described

approaches in that it takes a broader perspective, asking students to examine the bigger

picture. That is, rather than asking students to list facts about an article (e.g., “who was

the author?”, “how many participants were in the study?”), it asks students to think about

the empirical article in terms of its broader goals (i.e., asking and answering questions).

Additionally, the QALMRI method is meant to be a simple outline that is easily remembered

and can be applied more broadly to any empirical article.

The aim of the current study was to evaluate the use of the QALMRI method in

undergraduate psychology courses. In study 1, the QALMRI method was introduced and

used in an introductory psychology course. In study 2, the QALMRI method was introduced

and used in two third-year experimental psychology methods courses.

Study 1

Methods

Participants

Participants were 28 students enrolled in the Introductory Psychology course at the

College of Staten Island of the City University of New York. Participants completed the

QALMRI assignments as part of their course requirements.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 6

Materials

Participants completed QALMRI and written summaries of ﬁve empirical articles

throughout the semester. Articles were chosen on the basis of their topics, such that the

article content was relevant to the current course and textbook material. To align the

technical level of the articles with the reading level of the ﬁrst-year students, articles were

selected from the Psi Chi Journal of Undergraduate Research (e.g., Sego and Stuart, 2016;

see Appendix A for a list of chosen articles).

Procedure

Throughout the semester, students completed ﬁve QALMRI sessions. During each

session students read an empirical article and complete a QALMRI summary in-class. The

instructor reviewed the article and QALMRI content providing additional instruction and

demonstration on how to ﬁnd the relevant information in the article. Critically, students

did not receive any prior instructions on how to use the QALMRI before the ﬁrst session.

They were simply given the article and QALMRI outline and told to read the article and

answer the questions. The ﬁrst session therefore serves as a baseline (pre-intervention) to

compare subsequent performance.

QALMRI responses were evaluated per section on a pass/fail basis. That is, students

received a pass (for the “Alternatives” section, for instance) if they provided a response that

resembled the answer key and received a score out of a possible 6. QALMRI responses were

scored by two independent raters and any conﬂict reconciled between the raters. Given

students could provide a range of possible (and possibly correct) responses to the “broad

question”, we only assess responses for the “speciﬁc question”.

Results and Discussion

We were primarily interested in understanding how well students in an introductory

course would learn to use the QALMRI as a method for summarizing empirical articles.

To that end, we compared performance on the QALMRIs prior to any instruction (pre-

intervention) to the subsequent four QALMRIs. Participants who failed to complete at least

3 out of the 5 QALMRIs were excluded, which, in this case, did not exclude any participants.

We used a linear mixed eﬀects model with QALMRI session as a ﬁxed eﬀect and participant as

the random eﬀect (linear mixed models use maximum likelihood estimation and is capable of

handling missing data). The resulting analyses showed that participants scored signiﬁcantly

higher on the QALMRI after receiving the instruction intervention on each of the following

sessions (see Figure 1): Session 2,

(100

59) = 6

27 ,

p < .

001; Session 3,

(100

97) = 8

73 ,

p < .

001; Session 4,

(100

56) = 11

28 ,

p < .

001; and Session 5,

(100

56) = 11

51 ,

p < .

001.

Study 2

Methods

Participants

Participants were students enrolled in third-year experimental psychology methods

courses at the College of Staten Island of the City University of New York (N = 38) and

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 7

2 3 4 5

QALMRI Session

QALMRI Score

Figure 2 . Results from Study 2 showing participant performance on the QALMRI summaries.

The dashed line shows performance on the ﬁrst QALMRI session (prior to the instruction

intervention) and error bars represent 95% conﬁdence intervals around the mean.

Brooklyn College (N = 85). Participants completed QALMRIs as part of their required

course assignments.

Materials

Five empirical articles were chosen to align with the course content. In this case,

articles were chosen based primarily on their experimental design and increased in diﬃculty

according experimental design (e.g., single factor with two groups, repeated measures designs,

two by two factorial designs, etc.). See Appendix B for a complete list of the chosen articles.

Procedure

The procedure is largely the same as Study 1. In addition to the in-class QALMRI

summaries, the College of Staten Island students subsequently completed a written summary

of the same article. Students performed in-class replications of the experiments (using

themselves as participants) and wrote APA-style research reports detailing the attempted

replication and results. The original article was summarized in the introduction of the

research report as part of the literature review.

Results and Discussion

QALMRI performance

To evaluate QALMRI performance we compared scores on the QALMRIs prior to any

instruction (pre-intervention) to the subsequent four QALMRIs. Participants who failed to

complete at least 3 out of the 5 QALMRIs were excluded, which excluded 10 participants.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 8

0 1 2 3 4 5 6

QALMRI Score

Written Report Score

Figure 3 . Results from Study 2 showing participant performance on the QALMRI summaries

plotted against written summary scores, r = .51, 95% CI [.23, .71], t(36) = 3.56, p = .001.

We used a linear mixed eﬀects model with QALMRI session as a ﬁxed eﬀect and participant

and article as random eﬀects. We collapsed over class, as we were not interested in diﬀerences

between classes, however, we included article as a random eﬀect to take into account the

variability in diﬃculty between articles. The resulting analyses showed that participants

scored signiﬁcantly higher on the QALMRI after receiving the instruction intervention on

each of the following sessions (see Figure 1): Session 2,

39) = 3

99 ,

= 0

003; Session 3,

39) = 4

92 ,

p < .

001; Session 4,

78) = 2

57 ,

= 0

029; and Session 5,

21) = 3

34 ,

p = 0.008.

QALMRI summaries as compared to traditional written summaries

The College of Staten Island participants also completed traditional written summaries

of the same articles. We compared performance between the average written QALMRI

summary scores and found a signiﬁcant positive correlation (see Figure 3),

51, 95%

CI [

23,

71],

(36) = 3

56,

001. This result is perhaps unsurprising in that students

who do well in the written summaries also do well on the traditional written summaries.

However, it validates the QALMRI method both an instructional tool and assessment tool,

in place of more traditional summary assignments. Furthermore, it suggests the QALMRI

could be a useful tool for both scaﬀolding reading and written assignments.

QALMRI performance by item

Finally, we were interested in how we could improve and revise the QALMRI method.

Therefore, we analyzed performance across QALMRI items to determine where students

were having diﬃculty. To that end, we analyzed QALMRI scores from sessions 2 through

5 using a repeated measures ANOVA with QALMRI item (Q,A,L,M,R, and I) as the

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 9

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Q A L M R I

Average Score

Figure 4 . Results from Study 2 showing QALMRI item scores averaged across QALMRI

sessions 2 through 5.

repeated measures factor and found a signiﬁcant diﬀerence across items,

365) = 90

87,

MSE = 0.04, p < .001, ˆη

= .555, 90% CI [0.5, 0.59]. Generally speaking, performance was

worse for the Methods and Inferences sections as compared to the rest of the items.

General Discussion

In two studies we evaluated student performance using the QALMRI method to

summarize empirical articles. In general, we found that students in a ﬁrst-year introductory

course and a third-year experimental methods course learned very quickly how to use the

QALMRI after only a single instructional session. This suggests the QALMRI method

straightforward tool that students can easily adopt, and demonstrates its usefulness across

varying educational levels. We also found a strong correspondence between QALMRI

performance and later performance on a more traditional written summary assignment. This

result suggests that the QALMRI assignment could substitute for more traditional forms of

scientiﬁc literacy assessments. Furthermore, it suggests that the QALMRI could also be

used to scaﬀold writing assignments.

We also note however, that students did not improve to the same degree across all the

QALMRI items. In study 2, we found that students performed much worse, on average, on

the methods section, and to a lesser degree, the inferences section. The low scores in the

methods section of the QALMRI particularly stand out: students scored very low across

all the QALMRI sessions. The methods section contains the most factual information and

missing any of the required information could result in a score of 0. So, low performance,

may be an artifact of how we assessed the QALMRIs. However, low performance could

also indicate diﬃculty in understanding which aspects of a methods section are important.

Qualitatively, student responses varied a great deal. For instance, some students focused

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 10

Table 2: An outline of the revised QALMRI method (2020)

Questions What was the broad question being asked by this research project?

What was the speciﬁc question being asked by this research project?

Alternatives What was the author’s hypothesis?

What were the alternative hypotheses?

Logic What was the logic of the hypothesis?

i.e., if the hypothesis was true, what should we expect to happen?

Methods Brieﬂy describe the study design

What were the independent and dependent variables?

Brieﬂy describe the study procedure in everyday terms

Results What were the important results?

Inferences What did the authors conclude from their study?

How did the authors use the results to make inferences and conclusions

about the hypothesis and research question?

Adapted from Kosslyn and Rosenberg (2003); See also Brosowsky and Parshina (2017)

for a more detailed description and additional instructional materials

heavily on minor details (e.g., number of trials, detailed experimental procedures) and

missed important aspects (e.g., identifying the independent variables), while other students

presented only vague responses. The inferences section was also challenging for students.

Again, qualitatively we found that students had diﬃculty separating their own conclusions

and inferences from those of the authors. This might be the result of the vaguely worded

prompt which does not speciﬁcally indicate which it is referring to (“What inferences about

the hypotheses and questions can be made based on the results?”). A ﬁnal limitation we

found with using the QALMRI method, is its inability to deal with multi-experiment studies.

The sections simply do not have enough space and are not organized in a way that allows

students to summarize each experiment.

Given these limitations and student performance on the QALMRI, we have revised

the QALMRI (see Table 2). First, we have added more speciﬁc questions about the methods

that still generalize across studies, but provide more guidance for students. Second, we

have re-worded the inferences section to clarify that that the students should identify the

conclusions the authors are drawing from their results. Finally, we have created an alternative

multi-experiment QALMRI which simply bookends the original QALMRI with the broader

questions the entire article is asking and the inferences the author’s draw from the entirety

of the article. Instructors can include as many experiment-speciﬁc QALMRIs as needed.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 11

References

Anisfeld, M. (1987). A Course to Develop Competence in Critical Reading of Empirical

Research in Psychology. Teaching of Psychology, 14 (4), 224–227. https://doi.org/10.

1207/s15328023top1404_8

Bachiochi, P., Everton, W., Evans, M., Fugere, M., Escoto, C., Letterman, M., & Leszczynski,

J. (2011). Using Empirical Article Analysis to Assess Research Methods Courses.

Teaching of Psychology, 38 (1), 5–9. https://doi.org/10.1177/0098628310387787

Baram-Tsabari, A., & Yarden, A. (2005). Text genre as a factor in the formation of scientiﬁc

literacy. Journal of Research in Science Teaching: The Oﬃcial Journal of the

National Association for Research in Science Teaching, 42 (4), 403–428.

Britt, M. A., Richter, T., & Rouet, J.-F. (2014). Scientiﬁc Literacy: The Role of Goal-

Directed Reading and Evaluation in Understanding Scientiﬁc Information. Educa-

tional Psychologist, 49 (2), 104–122. https://doi.org/10.1080/00461520.2014.916217

Brosowsky, N. P., & Parshina, O. (2017). Using the QALMRI Method to Scaﬀold Reading

of Primary Sources . In A. Schwartz, C. Shane-Simpson, P. J. Brooks, & R. Obeid

(Eds.), How We Teach Now: The GSTA Guide to Student-Centered Teaching (pp.

312–328). Society for the Teaching of Psychology.

Carkenord, D. M. (1994). Motivating Students to Read Journal Articles. Teaching of

Psychology, 21 (3), 162–164. https://doi.org/10.1177/009862839402100309

Chamberlain, K., & Burrough, S. (1985). Techniques for Teaching Critical Reading. Teaching

of Psychology, 12 (4), 213.

Christopher, A. N., & Walter, M. I. (2006). An Assignment to Help Students Learn to

Navigate Primary Sources of Information. Teaching of Psychology, 33 (1), 37–63.

https://doi.org/10.1207/s15328023top3301_9

Coil, D., Wenderoth, M. P., Cunningham, M., & Dirks, C. (2010). Teaching the Process of

Science: Faculty Perceptions and an Eﬀective Methodology. CBE—Life Sciences

Education, 9 (4), 524–535. https://doi.org/10.1187/cbe.10-01-0005

Dasgupta, A. P., Anderson, T. R., & Pelaez, N. (2014). Development and Validation of

a Rubric for Diagnosing Students’ Experimental Design Knowledge and Diﬃcul-

ties. CBE—Life Sciences Education, 13 (2), 265–284. https://doi.org/10.1187/cbe.

13-09-0192

Day, J. D. (1986). Teaching Summarization Skills: Inﬂuences of Student Ability Level and

Strategy Diﬃculty. Cognition and Instruction, 3 (3), 193–210. https://doi.org/10.

1207/s1532690xci0303_3

Easterday, M. W., Aleven, V., & Scheines, R. (2007). Tis better to construct than to receive?

The eﬀects of diagram tools on causal reasoning. Frontiers in Artiﬁcial Intelligence

and Applications, 158, 93.

Gillen, C. M. (2006). Criticism and Interpretation: Teaching the Persuasive Aspects of

Research Articles. CBE—Life Sciences Education, 5 (1), 34–38. https://doi.org/10.

1187/cbe.05-08-0101

Halonen, J. S., Buskist, W., Dunn, D. S., Freeman, J., Hill, G. W., Enns, C., & others.

(2013). APA Guidelines for the Undergraduate Psychology Major (Version 2.0).

Washington, DC: APA.

Hmelo-Silver, C. E. (2006). Design principles for scaﬀolding technology-based inquiry.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 12

Collaborative Learning, Reasoning, and Technology, 147–170.

Hmelo-Silver, C. E., Duncan, D., Ravit Golan, & Chinn, C. A. (2007). Scaﬀolding and

Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner,

Sweller, and. Educational Psychologist, 42 (2), 99–107. https://doi.org/10.1080/

00461520701263368

Karcher, S. J. (2001). Student reviews of scientiﬁc literature: Opportunities to improve

students’ scientiﬁc literacy and writing skills. In S. J. Karcher (Ed.), Tested studies

for laboratory teaching (Vol. 22, pp. 484–487).

Kardash, C. M., & Noel, L. K. (2000). How Organizational Signals, Need for Cognition,

and Verbal Ability Aﬀect Text Recall and Recognition. Contemporary Educational

Psychology, 25 (3), 317–331. https://doi.org/10.1006/ceps.1999.1011

Kershaw, T. C., Lippman, J. P., & Fugate, J. M. B. (2018). Practice makes proﬁcient:

Teaching undergraduate students to understand published research. Instructional

Science, 46 (6), 921–946. https://doi.org/10.1007/s11251-018-9456-2

Kosslyn, S. M., & Rosenberg, R. S. (2003). Fundamentals of Psychology: The Brain, the

Person, the World. Allyn; Bacon.

Latour, B. (1987). Science in Action: How to Follow Scientists and Engineers Through

Society. Harvard University Press.

Levine, E. (2001). Reading your way to scientiﬁc literacy. Journal of College Science

Teaching; Washington, 31 (2), 122–125.

Newell, G. E., Beach, R., Smith, J., & VanDerHeide, J. (2011). Teaching and Learning

Argumentative Reading and Writing: A Review of Research. Reading Research

Quarterly, 46 (3), 273–304. https://doi.org/10.1598/RRQ.46.3.4

Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration

in students’ writing. The Journal of Experimental Education, 76 (1), 59–92.

Oldenburg, C. M. (2005). Use of Primary Source Readings in Psychology Courses at

Liberal Arts Colleges. Teaching of Psychology, 32 (1), 25–29. https://doi.org/10.

1207/s15328023top3201_6

Pennington, H. (1992). Excerpts from Journal Articles as Teaching Devices. Teaching of

Psychology, 19 (3), 175–177. https://doi.org/10.1207/s15328023top1903_15

Price, D. W. W. (1990). A Model for Reading and Writing about Primary Sources: The

Case of Introductory Psychology. Teaching of Psychology, 17 (1), 48–53. https:

//doi.org/10.1207/s15328023top1701_12

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., . . . Soloway,

E. (2004). A scaﬀolding design framework for software to support science inquiry.

The Journal of the Learning Sciences, 13 (3), 337–386.

Reiser, B. J. (2004). Scaﬀolding complex learning: The mechanisms of structuring and

problematizing student work. The Journal of the Learning Sciences, 13 (3), 273–304.

Rogoﬀ, B. (1990). Apprenticeship in thinking: Cognitive development in social context.

Oxford university press.

Sanchez, R. P., Lorch, E. P., & Lorch, R. F. (2001). Eﬀects of Headings on Text Processing

Strategies. Contemporary Educational Psychology, 26 (3), 418–428. https://doi.org/

10.1006/ceps.2000.1056

Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. M. (2010). Computer-supported

argumentation: A review of the state of the art. International Journal of Computer-

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 13

Supported Collaborative Learning, 5 (1), 43–102.

Sego, S. A., & Stuart, A. E. (2016). Learning to Read Empirical Articles in Gen-

eral Psychology. Teaching of Psychology, 43 (1), 38–42. https://doi.org/10.1177/

0098628315620875

Suppe, F. (1998). The Structure of a Scientiﬁc Paper. Philosophy of Science, 65 (3), 381–405.

https://doi.org/10.1086/392651

Suter, W. N., & Frank, P. (1986). Using Scholarly Journals in Undergraduate Experimental

Methodology Courses. Teaching of Psychology, 13 (4), 219.

Taylor, K. K. (1983). Can College Students Summarize? Journal of Reading, 26 (6), 524–528.

Van Lacum, E. B., Ossevoort, M. A., & Goedhart, M. J. (2014). A Teaching Strategy

with a Focus on Argumentation to Improve Undergraduate Students’ Ability to

Read Research Articles. CBE—Life Sciences Education, 13 (2), 253–264. https:

//doi.org/10.1187/cbe.13-06-0110

Van Lacum, E., Ossevoort, M., Buikema, H., & Goedhart, M. (2012). First Experiences with

Reading Primary Literature by Undergraduate Life Science Students. International

Journal of Science Education, 34 (12), 1795–1821. https://doi.org/10.1080/09500693.

2011.582654

Varnhagen, C. K., & Digdon, N. (2002). Helping students read reports of empirical research.

Teaching of Psychology, 29 (2), 160–165.

Vygotsky, L. (1978). Interaction between learning and development. Readings on the

Development of Children, 23 (3), 34–41.

Ware, M. E., Badura, A. S., & Davis, S. F. (2002). Using student scholarship to develop

student research and writing skills. Teaching of Psychology.

Wood, D., Bruner, J. S., & Ross, G. (1976). The Role of Tutoring in Problem Solving*.

Journal of Child Psychology and Psychiatry, 17 (2), 89–100. https://doi.org/10.1111/

j.1469-7610.1976.tb00381.x

Yarden, A. (2009). Reading scientiﬁc texts: Adapting primary literature for promoting

scientiﬁc literacy. Research in Science Education, 39 (3), 307–311.

Zieﬄer, A. S., & Garﬁeld, J. B. (2009). Modeling the growth of students’ covariational

reasoning during an introductory statistics course. Statistics Education Research

Journal, 8 (1), 26.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 14

Table 3: A multi-study QALMRI

Questions What was the broad question being asked by this research project?

Study 1:

Questions What was the speciﬁc question being asked in this study?

Alternatives

What was the author’s hypothesis?

What were the alternative hypotheses?

Logic What was the logic of the hypothesis?

i.e., if the hypothesis was true, what should we expect to happen?

Methods Brieﬂy describe the study design

What were the independent and dependent variables?

Brieﬂy describe the study procedure in everyday terms

Results What were the important results?

Inferences What did the authors conclude from their study?

How did the authors use the results to make inferences and conclusions about the

hypothesis and research question?

Study 2:

Questions What was the speciﬁc question being asked in this study?

Alternatives

What was the author’s hypothesis?

What were the alternative hypotheses?

Logic What was the logic of the hypothesis?

i.e., if the hypothesis was true, what should we expect to happen?

Methods Brieﬂy describe the study design

What were the independent and dependent variables?

Brieﬂy describe the study procedure in everyday terms

Results What were the important results?

Inferences What did the authors conclude from their study?

How did the authors use the results to make inferences and conclusions about the

hypothesis and research question?

Inferences

What did the authors conclude from the results of all the studies and how do those

conclusions they relate to the broad question?

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 15

Appendix A

Articles from Study 1

Hosek, C. A., Phelps, B. J., & Jensen, J. (2004). Average sleep times

among undergraduate college students. Psi Chi Journal of Undergraduate

Research, 9, 57-61.

Colleen, F., McKenna, M.B., Brooks, C.I., O’Brien, J.P. (1996). Self-esteem

and health-related behaviors in college students and their parents. Psi

Chi Journal of Undergraduate Research, 1, 71-77. 3..Smith, C. (2007).

Social Anxiety and Public Self-Consciousness as Predictors of Appearance

Accuracy. Psi Chi Journal of Undergraduate Research, 12(2).

Snyder, A.J. (2003). The eﬀect of text organization and alignment on

memory. Psi Chi Journal of Undergraduate Research, 8, 121 - 127.

Beals, J. M., Arruda, J. E., & Peluso, J. P. (2002). The eﬀects of language

proﬁciency on test performance. Psi Chi Journal of Undergraduate Research,

7, 155-161.

TEACHING UNDERGRADUATE STUDENTS TO READ EMPIRICAL ARTICLES 16

Appendix B

Articles from Study 2

Brooklyn College

Song, H., & Schwarz, N. (2008). If it’s hard to read, it’s hard to do:

Processing ﬂuency aﬀects eﬀort prediction and motivation. Psychological

Science, 19 (10), 986-988.

Nairne, J. S., Thompson, S. R., & Pandeirada, J. N. (2007). Adaptive

memory: Survival processing enhances retention. Journal of Experimental

Psychology: Learning, Memory, and Cognition, 33 (2), 263.

Stoet, G., O’Connor, D. B., Conner, M., & Laws, K. R. (2013). Are women

better than men at multi-tasking? BMC Psychology, 1 (1), 18.

Raz, A., Kirsch, I., Pollard, J., & Nitkin-Kaner, Y. (2006). Suggestion

reduces the Stroop eﬀect. Psychological Science, 17 (2), 91-95.

Yin, R. K. (1969). Looking at upside-down faces. Journal of Experimental

Psychology, 81 (1), 141.

College of Staten Island

Stroop, J. R. (1935). Studies of interference in serial verbal reactions.

Journal of Experimental Psychology, 28, 643-662.

Roediger, H. L., III, & McDermott, K. B. (1995). Creating false memo-

ries: Remembering words not presented in lists. Journal of Experimental

Psychology: Learning, Memory, & Cognition, 21(4), 803-814.

Masters, M. (1998). The gender diﬀerence on the mental rotations test is

not due to performance factors. Memory and Cognition, 26(3), 444-448.

Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing

pairs of words. Journal of Experimental Psychology, 90, 227-234.

Treisman, A. M., & Gelade, G. (2000). A feature-integration theory of at-

tention. In Yantis, S. (Ed.), Visual perception: Essential readings. (pp. 347-

358.) Psychology Press.