Foundations for a Statistical Mindset

Preface

How we think about data about how we think

Not Another Teen Movie Psych Stats Book

Every university program in the psychological and brain sciences requires its students to take a statistics class at some point. The students dutifully comply in order to get their degree, though some with more enthusiasm than others.

In my own psychological training, years ago, the only thing I remember of my undergraduate stats class was that I didn’t really want to be there. It felt like a book of magic spells I was supposed to memorize in order to be allowed to do what I was actually in interested in, studying the basics of why people do what they do.

It wasn’t until half way through graduate school when I enrolled in a statistical modeling class, and things finally made sense. The logic of what statistical models mean, why they are useful, and how they are deeply relevant to testing scientific theories (indeed, are the theories) was finally clear. And importantly, I learned how to ask statistical questions relevant to my interests, not just compute statistical answers to questions I may or may not be interested in. Not only did I finally understand statistics better, I became more passionate about it.

I don’t think this is an uncommon experience for psychology students. I have been consulting on and teaching data analysis methods to undergraduate students for many years now, and very frequently new students ask me “what kind of test can I use on my data?” while looking at a drop down list from a stats software menu. When I ask them to tell me more about their research, they list details about their variables but give little indication of a specific question they’re trying to answer with those variables. Sometimes they don’t even have a quantifiable research question, and are just designing datasets around a memory that analysts are supposed to calculate associations and significance somehow.

It is also common in the wider psychological research community to see fundamental problems with the ways people perform and interpret their data analyses. Despite the fact that statistics has been required as part of undergraduate and graduate training for a century, researchers regularly misspecify their research questions, make conclusions that are not warranted by the evidence, or just generally spin their wheels with analyses that have no hope of advancing actionable theory about the mind and brain.

I think this lack of understanding stems in part from the way statistics is traditionally taught to psychology students. It is often taught as a collection of tools that provide answers. There are a lot of different tools that provide a lot of different answers, so an introductory curriculum tries to check many different tools off the list that are easily derivable and thus easy for an intro student to compute the answers. But this curriculum never really clarifies why these answers? Why compare two groups? Why test this difference against 0? Statistician David Hand describes the problem in his article “Deconstructing Statistical Questions” as:

Too much current statistical work takes a superficial view of the client’s research question, adopting techniques which have a solid history, a sound mathematical basis or readily available software, but without considering in depth whether the questions being answered are in fact those which should be asked.

In other words, an emphasis on a long list of statistical tests tells students that these are the answers they should be pursuing. But it doesn’t train them to ask their own questions, turning their thoughts into statistically testable hypotheses. The result is a constrained sense of imagination regarding what types of research designs and hypotheses are done, or a mismatch between the statistics that people run and what conclusions they try to draw from them.

My gamble is that we can teach psychological statistics in a way that trains good statistical questioning from the beginning. By defining different estimands and operationalizations in messy data, students learn to turn verbal research questions into quantifiable data phenomena. By specifying models of data generation processes, they learn to be explicit and precise about their expectations and beliefs for theoretical mechanisms. And by generating data outcomes through numerical methods and simulated sampling, students are empowered to pose and test hypotheses that go beyond the null hypothesis. This approach backs off of the mathematical derivations of simple tests that dominate the early lessons of a traditional statistics class, but I believe this alternative method inspires more persistence and passion for statistics as a tool to solve personally-relevant problems. Once the “why” foundation is in place, further analytic training in the “how” and better theory building is motivated.

I am not alone in this thinking, and this book makes reference to many other excellent resources that take such an approach. However, most of these resources are pitched for advanced students, written to help them overcome misconceptions they learned previously. I haven’t seen as many good resources written for introductory students, aimed at instilling modern statistical instincts from the beginning.

That is the genesis of this book. Readers should not expect it to be a thorough reference for all statistical modeling approaches or a deep dive into the mathematical derivations of modeling tools. However, I hope it can function as a sort of statistical “stem cell,” giving students foundational ways of thinking about data mechanisms and statistical questions.

Core Ideas of this Book

With that background in mind, these are the book’s main learning goals:

• be specific: about your research focus, question, measurements, population, hypotheses
• there is uncertainty, variation in a variable we want to know about
• information from other variables can help us reduce that uncertainty (model building)
• research questions focus on different reasons for doing this: documenting the association (description), reducing the uncertainty as much as possible (prediction), understanding how important predictors change uncertainty in the outcome (explanation)
• different levels of specificity for hypotheses: any sort of uncertainty reduction (NHST), comparing different options for least uncertainty (model comparison), making point predictions

And its strategy to achieving them:

• Organized around types of questions/needs, not specific tools
• Model-based rather than collection of tools – build understanding. Leaves out some things that might otherwise appear in an intro stats class like chi-square tests, but builds a foundation that more flexibly prepares for future research possibilities with modeling.
• De-emphasize hypothesis testing
• Interpretation more than calculation
• Doing stats, with code
• Learning stats with domain-specific guidance

Who this Book is For

Undergraduates or early grad students who wish to read and conduct their own research. Brain & behavior topics (neuro, psych, cog sci, behavioral science, etc.); non-academics who wish to be more critical consumers of cognitive & behavioral research.

Intended Workflow

Read, practice inline code, do problem sets, keep this for reference.

Assumed Knowledge

Mathematics up through algebra, familiarity with intro psych content

Disclosures & Acknowledgements

Chapter vector images were created by Claude Sonnet 5 from author’s written prompts about intended style. Javascript code for interactive widgets in some chapters was also written by Claude Sonnet 5 with parameters and formatting tweaked by the author. The remainder of the code and all of the text in this book was written solely by the author.

I am grateful to Mary Peterson, Miriam Schwyck, Monica Thieu, and Elisa Baek for their encouragement and helpful comments on drafts of this book. I am also grateful for my undergraduate students at Pomona College enrolled in PSYC158 for being good-natured guinea pigs in early versions of these lessons.