Prof. Ralph teaches empirical research methods around the world. He is available to lead short courses, workshops, summer schools, and junior faculty consortiums on the following topics. Courses can be tailored to undergraduate, postgraduate, or faculty and are available in-person or remotely. To inquire about hosting an event, please email Prof. Ralph directly.

Overview of Empirical Methods for Computing Research

This course begins with a brief analysis of the shift from mathematical proof to empirical research in computing research, followed by an overview of common empirical methods (controlled experiments, benchmarking, case studies, systematic reviews, etc.). Students will pose research questions relevant to their theses and assess the appropriateness of various empirical methods for addressing these questions. Much of the course consists of helping students determine which method(s) would be best for them and explore critical success factors for each of these methods. Philosophical implications of method choice will be described. This course is most appropriate for students in areas of CS that emphasize empirical testing, e.g., software engineering, human-computer interaction, CS education.

Measurement and Construct Validity

While the methodological rigor of computing research has improved considerably in the past two decades, quantitative computing research is hampered by immature measures and inattention to theory. Measurement—the principled assignment of numbers to phenomena—is intrinsically problematic because observations and measures are theory- and value-laden. For example, counting bugs necessitates a theory of bugs or else how do you know what is and is not a bug (theory-laden)? Moreover, one person’s “bug” could be another’s “feature” because different people value different things (value-laden). Ignoring such issues produces an endless stream of junk data. In contrast, embracing realist, model-based measurement approaches permits more reliable, valid conclusions with less effort, fewer resources, and smaller samples.  This course is suitable for junior faculty, postdoctoral researchers and graduate students with some research experience or training, from Software Engineering, Human-Computer Interaction, Management Information Systems, CS Education and similar fields. Practical advice regarding common measurement challenges will be provided and content will be tailored for participant’s topic areas.

Theory Development and Evaluation

Computing research struggles to produce, maintain and evolve a cumulative body of knowledge due to sustained inattention to theory. This course covers the different kinds of theories used in computing: variance theories, process theories and taxonomies. Qualitative, quantitative and mixed-methods approaches to generating and evaluating theories are discussed. Students are asked to brainstorm theoretical lenses appropriate to their research, or frame their research as a theoretical question.

Qualitative Research

A typical introduction to qualitative methods including case study, grounded theory, phenomenology, ethnography, and action research. Students practice interviewing and document analysis (i.e. qualitative coding). Philosophical foundations of qualitative research are discussed. This is a good option for students who’ve already had exposure to quantitative research methods but are new to qualitative methods.