This page contains all of my notes related to scientific methods and research. My goal is currently to develop a strong foundation for scientific thinking.
Quantitative Methods (Course 1 Notes):
Source(s): Statistics and Research Methods: University of Amsterdam (Specialization) Course 1 Quantitative Methods
Vocab:
- Empirical: Verified by observation/experience rather than theory/logic
- Scientific Method Principles:
- Empirically Testability: Physical Evidence or Observations that either support or contradict hypothesis
- Replicability: Repeatedly confirmed
- Objectivity: A clear and explicit definition of procedure and methods should produce the same results.
- Transparency: All procedures, definitions, and assumptions are publicly shared.
- Falsifiability: There is a possibility for there to exist evidence that contradicts the hypothesis.
- Logic Consistency: The logical basis of a hypothesis and its subsequent conclusions are logically sound.
- Observation: General relation between properties without explanation
- Hypothesis: General relation between properties with explanation
- Law: Super precise description of relation between properties; typically mathematical formula
- Theory: Overarching explanation of many related phenomena
- Universals: Abstract and unobservable properties (e.g. love, gravity)
- Particulars: Specific examples/instances of universal (abstract properties) (e.g. object falling, cat that doesn’t like to be pet sitting next to you)
- Negative Case Analysis: Seeking contradictory evidence for a hypothesis.
- Summative: Focus on the outcome of a program
- Formative: Focus on the on the process of a program
- Construct: An abstract concept that cannot be directly measured.
- Variable: A specific aspect/instance of a construct.
- Operationalization: A specific and concrete way/method to measure a variable.
- Measurement: representation of the relations between things using numbers
Concepts:
Philosophical Realism: The belief that the external world exists independently of human thought.
Essentially schools of thought from which the scientific method emerged.
Epistemology: How can we know it? What is accessible and can be accessed? Study of knowledge
(1.) Rationalism: Physical world is an imperfect reflection of (ideal) reality. Knowledge about reality can be attained through reasoning. Innate knowledge. (Nature?)
(2.) Empiricism: Physical world is the reality. Knowledge about reality can be attained through observation via the senses. Sensory knowledge. (Nurture?)
(3.) Constructive Empiricism: Acknowledges unobservables and states that the truth/falsity cannot be determined as claims are provisional (evidence can always come up).
Ontology: What is there to know? What exists or is real?
(1.) Materialism: The world is made of matter, including our mental processes (brain interacting in physical world).
(2.) Idealism: The world is in our minds. The world is a mental constructions/representations.
(3.) Realism: The world is made of matter, but unobservables/universals do exist. Platonic Realism states they exist on a separate abstract plane. Scientific Realism states they can be used to support claims
(4.) Nominalism: The world is made of matter, but unobservables/universals do not exist. Gravity is simply a term that represents something falling.
Research Designs and Internal/External Validity
Threats to Internal Validity:
1. Maturation: Time can serve as an alternative explanation (for example, aging, healing, growth, development, etc.)
2. Mortality: Participant drop out can skew results (or affect outcomes).
3. History: Something could happen in the world that affects the participants in the study (a study on health is affected by the covid outbreak) and therefore, affecting the outcome of the study.
4. Instrumentation: The instrument could be modified (different pre-test vs. posttest) or have an error that affects the outcome.
5. Testing: Testing can change participant perception (sensitize them to the questions, accidentally reveal the group they are in, etc.) which could affect the outcome.
6. Regression to the mean: A statistical phenomena where previously extreme values tend to be closer to the mean/avg in subsequent tests (skewing differences between pre-test and post-test).
7. Bracketing:
8. First Order Constructs:
9. Second Order Constructs:
Designs
Purpose of a research design is to prove that we are not only measuring what we intended to, but we “prove” (or provisionally prove) our hypothesis.
Three important components are:
1. Manipulation: Being able to control the cause implementation allows us to study it.
2. Comparison: Being able to compare the cause implementation allows us to understand when it’s not there or when circumstances change (relativity).
3. Randomization: Being able to apply it to a randomized sample allows us to reduce any biases/confounding variables that would serve as alternative explanations.
a. Random Selection vs. Random Assignment: Random Selection pertains to the process of choosing participants randomly (choosing participants randomly). Random Assignment pertains to the process of choosing participant’s group randomly (organizing already chosen participants randomly).
Types of Designs
- Pretest Posttest Control Group:
a. Randomly Assigned.
b. Two Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
Purpose: To determine if the treatment had an effect. - Solomon Four Group Design:
a. Randomly Assigned.
b. Four Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
iii) One group gets treatment and posttest.
iv) One group gets posttest.
Purpose: To determine if the treatment and testing had any effects. - Placebo Control Group Design:
a. Randomly Assigned.
b. Pretest and posttest optional.
c. Three Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
iii) One group gets pretest, placebo, and posttest.
Purpose: To determine if the treatment and perception had any effects. - Repeated Measures:
a. Only one group of participants; order of treatments is randomly assigned.
b. All participants receive same set of treatments in randomized order.
c. Can be compared within subjects.
Typically:
i) One group gets pretest, treatment, posttest, another treatment, posttest, . . .etc.
ii) One group gets pretest, treatment, posttest, another treatment, posttest, . . .etc.
iii) etc. . .
Purpose: To determine if the treatments had any effects within the same group and compare treatments within group. - Switching Replications Design:
a. Randomly Assigned.
b. Two Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
c. Then roles are switched.
i) Control then gets treatment, and posttest.
ii) Experimental then gets posttest.
Purpose: To determine if the treatment had any effects compared to control, replicate treatment, and determine long term treatment effects. - Multiple Experimental Group with One Control Group Design:
a. Randomly Assigned.
b. More than 2 Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest, treatment 2, and posttest.
iii) etc. . .
ii) One group gets pretest and posttest.
Purpose: To determine if the treatment had any effects and compare different treatments. - Factorial Design:
a. Randomly Assigned.
b. Multiple independent variables, each with levels.
c. Treatments are the number of levels of each independent variable multiplied.
d. Groups:
i) One group gets pretest, combination (treatment), and posttest.
ii) One group gets pretest, combination 2, and posttest.
iii) One group gets pretest, combination 3, and posttest.
iv) etc. . .
Purpose: To determine if the treatments had any effects and compare the combined effects of each group. - Quasi-experimental Design:
a. NOT Randomly Assigned.
b. Non-equal groups.
c. Sometimes no manipulation/comparison.
d. Investigates causal relations.
Purpose: To determine if the treatments had any effects within the constraints of the groups. - Non-equivalent Group Design:
a. Naturally occurring groups; not randomly assigned.
b. Two Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
Purpose: To determine if the treatment had any effect. - Double Pre-test Design:
a. Naturally occurring groups; not randomly assigned.
b. Two Groups:
i) One group gets pretest, posttest, treatment, and posttest.
ii) One groups gets pretest, posttest, treatment, and posttest.
Purpose: To determine if the treatment and testing had any effect. To also determine maturation. - Proxy Pre-test Design:
a. Different measures for pre-test/posttest; used another measure.
b. Two Groups:
i) One group gets pretest, treatment, and posttest.
ii) One groups gets pretest and posttest.
Purpose: To determine if the treatment had any effect and save resources. - Multiple Time-Series Design
a. Multiple tests over a long Period of time.
b. Two Groups:
i) One group gets test, test, test, test, treatment, test, test, test, test.
ii) One groups gets test, test, test, test, test, test, test, test.
Purpose: To determine if the treatment had any effect long term and also determine maturation if any. - Switching Replications Quasi-Experimental Design
a. Naturally occurring groups; not randomly assigned.
b. Two Groups:
i) One group gets pretest, treatment, and posttest.
ii) One group gets pretest and posttest.
c. Then roles are switched.
i) Control then gets treatment, and posttest.
ii) Experimental then gets posttest.
Purpose: To determine if the treatment had any effects compared to control, replicate treatment, and determine long term treatment effects.
Correlational Designs
- Cross Sectional: Measures a section of the population at a given time
- Time Series: Measures one individual at several points in time
- Panel: Measures a group of individuals at several points in time
Other Designs
- Case Study: One group or one individual
- Evaluation Research: A type of applied research that evaluates the effectiveness of a policy/program
- Intervention Studies: A type of experimental research that evaluates the effectives of a treatment for individuals
- Validation Research: A type of research that evaluates quality of an instrument
Levels of Measurement
- Nominal: Whether or not things are the same. (Categorical)
- Ordinal: There is an order for things.
- Interval: No meaningful zero value and differences between values that are interpreted.
- Ratio: A meaningful zero value.
Qualitative Methods (Course 2 Notes):
Vocab:
- Naturalism: Research in natural settings.
- Normative: Opinion based.
- Verstehen: Deep understanding.
- Gestalt perception: The whole (of actions) is more than the sum of its parts (acts).
Ontology in Social Settings:
1. Objectivism/Realism: There is a social reality independent of human thought that does not rely on people for it to exist.
2. Constructivism: There is a social reality that is a not independent of human thought that is constructed by people. While people individually interact with one another, these interactions become shared meanings across groups.
3. Pragmatism: Social reality is best understood by its usefulness.
4. Subtle Realism: The observer is part of the social reality they observe, but the social reality exists independently of human thought. Therefore, the knowledge acquired is limited/error-prone by the fact that the observer is a part of it.
Epistemology in Qualitative Research:
1. Positivism: Gain knowledge through senses (empiricism), deductive and inductive (theory making and testing), and objectivity. Scientific vs normative statements (uses scientific ones).
2. Interpretivism: Meaning is subjective and interpretation of the world varies from individual to individual.
Hermeneutics:
The process of understanding the interpretation/meaning of a text as well as the text itself. In the social sense of the idea, this means understanding the cultural/social context surrounding an individual to understand better the individual and the world in which they exist. The main goal of hermeneutics is to interpret meaning. It can be circular (Hermeneutics Circle) where understanding the individual informs the understanding of the world and vice versa.
Phenomenology:
The study of phenomenon as the subjective and conscious experience that it is for people. It requires a description of the actual experience, an interpretation of this experience in relation to other experiences, and an analysis on its meaning. First order constructs are the phenomenon as seen and experienced by the participant first hand. Second order constructs are the phenomenon and participant experiences as seen by the researcher (interpretation).
Pragmatism
Knowledge is valued if it is useful and has effects/consequences in the world. Knowledge should serve as a solution for problems that serve a purpose for society. Truth is what best works in practice consistently and is relevant to the real world.
Types of Observation:
– Ethnography: Researcher is part of a social setting for a long period of time
Main Method: Participant Observation
Other Methods: Direct Observation, Unobtrusive Observation, Open interviews, Biographical Interviews, Focus Group Interviews, Document Analysis
Objective: To understand the people in the study
Content: Both method and written products
A kind of fieldwork.
Participant Observation:
1. Types of Data:
– Field notes, Pictures, Audio, Written, Documents
– Naturally Occurring data (excluding small amount of reactivity)
– In-depth data
– Contextualized data -> Broad within a context (social and historical)
– Also provides context to survey results.
2. Important Elements:
a. Participant:
– Background
– Skills
– Impression Management -> depends on the research and situation. Defined role that helps navigate social situations. Affects access and reactivity. Be more naive/dumber than you are in order to learn.
b. Observation:
– Watching
– Listening
– Smelling
– Feeling
– Tasting
– Posing Questions
Other:
– Writing
To conduct participant observation, personal background matters in the case of reactivity. Personal skills also come into play which ties into both access and reactivity.
Observing participant vs. Participating observant
Different data when you are simply observing the social situation vs. when you are personally interacting with it.
Observation
Using your senses to gain information?
Focus is essential but it could mean selection and thus, exclusion.
Paradigms:
1. Simmel -> Focuses on forms of interaction and social types
2. Goffman ->Focuses on how people act, interact, and form relationships to determine meaning
3. Lofland -> Focuses on city and public/private
4. Crapanzano -> Focuses on self-observation and ethnography (the self as a member of society)
5. Garfinkel, Sacks -> Focuses on how individuals’ interactions shape everyday life (how individuals create social order)
Levels/Steps of Observation
(Spradley, 1980)
1. Descriptive Observations
General description of group, area, etc. Creating broad stokes of characteristics.
Nine Dimensions of Social Situations:
1. Space
2. Actor
3. Activity
4. Object
5. Act
6. Event
7. Time
8. Goal
9. Feeling
Grand Tour: Broad descriptions using one dimension
Mini Tour: More specific description combing two (or more?) dimensions
2. Focused Observations
Answering research questions.
Outcomes: Taxonomies, categories, and relationships.
3. Selective Observations
Focused on comparisons and nuances by looking for more evidence or even contradictory evidence. Relations are tested.
Dewalt and Dewalt (2002):
1. What happens and why?
2. Regular vs. Unsual?
3. Variations & Exceptions
4. Similar examples/situations, systematic observations, change over time to support/contradict existing hypothesis or theories.
The Problem with Focus
1. Confirmation Bias
When something is brought into focus, it’s easier to forget or neglect the other elements. Because of this, a statement can be affirmed without remembering the contradictory evidence.
2. Inattentional Bias
You are not looking for something you are not looking for. Because you are looking for something specific, you end up missing out on other elements in view.
3. Change Bias
When you are focusing on something else, you miss changes happening in/for things outside of the focus.
Privatizing Public Space (Lofland)
Entrance Sequences: (some)
1. Check for readiness
a. Adjust clothing and/or hair or look at themselves in the mirror
2. Read the room
a. Quick and Covert eye scan
b. Do an action slowly to give time to scan the room
c. Do an action while quickly scanning the room
d. In groups, letting the “lead” do the reading
3. Positioning
a. B-line Tactic to sit in empty spot
Waiting Styles: (some)
1. Sweet Young Thing: Focusing on a task (reading, phone, etc.) while aware of own position and others
2. Nester: Spreads/Adjusts personal belongings/props around own position.
3. Investigator: Investigates environment/objects that are usually unimportant.
Principles: Social rules in public spaces.
1. Minimize Expressivity
2. Minimize Body Contact
3. Look before you sit
a. Determine comfortable space between seats
4. Minimize eye contact
5. When in doubt, Flee
a. Confrontation is generally not the way to go
Field Notes
What you note depends on focus, structure, topic/research, view point (ontological vs epistemological)
Structured Observation
1. Code System (a guide to follow)
2. Clear focus
3. Simple
4. Types of behavior that you are looking for and how to assign the behavior to a specific category (interpretation rules)
5. Categories need to be mutually exclusive
Unstructured Observation
- Inscription
- Notes while in the field
- Mental notes
- Jotting (scratch notes)
- Photos
- Videos
- Audio recordings
- Notes while in the field
- Description
- Full field notes after the event/writing reflectively
- Descriptions of observations
- Minutes of conversations/meetings
- Situation descriptions
- Interview summary
- Full field notes after the event/writing reflectively
- Transcription
- Interview transcriptions
- Interaction transcriptions
- Audio transcriptions
- Video transcriptions
- Other
- Comments, asides
- Memo’s
- Reflections
- Diary/log
- Theoretical memo
- Methodological memo
Organization is key in order to avoid creating sloppiness in data.
You have to allocate a certain amount of time to take notes if needed. Keep your appointments. Prepare and review data collection. Reflection is key.
Plan the kind of field notes you will take to provide yourself with a consistent method of data collection (to help you keep data collection structured).
How to Structure?
You can organize by date, type of material, theme, people, groups of people, and etc. You can use card boxes or even have a software on your device.
Observation is a combination of recording and interpretation. Example: Winking vs Twitching vs Fake wink
Thin Description: The baseline observation (contraction of eye in terms of winking)
Thick Description: In-depth and contextual description in order to convey meaning (winking is to secretly convey a message or fake winking is to amuse people)
To avoid inattentional bias, you need to be careful in how you interpret. If you see a specific type of behavior that is typically done by a certain type of person/category of behavior, then use that lens. Example: The same person standing super close to different unassociated groups of people and directly looking at their purse/wallets seems like a pickpocket, so see them as such.
Good Practices in Qualitative Research
Interpret the data with flexibility and creativity.
Definitive concepts in Quantitative Methods:
1. Imposes theory on social world
2. Common features of phenomenon
3. Indicators and Variables
Theory led operationalize approach where theory is divided into questions and divided further. You define the idea and then measure it.
Sensitizing concepts in Qualitative Methods:
1. General sense of reference and guidance
2. Discovery of varied forms of phenomenon
3. Gradually narrowing it down
Iterative knowledge production where you go back and forth and refine theory based on findings and apply theory to the social world.
Serendipitous Findings: Accidental findings, unanticipated, anomalous, and strategic
The Serendipity Pattern: Unexpected finding leads to unexpected path to new hypothesis
Triangulation:
1. Methodology/Data collection: (e.g. Interview and Observation)
2. Theory: Compare theories and fit our own problem
3. Researchers/Observers/Coders: Multiple observers
4. Respondents/Informants: Multiple
1. Can be used to find the closest truth.
2. Can be used for indefinite positioning.
3. Can be used to identify complementary information.
4. Epistemological dialogue or juxtaposition?
If context bound, then you can’t compare two scenarios. Testing/effecting/observer can come into play.
Purpose: To verify, falsify, or deepen ideas
Abduction: If A is true and C would also be true, then one might suspect that C is true as well.
Can turn into conspiracy because it relies on a guess. You have to pair it with induction and deduction in order to verify it.
Conspiracies rely on wrong induction and wrong deductions, cherry picking details, bending the story to fit a narrative, and rebuttal is a cover-up.
The Balance of Context and Details
A lot of detail can good, but sometimes too much. Detail should reveal the nuances of a phenomenon that add to the overall idea. Too much detail can shift focus away from the big picture and can create noise.
Context is important for understanding the importance/weight of actions and behaviors. Too much context can be unnecessary and can take away from the key behavior observed. Where does context begin and end?
Look for Contradiction
Deviant Cases:
1. Provide additional support to existing theory/hypothesis
2. Require modification of theory (something unexpected) of existing theory/hypothesis
3. Is simply an exception
Comprehensive Treatment: Showcase all data which shows nuance and depth, but also transparency is essential for truth
Reflexivity (How you as a researcher co-constructs reality):
1. Personal
a. Confession regarding the research. How researchers inform the research. Purpose is to downgrade authority which can also be claiming authority.
2. Method
a. As a reminder for the working hypothesis, procedures, interpretations.
b. Gut feelings showcase accountability.
c. Audit trails allow you to be transparent. (Choices you make)
3. Theory
a. Display of theoretical orientations, assumptions, prejudices.
b. You may not know your own blind spots.
How many cases are enough?
Generalization is not towards a population like a survey design.
Which cases are enough?
Generalization towards theory (cases selected based on relevance, variation, or deviance).
Transferability: You can use thick descriptions to transfer processes. You need to understand what you are transferring and what you are transferring to.
Which details do you select?
Theoretical sampling:
a. Select cases
b. Collect data
c. Organize
d. Analyze and Interpret
e. Select new cases
When are you done?
Theoretical Saturation: When your theoretical concepts are filled. No additional data can’t be found to create or fill categories.
Three forms of saturation:
1. Redundancy
a. New data does not lead to new information.
b. Saturation or sufficiency?
2. Variation
a. You try to fill each case/cell in the table.
b. Want to fill the range of cases.
3. Theory
a. Observing what exists and finding all cases that are theoretically possible because of it.
Criteria for Determining if Research is “Good Enough”
Quantitative
Validity
1. Measurement Validity
2. Internal Validity
3. External Validity -> Generalizations?
4. Ecological Validity -> Account for real life?
Reliability
1. Stability
2. Internal Reliability
3. Inter coder Reliability
Replicability
Qualitative Research
Mason (1996)
1. Rigour
2. Quality
3. Wider Potential
LeCompte en Goetz (1982)
1. External Reliability (replicability) -> theoretical replicability
2. Internal Reliability (inter coder reliability)
3. Internal Reliability (partly causality and measurement) -> how good is the interpretation from data.
4. External Validity (Generalizability)
Guba and Lincoln – Trustworthiness
1. Credibility -> triangulation
2. Transferability -> transfer conclusions to other case studies
3. Dependability -> researcher choices
4. Confirmability -> which side are we on? Researcher should reflect values.
Authenticity
1. Fairness
2. Ontological
3. Educative
4. Catalytic
5. Tactical
Hamersley 1992 – A midway position
Researcher constructs reality
Reformulated validity
– Empirical account must be plausible
– Have access to credibility of researcher’s truth claims
– Adequacy of evidence as true representation
Relevance
– contribution the study makes to the field
Criteriology
Jevaria Onymous 