The research process and study design

I was talking to another PhD student the other day that was presenting a schedule of the work that she was planning to do over the next few months. One project that she is deeply involved in at the moment in is analyzing data that has been collected from a set of real organizations. However, she also wants to then test these findings in the lab. I thought this might be a good opportunity to talk briefly about study design and what different kinds of research exist within social science.

Ideally, the research process goes in an order vaguely like this: A researcher comes up with an idea about how the world works, the relationship between some set of variables, etc. In most branches of social science, the researcher then creates a set of predictions about how different variables will be related to one another. This is less necessary in some fields such as (non-behavioral) economics. The next thing the researcher decides is what is the best way to determine if this relationship exists. Sometimes the question itself will inform what data should be used to test for an effect. If the question is, for example, about the relationship between stock price and employee stealing, then looking at a real organization may be ideal. Once a data source is identified, the researcher collects the data and does analysis on these data. After the researcher has interpreted the results, the work will go onto the publication process, either into a journal article, book, book chapter, or conference presentation.

I work in a very small world where I have used experiments in all of my work. My experiments, though not identical, have certain elements of design that I consistently use which adds familiarity to the design process for my studies. I know the manipulations and the kinds of acceptable tasks very well. Though the specifics have taken some time in the past to work out, I don't think it took me more than a few days to design each of the studies that I have used. The longest time has always been determining the task to use. The difficulty with tasks sometimes is the balance between creating a new, novel task that the participants won't be familiar with and choosing a task that has been tried and tested by you or your colleagues.

When I looked at the other student's schedule, I was genuinely surprised that she had 3 weeks scheduled for study design. When I talked to another student, she thought that 3 weeks was just about enough time. This interaction got me thinking why I was so surprised that the student chose such a long period of time to dedicate to study design. I don't think I am overly skilled at study design, but I could be using a different definition of study design than they were.

When a lab study is designed, the major decisions that have to be made are the task, the manipulations, and the measures. My manipulations have always been rather blunt and heavily tested: employee turnover or restricting communication. The manipulation of more delicate factors, such as feelings of group belonging, of fear, or feelings surrounding the exchange of favors, are, I imagine, much more difficult and may have smaller impacts on people. There are huge literatures investigating these factors, which may actually instead the time it takes to choose a manipulation because the researcher may feel like they need to be familiar with most of the prior work. I don't mean to come off as dismissive of other work, but if you spend all of your time reading all the published literature in your area, you'll never add to that literature yourself. It is a dangerous game of academia, unless you work along narrow specialties (which has been my strategy).

Once the core vision of a study has been determined and the three decisions mentioned earlier (task, manipulation, and measure) have been chosen, the materials have to be put together. I don't typically think of this as design, but it is a necessary part of the research process. This is the phase where study materials are drafted, the specifics of the task are decided, materials are purchased, and advertisement materials are readied. Another unsung, but important aspect of this process is the writing of a script. I was fortunate to have a reader on a student project strongly suggest I write one for my first solo project and graciously provided me an example. The script lists all the actions the experimenter does to prepare for the study, all the things the experimenter says, when things occur in relationship to one another, and the timeline of the study. Writing the script always has a way of highlighting to me glaring issues with the design of the study in both a shallow sense (operalization) and a deeper (theoretical) sense.

I hope this post provides you with some insight into the nuts and bolts of the social science research processes and may provide some tips to other scientists.

Goodman

Coal miner study

One part of my graduate education that I count as one of the most fortunate was the limited amount of experience I had interacting with Professor Paul Goodman. Paul Goodman unfortunately passed away shortly after I passed my qualifying exams. Paul Goodman was an extremely interesting and committed researcher that allowed his personal feelings of justice influence the direction of his work in a very real way without allowing them to cloud the scientific process. Paul was truely one of a kind.

After Paul passed away, I spent some time talking to his wife and children as they discussed his upbringing and what motivated some of this work. From what I recall, both of his parents were liberal social activists in New England. From an early age they instilled in Paul that organizations have a responsibility to treat their employees well. Though I'm sure many other things influenced his choice of career, Paul eventually began studying the ways that employees interact with management in organizations. Paul was an avid film-maker who did a series of videos about the current state and future of work. He typically interviewed average people in industries that were changing. Many of these films can be found at a permanent collection at Carnegie Mellon's library website: http://dli.library.cmu.edu/paulgoodman/

The last two projects that I know of Paul perusing was a long-term project on the science of science teams. Though I do not know his specific motivation, scientists often apply much less social science to their organization than what we actually know. After Paul died, this project dissolved due to the cohesive power of Paul's personality disappearing. The other was a more amorphous process that I think perfectly sums up Paul's outlook on the world. He and his assistants conducted hundreds of long-form interviews asking average people what they thought the American dream was, if they strove for it, and what kind of world they wanted for their children.

Though Paul completed a lot of interesting work, what I'd like to talk about today was some of the work that came out of his multi-year coal-mining project. In this work, Paul went to coal mines in the mid-Atlantic and interviewed miners in their place of work. By that I mean underground in the mine itself. Paul told me on multiple occasions that he thought that the ability to conduct the interviews and collect data in the mine itself gave him a much more accurate perception of what it was like to work in this environment. My father, who is from Pennsylvania described to me when I was very young that my great grandfather's worked in a coal mine. This profession and the work that Paul did therefore always seemed to touch me a bit closer as I always imagined my great grandfather in the place of the miners in the papers.

The paper I would like to describe of Paul's is one that he wrote with Dennis Leyden. This work was supported by the U.S. Bureau of Mines. I can only make a guess but I think that the Bureau were interested in how the relationships between the individual workers in the mine were related to mine outcomes. Mines vary in productivity enormously and one possible reason is the kinds of relationships the individual workers have with one another.  Goodman and Leyden proposed that the mines provided a good opportunity to look at the effects of familiarity on the small teams that work together within a coal mine. (In a prior study, the researchers had already identified that an individual with little familiarity with a mine was more likely to have an accident.)

Mining crews were sets of workers doing one of three unique roles. Those roles were: the miner operator, the bolter, and the car operator. Each crew typically had a pair of people performing each role. Though each role is unique and there are skills associated with the roles, the authors argue that the specific strategies the individuals use vary from crew to crew based on personal differences and the features of the part of the mine the group is in. Though the researchers do not mention it specifically in this paper, another factor that I imagine is important is the cognitive interdependence of the individuals on one another.

Without getting into too much analytical detail, the researchers used information about which groups individual were working on to create a measure of whether individuals had worked with one another before and to what extent a given group's members were familiar with one another. Overall, the researchers found that the levels of familiarity between the group members was predictive of the overall mine productivity. They found some evidence that different kinds of familiarity mattered more than others but they felt that overall familiarity mattered more.

In rereading this paper, I found myself reminded of some other interesting work by Karl Weick on aircrews. Like I am attempting to do in this blog, Weick preferred description over analytics and wrote extremely provoking papers based on his reading and observations of real events. Weick's observation of air crews found very similar effects of familiarity on the air crews ability to perform without errors. I'm sure I will discuss some of his other work later in this blog.

Coginitive Interdependence [Deep Dive] - Moreland series - Part 3

This post describes the studies commissioned by the Army that Levine et al. explored. In Part 2, the studies on productivity were explored. In this post, I focus on the experiments about innovation.

Creativity Experiment 1 -Assigned and/or maligned (published as Choi & Levine, 2004)

The researchers then shifted away from performance as the primary variable of interest and into the effect of turnover on group innovation. These studies used a air-surveillance task that John Levine and his students have used in several papers that I know about. Groups work together to monitor the radar at a base and assign threat levels to the different radar contacts. In each of the three member groups, 2 individuals were specialists and 1 acted as the commander. The specialists essentially collect information about the radar contacts and the commander receives that information and is tasked with making a decision. There were two different strategies that could be used in the strategies to collect information that varied on whether the importance of the information the specialists collected was the same for both or whether the difficulty of getting the information was the same for both.

In the first experiment, the researchers manipulated whether the group was able to choose their strategy and how well their feedback suggested that they had performed. In the experimental setup, the group was either assigned one of the two strategies above or they were allowed to choose one. Then the group performed the task. Half the groups were told that they had performed well and the other half were told that they had performed below a passing rate. One of the specialists was then chosen and replaced with a confederate. In social psych research, a confederate is someone who pretends to be a normal participant but has been coached to act in a particular way. The newcomer then proposed that the group switch to the opposite strategy of whichever they had chosen in the first trial.

The researchers used whether the group accepted or rejected the strategy the newcomer proposed as the variable of interest. Because this could be affected by a multitude of factors, the researchers measured how committed the members were to the previous strategy, how much they liked the team, performance in the first trial, etc. The researchers found results that were inline with what they anticipated. If groups were told they failed to perform well in the first trial, they were more willing to accept the newcomers idea. The groups were also more likely to accept the idea of the newcomer if the group had not been allowed to choose their own strategy.

The researchers then did some additional analyses and proposed what led to the group's receptivity to the newcomer's proposal. The two variables the researchers proposed mediate the effect of team choice on the acceptance of the newcomer: commitment and perceived performance. If the group had a choice in their strategy, they were more committed to their strategy and they perceived their performance as better.  The researchers were fairly satisfied in these findings but they also thought that the way the newcomer proposed their innovative idea likely would have an effect on whether the group accepted it. This led to the second experiment.

Creativity Experiment 2 -An Assertive Story

This study was run very similarly to the first creativity study except that the kind of language the newcomer used was varied. As before, groups are more likely to accept the newcomer's innovation when the group was told that they had failed in the first trial. There was also what is called a statistically significant interaction. An interaction just means that whether one variable has an influence depends on another variable. When the groups were told that they had succeeded, it did not matter whether the newcomer was assertive or not, the acceptance rate was always about 45%. If the group had been told they failed, however, they were more likely to accept the ideas of the newcomer if the newcomer was assertive (~85%) versus if the newcomer was not assertive (~60%). [Note: this effect is only 'marginally significant' meaning that our confidence in the effect is not overly high.] The researchers had hoped for stronger effects but still thought this study was valuable.

Computational simulations, the shallowest dive

The last part of the technical report provides some information about a series of computational simulations that were included in this project. Very briefly, a computational simulation puts a bunch of agents into a box. Each agent represents a person, organization, etc. The agents are given some rules to live by, some of which may vary systematically (share information with another agent if they are within 2 spaces vs. share information with another agent if they occupy the same space). There is also a level of randomness that is added to the agents decisions to help simulate the real world. Simulations are becoming more and more accepted within management-type research though I am not sure how accepted they are within general social psychology.

In the series of simulations presented in the report, the authors focus on the effect of transactive memory and changes in the environment. In the first simulation, the researchers find some evidence that suggsts that the value of a transactive memory is curvilinear with the size of the group.  They found that if the group is fairly small, the difference in speed to completion of a task by the agents was about the same regardless of whether the group had a transactive memory or not. There was a definite benefit of TMS when groups were larger (between 15 and 27), but the benefit reduced for larger groups (35). I personally think that this is an artifact of how the agent's task is structured, but it does seem fairly reasonable. The last simulation suggested that transactive memory is particularly useful if the group completes multiple different kinds of tasks that are completed in alternating order. A transactive memory allows the groups to more quickly shift tasks, leading to a consistency in time to completion.

Though the studies in this report were not all successful, I found it particularly interesting. The ability to try out new ideas that this study provided also certainly helped the researchers develop their later studies and directed other researchers toward these topics.

I think this post completes my sequence on cognitive interdependence for now, though I'm sure it will crop back up :P