Approaches to Handle Subjectivity

This article is part of a series: Subjectivity in Earth Scinece Data.

Follow reading here

Since the 1970s, psychologists developed a growing interest in studying individual differences in human cognition, judgment and decision making [Aczel et al., 2015] [Berthet, 2021]. Various approaches have been developed striving to enable the quantification of subjectivity in information retrieval and subsequent decision making. In the following we review some approaches that gained popularity in Earth Sciences and beyond focusing on semantic uncertainty associated to information retrieval from data.

The Apparently Naïve Approach – Trying to Make the Impossible Possible

An apparently naïve approach is to ask a human agent to quantify the uncertainty or trustworthiness of the information retrieved. In fact, this approach appears like a circular loop and is certain to deliver uncertain results. It tries to push the inability of a human agent to quantify the precision and accuracy of the method followed when retrieving information one level to the back by providing information achieved by an absolute or comparative self-assessment of the uncertainty of the information retrieval methodology followed (e.g., [Clarke, 2004],[Wilson et al., 2021]). This can be realized by adhering to standardized protocols trying to homogenize the assessment methodology used. Whatever the outcome of the self-assessment procedure will be, it comes with ontological uncertainty, even when expressed quantitatively on a numerical scale. In analogy, such approaches can also be used for geoscientific site assessment under a number of subjective perceptions, e.g., such as rareness on regional or national level, in a geomorphosite assessment ([Pereira et al., 2007]).

Nevertheless, such self-competency assessment approaches are popular and may have their justification, since they focus on a homogenization of the methodology followed by human agents to make their results comparable and methodologically compliant. Such approaches are rooted in science and in fact mimic some objectives of sciences but narrowed to a very specific field of investigation. For example, geoscience students experience a homogenization of their geoscientific skills and methodology by learning current state-of-the-art methodology of their scientific domain. Individual methodology is increasingly replaced by current state-of-the-art methodology which turns them from laymen into experts in their scientific domain. Despite this training, there is still room for individual differences and subjectivity when, for example, employing geological methodology, which may become visible in different geological label assignment to a rock or different ideas about the geological evolution of a region. However, without the common training, the results of the individual human agents would likely be more diverse since the divergence in used methodologies might potentially be larger. Over time, such an approach can reduce the ontological uncertainty associated with scientific methods and insights, as it is assumed to do as a scientific method in general, and thus indeed help to develop increasingly realistic estimates of uncertainty.

Expert Elicitation – Randomizing Subjectivity to Turn it into Aleatory Uncertainty

Expert elicitation randomizes the subjective impact on information retrieval from data based on the acceptance that an individual human agent cannot realistically quantify the uncertainty related to it. Expert elicitation is a process used to gather and fuse expert opinions on the optimal information retrieval from data, particularly when the data are band-limited or incomplete. It is a scientific consensus method related to a Monte Carlo assessment of the randomness inherently present in a human agent’s information retrieval methodology [Slottje et al., 2008],[Morgan, 2014]. There are a number of subtypes of this method, all of which aim to minimize or quantify the uncertainties associated with this method. A general overview of how expert elicitation for uncertainty quantification works is given in Table 2.

Table 2 General steps of an expert elicitation and problems of uncertainty assessment related to them.

Task	How to realize	Related problems
Identify purpose and scope	Clearly define the problem for which uncertainty needs to be quantified. Determine the scope of the expert elicitation process, including the specific variables or parameters of interest.	Too narrow definition of scope and parameters of interest may a priori bias the analyses towards over-optimistic uncertainty quantification
Select experts	Identify experts who have relevant knowledge and expertise in the subject matter. Ensure that the experts have a good understanding of the problem at hand and the uncertainties associated with it. If necessary, provide briefing or training.	Experts working under the same scientific paradigm might be affected by the same bias, which then cannot be revealed and considered. Common briefing of all experts on the problem and the uncertainties associated with it might introduce biases.
Elicit expert opinions	Conduct structured interviews or workshops with the experts. These interviews can be one-on-one or in a group setting. Use appropriate elicitation techniques to gather expert opinions on uncertain parameters. Encourage experts to express their opinions in terms of probabilities, ranges, or other quantitative metrics easy to compare.	Individual and social aspects, e.g., personality traits or reputation of group members, might bias the elicitation. One-on-one assessment might limit the own information retrieval methodology by having low chances to learn from others and revises the own methodology by having a scientific discourse with peers.
Aggregate expert opinions	Combine the individual expert opinions to create a consensus estimate of uncertainty. Various aggregation methods can be used, including weighted averaging, Delphi method, or Bayesian updating. Consider the credibility and expertise of each expert when aggregating opinions.	Choose of improper methodology w.r.t. weighting or fusion of opinions.
Characterize uncertainty	Use the expert-derived information to produce probability distributions or other uncertainty models for the parameters of interest. Conduct sensitivity analyses to understand how changes in expert opinions impact the results and to identify influential parameters.	Selection of improper methodology, e.g., related on invalid Gaussian assumptions or rules of large numbers.
Iterate and refine	Depending on the feedback and the availability of new information, the expert elicitation process may need to be iterated and refined.	Personality traits, such as a “competition winner” or “I have said already what I had to say” attitude might bias results towards individual opinions.
Documentation	Document the entire expert elicitation process, including the selection of experts, elicitation methods, and results.	Incomplete documentation.
Communicate results	Clearly communicate the results of the expert elicitation, including the estimated uncertainties and the basis for those estimates.	Incomplete communication of results.

The applicability of an expert elicitation requires that all experts have access to the same data or source of the data, i.e., environmental reality. The latter is necessary when human agents are involved in the data production by human sensory engagement. An example for this is geological mapping, which largely builds on the human sense of sight. Despite geologists have to go through a long training phase a variable degree of subjectivity may exist in geological label assignment. Theoretically, this could be overcome by an expert elicitation if all selected experts could access the geological field site. Practically, this is usually impossible to realize or economically impossible, particularly when mapping in remote areas. The success of an expert elicitation depends critically on the selection of the experts. Expert have acquired experience with the matter and might have undergone a training procedure trying to homogenize their methodology with those of their predecessors. In other words, at a certain point on their way to become an expert, they might have learned the same state-of-the-art methodology. If this is the case for all selected experts, they might not be able to quantify the preconceptions their expert opinions rely one, since they are not randomized over the selected experts. This bears ontological uncertainty about a proper use of the expert elicitation methodology which is out of the analysis scope of an expert elicitation.

Crowd Annotation – in Expert Elicitation for Laymen

Crowd annotation is related to expert elicitation but involves usually larger groups of individuals. The necessary level of data and methodological understanding of each human agent in the crowd is more relaxed. The crowd is composed of interested agents with a potentially variable level of expertise on the problem (e.g., [Ørting et al., 2020]). A prior training or briefing procedure may be necessary but may in turn artificially homogenize the crowd with regard to the information retrieval methodology used by their members. This could cause similar problems related to biased group member voting as known from expert elicitation. However, the huge diversity of experience and knowledge levels about the data and information retrieval methodologies usually represented in a crowd makes it in many cases more robust to uncertainty underestimation due to unrecognized methodological biases. The information retrieval methodology of individual crowd members may be prone to gross errors resulting from a mis-understanding of the data or information retrieval methodology [Ørting et al., 2020] . This may result in unreasonably large ranges of quantified uncertainty achieved by the voting’s of the crowd members. As for expert elicitation, crowd annotation requires that all crowd members have access to the same data or source of the data, i.e., environmental reality, which practically limits the applicability of the approach.

References

ABS+15

Balazs Aczel, Bence Bago, Aba Szollosi, Andrei Foldes, and Bence Lukacs. Measuring individual differences in decision biases: methodological considerations. Frontiers in Psychology, 6:1770, 2015.

Ber21

Vincent Berthet. The measurement of individual differences in cognitive biases: a review and improvement. Frontiers in psychology, 12:630177, 2021.

Cla04

S Clarke. Confidence in geological interpretation: a methodology for evaluating uncertainty in common two and three-dimensional representations of subsurface geology. British Geological Survey, 2004.

Mor14

M Granger Morgan. Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National academy of Sciences, 111(20):7176–7184, 2014.

PPCA07

Paulo Pereira, Diamantino Pereira, and Maria Isabel Caetano Alves. Geomorphosite assessment in montesinho natural park (portugal). Geographica helvetica, 62(3):159–168, 2007.

SVdSK+08

Pauline Slottje, JP Van der Sluijs, Anne B Knol, and others. Expert Elicitation: Methodological suggestions for its use in environmental health impact assessments. National Institute for Public Health and the Environment, 2008.

WWB+21

Cristina G Wilson, Randolph T Williams, Kathryn Bateman, Basil Tikoff, and Thomas F Shipley. Teaching uncertainty: a new framework for communicating unknowns in traditional and virtual field experiences. Solid Earth Discussions, pages 1–13, 2021.

OrtingDvH+20(1,2)

Silas Nyboe Ørting, Andrew Doyle, Arno van Hilten, Matthias Hirth, Oana Inel, Christopher R Madan, Panagiotis Mavridis, Helen Spiers, and Veronika Cheplygina. A survey of crowdsourcing in medical image analysis. Human Computation, 7:1–26, 2020.

Authors

Jalil Asadi