Often, while reading a paper on populations and how different human or archaic groups contributed to their genetics, I wonder about the validity of the computer program tools used by the authors of these papers.
Basically these are computer programs that sift through certain data and apply the border conditions defined by the authors, and then just do the numbers
As an engineeer I am aware of the power and usefulness of advanced statistics. And it is clear that f2, f3, and f4 statistics are great tools in theory, but when they come out of a black box as an output based on some unknown program, can anthropologists, and archaeologists be certain of their validity? Just as I am not trained in their base field, they are not trained in advanced statistics. Mind you, in University I had very tough statistics courses and had to understand both the theory and the practical aspects for using them. I did the numbers, not a program, and had to interpret the data myself.
Today I read an interesting paper: Robert Maier, Pavel Flegontov, Olga Flegontova, Ulaş Işıldak, Piya Changmai, David Reich (2023) On the limits of fitting complex models of population history to f-statistics eLife 12:e85492. https://doi.org/10.7554/eLife.85492. It looks into these questions and reaches some interesting conclusions.
The authors are promoting a new tool, the "findGraphs tool within a software package, ADMIXTOOLS 2, which is a reimplementation of the ADMIXTOOLS software with new features and large performance gains." In the paper, they point out the caveats in current tools. Below are excerpts from this paper:
- "Conclusions
Sampling AG [admixture graph]space is a useful method for modeling population histories, but finding robust and accurate models can be challenging. As we demonstrated by revisiting a handful of published AGs and re-analyzing the datasets used to fit them, f-statistics are usually insufficient for identifying uniquely fitting AG models, making it necessary to incorporate other sources of evidence. This provides a challenge to previous approaches for automated model building" - "A challenge for fitting AG [admixture graph] models is that they are often not uniquely constrained by the data, with many providing equally good fits to the f2-, f3-, and f4-statistics used to constrain them within the limits of statistical resolution."
This means that the preconceived notions of the authors define which model is the one they like most, and may exclude other models! - "As we show in our discussion of case studies, the simple models explored with an exhaustive approach can lead to misleading conclusions about population history because not including additional populations can blind users to additional mixture events that occurred (and whose existence is revealed by examining data from additional populations). Furthermore, models with additional admixture events that are qualitatively different to the best-fitting parsimonious graph and that capture the true history, will sometimes be completely missed when constraining the number of gene flows."
Again, the choices of the authors of a paper influence the outcome, and their conclusions! - For manually built AGs, the sitution is similar, they are based on the intuition, knowledge, and why not, the biases of the builders: "Most AGs in the literature have been constructed manually... often acknowledging the existence of alternative models by presenting plausible models side-by-side, and this approach has been the basis for many claims about population history... A strength of this approach is that it takes advantage of human judgment and outside knowledge about what graphs best fit the history of the human or animal populations being analyzed. This external information is powerful as it can incorporate non-genetic evidence such as geographic plausibility and temporal ordering of populations or linguistic similarity, or other genetic data such as estimates of population split times, or shared Y chromosomes, or rejection of proposed scenarios based on joint analysis of much larger numbers of populations than can reasonably be analyzed within a single AG. Thus, while manual approaches explore many orders of magnitude fewer topologies than automatic approaches often do, they still may provide inferences about population history that are more useful than those provided by automatic approaches. These methods’ strength is also their weakness: by relying on intuition, following a manual approach has the potential to validate the biases users have as to what types of histories are most plausible (these may be the only types of histories that will be carefully explored). This can blind users to surprises: to profoundly different topologies that may correspond more closely to the true history, and we discuss examples of this in the Results section."
Maier et al's paper then looked into several published articles and sifted the data in a different way, with surprising outcomes: "we also identified many additional graphs that fit the data not significantly worse than the published ones. In every example, some of these graphs have topologies that are qualitatively different in important ways from those of the published graphs. Features such as which populations are admixed or unadmixed, direction of gene flow, or the order of split events, if not constrained a priori, are generally not the same between alternative fitting models for the same populations... for all of the publications except one (Shinde et al., 2019), there are alternative equally-well-or-better-fitting graphs..."
In other words they found different admixture graphs with the same statistical stregnth as the ones published in several papers, but with different proportions of admixture, different populations, gene flow directions, etc.
They give many examples and detail the differences they note. I chose one in particular as it mentions a population often used in relation to the peopling of America, Mal'ta:
"Sikora et al. came to the following striking conclusion relying on the "Western" AG (Table 2): the Mal’ta (MA1_ANE) lineage received a gene flow from the Caucasus hunter–gatherer (CHG) lineage. However, in our findGraphs exploration this direction of gene flow (CHG → Mal’ta) was supported by two of the 29 topologies, and the opposite gene flow direction (from the Mal’ta and East European hunter–gatherer lineages to CHG) was supported by the remaining 27 plausible topologies (Figure 4—source data 3). The highest-ranking plausible topology (Figure 4c) has a fit that is not significantly different from that of the simplified published model with six admixture events (p-value = 0.392). We note that the gene flow direction contradicting the graph by Sikora et al. was supported by published qpAdm analyses (Lazaridis et al., 2016; Narasimhan et al., 2019), and qpAdm is not affected by the same model degeneracy issues that are the focus of this study. Considering the topological diversity among models that are temporally plausible, conform to robust findings about relationships between modern and archaic humans, and fit nominally better than the published model, we conclude that the direction of the Mal’ta-CHG gene flow cannot be resolved by AG analysis."
The following graph is an example of the original Sikora AG and the one suggested by the authors as "nominally better fitting."
Population admixture graphs, originally published (Left), better fit (Right). Online
These remarks are concerning, the wrong flow of genetic input is serious!
Closing Comments
So my fears about the validity of these graphs is confirmed. The way they are created depends on the ability, biases, preconceptions, and willingness to adhere to previous findings. These then come together and shape the outcome which then conforms to these preconceptions. There is a high risk of Confirmation Bias!
And black boxes that provide answers after you provide an input require human criteria to assess the reasonability of the answer. I remember a professor at University when I was studying Industrial Engineering in the late 1970s. We all had our Casio scientific calculators and could calculate numbers with many decimals, while in the past you needed logarithms or slide rules to run calculations. He said, before you do the numbers you have to have an idea of what the answer is going to be, what use is finding an answer with five decimals if you input the wrong numbers and you are off target by an order of magnitude? (i.e. you calculated 3.567844, yet the answer should be somewhere between 35 and 36). Common sense based on sound theoretical and practical knowledge are paramount.
Tools are only as good as the craftsman that uses them. See a rather basic course for users of these programs on this webpage it tells them which data to input and how to use an F-statistics tool. And this paper on how to interpret the outputs.
Patagonian Monsters - Cryptozoology, Myths & legends in Patagonia Copyright 2009-2025 by Austin Whittall ©
No comments:
Post a Comment