Respect for the limits of quantification

Many of our deepest and most celebrated scientific laws and insights have depended crucially on quantitative approaches, from the Newton’s laws to recent advances in quantitative biology. I believe that quantitative approaches hold much promise for the future and we should aim to improve our ability to quantify and deeply understand nature as much as we can.

The phenomenal successes of quantitative approaches in some scientific realms have inspired scholars to apply pseudo quantitative approaches to some realms that we do not yet know how to quantify. Some prominent historical examples come to mind. Karl Marx claimed to have identified the immutable laws of history that indicated the inevitability of communism. Similarly, the early protagonists of eugenics believed that extending the beautifully quantitative laws of genetics supports their erroneous conclusions of ethnic supremacy. A more recent example is the quantitative assessment of individual scientific articles by the journal impact factor (JIF) of the journals in which the articles are published. This unfortunate practice has been brought to a new level by the innumerate advertisement of JIFs with 5 significant figures. Technically this is a case of false precision since the precision of the input data is often below 5 significant figures (Wiki article with explanation). This innumeracy is only the tip of the iceberg. The inadequacy of the approach (articulated in the San Francisco Declaration on Research Assessment) is a much more serious problem.

Making important decisions based on pseudo quantitative data and analysis is more than innumeracy; it is a HUGE problem. Being quantitative means respecting the limits of what can be quantified and limiting claims to what can be substantiated. The fact that somebody can somehow compute a number does not mean that the number should be taken seriously. Regretfully, sometimes pseudo quantitative approaches prove mightily influential.

In one of my favorite essays of all times, Marc Kirschner makes a compelling argument why pretending that we can evaluate the significance of scientific results contemporaneously with the scientific research is fool’s errand. We cannot, certainly not with 5 significant figures. Pretending otherwise is more than innumeracy; it is a HUGE problem. We should try by all means possible to improve our ability to assess and quantify research quality but let’s not pretend we can do that at present. We cannot, certainly not with 5 significant figures.

In the meantime, before we have the verdict of time and the community in the rear view mirror, we should be more humble. We should be honest about what we can and what we cannot do. Rather than pretending that we quantify what we cannot, we should quantify at least what we can: the reproducibility of the results. We should also avoid increasing the inequality of grant money distribution based on the shaky contemporaneous estimates of significance and impact. After all, the best and the brightest should prove their brilliance with the quality of their thought and research not with the quantity of their research output. At least historically, the scientific achievements that I judge as great and significant tend to be great in quality and not great in quantity. They depended more on creative thought, inspiration and serendipity than on great concentration of resources based on flawed, albeit quantitative, statistics.

Maybe in our efforts to be quantitative and objective, we have focused on what can be easily quantified (quantity) and pretended that it reflects what really counts (quality). A measure of humility and realignment is in order if we are to preserve and further the research enterprise.

Tell me about the science, not the prizes!

The more we focus on awards and advertise career building, the more we attract people seeking awards and glamorous careers, and the bigger the burden on the peer review system.

The independent and critical assessment of data and of analysis is at the core of the scientific method. Yet, the rapid growth of the scientific enterprise and the explosion of the scientific literature have made it not only hard but impossible to read, think deeply, and assess independently all published papers, or even the subset of all papers relevant to one’s research. This is alarming. It has alarmed many people thinking of creative and effective ways to evaluate the quality of scientific research. This exceptionally hard endeavor has attracted much needed attention and I am hopeful that progress will be made.

In this essay, I suggest another approach to alleviating the problem, starting with two related questions: Why is low quality “science” written up and submitted for publication and what can we do to curb such submissions? These questions touch upon the poorly quantifiable subject of human motivation. Scientists have a complex set of incentives that include understanding nature, developing innovating solutions to important problems, and aspirations for social status, prestige and successful careers. All these incentives are part of our human nature, have always existed and always will. Yet, the balance among them can powerfully affect the problems that we approach and the level of evidence that we demand to convince ourselves of the truths about nature.

In my opinion, scientific culture can powerfully affect the incentives of scientists and in the process harness the independent thought of the individual scientists — not only the external reviewers — in raising the standards and rigors of their own work. I see a culture focused on prizes and career building as inimical to science. If the efforts of bright young people are focused on building careers, they will find ways to game the system. Many already have. As long as the motivation of scientists is dominated by factors other than meeting one’s own high standards of scientific rigor, finding the scientific results worthy of our attention will remain a challenge even with the best heuristics of ranking research papers. However, if “the pleasure of finding things out” — to use Feynman’s memorable words — is a dominant incentive, the reward, the pleasure, cannot be achieved unless one can convince oneself of the veracity of the findings. The higher the prominence of this reward intrinsic to scientific discovery is, the lower the tendency to game the system and the need for external peer review.

A scientific culture that emphasizes the research results — not their external reflections in prizes and career advancement — is likely to diminish the tendency to use publishing primarily as a means of career advancement, and thus enrich the scientific literature of papers worthy our attention. We know that racial stereotypes can be very destructive and we have lessened their destructive influences by changing the popular culture. How can we apply this lesson to our scientific culture to focus on the critical and independent assessment of research and thus lessen the negative aspects of career building and glamour seeking?

A great place to begin is by replacing the headlines focused on distinctions and building careers with headlines focused on factual science. For example, the “awards” section in CVs, faculty profiles and applications for grants and tenure-track faculty-positions can be replaced by a “discoveries” section that outlines, factually, significant research findings. Similarly, great scientists should be introduced at public meetings with their significant contributions rather than with long lists of prizes and grants they received. One might introduce Egas Moniz as the great Nobel laureate and Dmitri Mendeleev as a chemist with few great awards. Much more informatively, however, one should introduce Egas Moniz as an influential protagonist of lobotomy and Dmitri Mendeleev as the co-inventor of the periodic table of elements. 

Admittedly Mendeleev and Moniz are prominent outliers but they are far from being the only examples of discrepancy between awarded prizes and scientific contributions. Still the worst aspect of focusing on prizes, grants and career building is not only the reinforcement of political misattribution of credit; far worse is the insidious influence of excessive focus on prizes and career building on the scientific culture. The more we celebrate awards, the more we attract people seeking awards and glamorous careers, and the bigger the burden on the peer review system.

We should celebrate research and examples like those of Einstein and Feynman, not the prizes that purport to reflect such research. Focusing on the work and not the prize would hardly diminish the credit. Indeed, the Nobel prize derives its prestige from scientists like Einstein and Feynman and not the other way around. A prize may or may not reflect significant contributions and we should be given the opportunity to evaluate independently the contributions. We should focus on the scientific contributions not only because critical and independent evaluation of data is the mainstay of science but because it nourishes constructive scientific culture, a culture focused on understanding nature and not gaming the system. Only such a culture of independent assessment can give the best ideas a chance to to prevail over the most popular ideas.

The next time you have a chance to introduce an accomplished colleague, respect their contributions with an explicit reference to their work, not their prizes. With this act of respect you will help realign our scientific culture with its north star: the independent and critical evaluation of experiments, data, ideas, and conceptual contributions.

An edited version of this opinion essay was published by The Scientist Accomplishments Over Accolades

On the Wings of a Seagull

The time had stopped. Watching the chaotic ballet of ripples in the lake, I was thinking of the deep connections that unify seemingly disparate phenomena. The direction of the wind was fluctuating from second to second, switching the direction of the ripples almost in synchrony with my thoughts. In the Platonic tradition, I have always searched for and marveled at the deep unifying connections between different aspects of life and the world around me.

Still, I never suspected that the same fundamental ideas and symmetries that lie at the heart of nuclear physics, quantum chromodynamics and a huge variety of critical phenomena in physical systems would apply to biochemical networks, would govern the finely orchestrated regulation of tens of thousands of genes, would make the miracle of life possible. I wanted to savor the glory of the moment, not to be conscious of time.

Nonetheless, my thoughts were interrupted by a couple of seagulls soaring majestically in the air. The same fickle blows that caused the ripples to lurch back and forth were converted into a graceful flight. Through occasional adroit movements of their wings, the seagulls were gliding elegantly in the air. They were harnessing the erratic wind for their graceful ascent.

My immediate impulse was to emulate the seagulls, not only literally – because of my desire to glide through the air and soar toward the sky – but also figuratively, because of my desire to harness the power of the winds and storms in my life that I cannot even hope to control. I felt an irresistible urge to make the most of what I have with the grace and mastery of nature, with an insightful understanding of and respect for the deep principles shaping the universe. The seagulls went sky-high without controlling the wind, an example worth admiration! An example of gentleness and power synergistically united into a beautiful ascent.

Basking in the light of these thoughts, I sensed the blueprint of the erratic wind imprinted onto the random Hamiltonian matrices of atomic nuclei. Similarly to the seagulls, an elegant flap of wings (chiral symmetry) is enough to transform the randomness into a robust system-characteristic behavior. This is a recurrent, pervasive pattern that charms with uncanny, magnetic appeal: Complex systems transcending stochastisity and shaping randomness into exquisite creative dynamics. I know of nothing that can rival the grace and power of nature to leave freedom for creativity and still direct and orchestrate the large-scale dynamics. The world is a beautiful picture painted with gentle and adroit touches of the nature’s brush.

Beauty

I open my eyes and see beauty, a gratifying perception that inspires respect and reverence, that directs my actions and endows them with meaning and purpose. What is it? It is beyond me to verbally articulate it and yet I feel compelled to share the emotion and power that beauty engenders in me, even try to understand its essence; it is so easy to trivialize and so hard to capture the deep meaning of beauty with its multifarious forms, form a drop of dew and a falling autumn leaf to a majestic mountaintop, form a bird song to a symphony orchestra, from a snowdrop to a beautiful woman.

What is the unifying feature among all of these? Is it just a spatiotemporally organized neuronal activity, the response of my brain to fundamentally different stimuli or is it a multifaceted diamond presenting itself in hologramic mosaic of images. Is it the dynamic interplay of elegant simplicity creating astounding diversity, an intricate world that puzzles with its variety and charms with its few powerful unifying principles?

I can hardly approach these questions without biases! Being educated in the Western scientific tradition, I seek and marvel at the immutable forms from that sublime world that Plato called superlunary and Aristotle considered quintessential. Now we call it with different names but more often just tacitly assume that it exists; the implicit assumption that we can explain diverse phenomena in terms of universal principles is the very foundation behind much of modern science. It was the assumption that Newton made in inferring the laws of motion from astronomical observations of “the heavenly bodies” and then boldly generalized the principles to all terrestrial phenomena. We have indeed pursued this seductive assumption with a remarkable success and found common unifying principles behind ostensibly disparate phenomena.

I would go further and claim that even our ability to perceive the world testifies to its structure, to a set of statistical dependencies that consistently and reliably guide us through the dynamical interplay of thousands of variables; when I fail that is because my intuition cannot grasp the sharp turns in the enormous phase-space of my life, the high-order combinatorial nonlinearities that paint the dynamical portraits of life, portraits that often appear quite impressionistic while carrying their distinctive characters: Fractal portraits imprinted by stochastic influences and yet unmistakably reflecting their underlying intrinsic dynamics. I perceive and marvel and still cannot articulate the awe-inspiring grandeur that animates my life, the beauty that surrounds me — it is a feeling, a wonderfully gratifying perception!