On the cover in @CellCellPress : cell-free DNA fragmentome in patients with liver cancer. The red hexagons are cancerous liver cells. The red pieces of DNA are cancerous DNA from the cancer cells and released into the bloodstream upon cell death.

happy to see this paper finally peer-reviewed & out. it raises & formalizes a profound point: are we actually using informative benchmarks to assess & understand Foundation Models on gene expression data? anybody who has ever pre-trained or benchmarked a bio FM from scratch understands that this is hard. for example, one of the most intuitive ways in which a gene expression model can be assessed is the arithmetic mean: by simply averaging the gene expression signal in the training cells, do we get a smaller error to the true expression for the cells in test than the model’s predictions on test? but this mean is often taken over really many genes, most of which don’t matter at all for the phenotype we want to measure. so why do people still look at so many genes then? the answer is that, oftentimes, we don’t have a real measurement of the phenotype we are truly interested in. we are not really interested in “will this gene go up or down”. we are interested in the effect of that perturbation on an (alive) organism, and even more simplified, we are interested if it can help cure a certain disease. but such an endpoint is so far off that we do not (and could not) measure it in a controlled setting that could causally link it to the perturbation. long story short, we often don’t know what else to measure, so we average across everything readily available. another approach is to look at key marker genes, or pathways. but then we have the inverse problem: the task becomes overly specific and well biology is noisy. also false negatives are super popular in biology… so we end up without much actual information. this paper proposes a new metric to help us better answer such benchmarking questions. this is a very important contribution in the direction of more profoundly evaluating FMs in bio. I am looking fwd for more progress in this area. one last thought is that this matches the conceptual trajectory we’ve seen in LLMs in the past 2 years: from over-indexing on the metrics, to benchmaxxing, to understanding that what we care about is a specific usecase hard to measure and that a list of metrics is hardly meaningful for the models’ true potential. bio FMs still have a long way to go to rise to their full potential, but i am really excited about the immense rate of progress. nature.com/articles/s41587-0…