As the volume of medical research continues to grow at an exponential rate, clinicians and researchers face an important yet often intractable problem, how do we stay up-to-date on important findings? 

One common approach, born in the pre-internet age, is focusing largely on impact factor(1). Given time limitations, it might seem prudent to focus attention almost exclusively on esteemed or high-impact journals. However, a journal’s impact factor is largely driven by the success of their top performing articles, and thus, often it does not truly reflect the performance of an “average” article (2). More to the point, there is no doubt that many of the most influential works in orthopedics are found outside the New England Journal of Medicine (3). Ultimately it is not where a study is published, but what a study reveals that defines its impact (2).

Moving beyond impact factor, OE has aimed to leverage advances in machine learning to better predict an article’s potential impact at the time of publication. We present OE High Impact, a machine learning powered metric that serves as a better representation of an article’s potential importance than journal impact factor alone. Much of this improvement can be derived from the fact that OE High Impact makes predictions based on the content in the article itself. 

In developing our neural network, we first generated a dataset of over 20,000 orthopedic clinical research articles from publicly available data sources. As an important caveat, journal metrics and article citations present in this dataset are slightly different from the “official” metrics calculated by the Journal of Citation Reports.

We subsequently trained a neural network to predict how many citations an article would receive based on both the article’s textual content (e.g., the article title, abstract text), and metrics on the article’s publishing journal (e.g., average citations per year, published articles per year, etc.). At the heart of this process is an inhouse developed “language model” which can convert the textual content present in an article into a numerical representation. In our final out-of-sample test-set, the citation prediction model was able to achieve a 1.72-fold stronger correlation with an article’s real-world citations than journal citation metrics alone (e.g., average citations per year) (Exhibit 1).

Using the OE citation prediction model, we identified a cut-off, to discern articles which are likely to be highly cited for those that are not. We called this threshold OE High Impact. OE High Impact boasts a high precision – 72% of OE High Impact articles lie within the top 75th percent of cited articles, and 90% OE High Impact articles lie within the top 50th percentile of cited articles (Exhibit 2A). Thus, readers can be confident that OE High Impact articles are meritorious and worth reading. Notably, these articles come from a range of journals spanning the impact factor spectrum (Exhibit 2B). The median average citations per article of OE High Impact publishing journals is 3.89, illustrating that impactful research is not restricted to the pages of high-impact journals.

Instead of being purely driven by the publishing journal, the OE citation prediction model is strongly dependent on an article’s content. As a “black box”, the neural network that powers OE High Impact cannot be simply asked to report a relative risk or odds ratio to confirm the relative importance of an article’s content, or a journal’s impact factor. However, we can still garner clues as to how it makes decisions by perturbing it. We therefore conducted two sensitivity analyses. In Exhibit 3A, we mutated every article such that it was published in a high-impact journal, median impact journal, and low-impact journal, and asked how this impacted the model’s prediction on future citations. As expected, the model learned that 99th percentile journals will on average yield an above-average number of citations, while a 1st percentile journal yields below-average citations. Interestingly, a median journal impact factor was found to be sufficient in producing articles with just as many citations as 99th percentile journals.

In Exhibit 3B, we performed the reverse perturbation and artificially mutated every publication to have the same content as a highly cited paper, a median paper, and a poorly cited paper, and asked the model how content (as compared to where it was published) influenced an article’s predicted number of citations. Here we can see that the model appears to be much more sensitive to article content. A 99th percentile article is almost always predicted to produce an above-average level of citations, irrespective of journal metrics. By contrast, a median, or low-citation article tends to produce fewer than average citations. Collectively, these sensitivity analyses suggest that the OE impact model believes that what an article says, and not where it was published, defines the impact an article will have.

Finally, leveraging the treatment tagging system of OE M.I.N.D. we asked which treatment tags were associated with the highest predicted 5-year citation count. The results are summarized in Exhibit 4. Among this list are colloquial “hot” topics in orthopedics, including stem cell research, cannabis, platelet rich plasma, and decompressive laminectomy, and others.

We therefore present OE High Impact (Exhibit 5). As we launch this new feature, some ACE reports will begin to display “OE High Impact” above their title. This tag indicates our belief that the identified paper provides important, impactful data, and thus, will ultimately prove to be highly cited. It’s important to note that no measure of impact is perfect. Thus, articles that are not labelled as “high-impact” should not be construed as being of poor quality and undeserving of attention. To reiterate the words of Nobel Laureate Sydney Brenner, “What matters absolutely is the scientific content of a paper, and nothing will substitute for either knowing or reading it”. Nevertheless, in revisiting the initial problem, how can clinicians better stay up to date on research? OE High Impact offers a searchable indicator that can help readers to cut through the noise, and refine their focus to potentially innovative works.