In a Part 1 of this post I showed how behavior analysis journals compare on several metrics of journal quality. If those data raise questions about the viability of some of them, well, they should. Elsewhere I’ve commented on the dangers of dividing our discipline into too many silos, by which I mean distinct professional organizations, boards, foundations, conferences and such. A discipline should have as many of these institutions as it needs, but when there are too many they tend to overlap in mission, waste resources, and dilute the potential impact of the discipline.
When it comes our journals, remember that they compete for our discipline’s finite store of manuscripts and authors and capable folks to do the editing. If we have too many journals, it stands to reason that maybe some will not be thriving… and data in Part 1 support that assumption. On multiple objective dimensions, we have several journals that aren’t doing so well. A logical conclusion: Maybe we’d be better off with fewer journals.
But assuming that insight is correct, and you could act on it, which journals should stay and which should go?
Stronger and Weaker Journals
Let’s start by estimating how impactful our journals currently are. You’re probably familiar with the Citation Impact Factor (IF), an estimate of how much scholarly attention is attracted per article in a given journal. The IF has a number of shortcomings, including that it can overstate a journal’s impact by including self-citations (when a journal cites itself) and citations from relatively obscure journals that don’t themselves have much scholarly impact. The Clarivate Article Influence Score (AIS) is an alternative metric that estimates a journal’s “prestige” by assessing citation relationships with others journals and considering the scholarly impact of the citing journals (see Postscript 1). The mean score for all journals is 1.
With that as context, here are AIS scores for some behavior analysis journals (those indexed here). Below you can see two things. First, if you’re familiar with our journals’ IFs, the hierarchy might be a little different than you expected based on raw citation counts. Second, and more importantly, only Journal of Contextual Behavioral Science tops the mean, and then only by a hair. In other words, by the standards of AIS, most behavior analysis journals are relatively weak. That doesn’t mean we should just shut everything down, merely that we have considerable incentive to strengthen those that can be strengthened.
What’s shown above is only one way of assessing journal “quality.” For instance, it’s generally agreed that strong journals influence diverse other journals. Weak journals rely heavily on self-citations. We might therefore ask which behavior analysis journals do this most. If the work appearing in some journals seems not to be registering with a broad scholarly audience. It might therefore benefit from appearing in a more widely noticed journal.
The opposite of broad reach occurs when a good portion of an article’s attention comes from the journal in which it was published. This is called self-citation, and it’s been suggested that good quality journals have a self-citation rate of 10% to 15% at most (see also here). But let’s be liberal and say up to 20% is allowable. That’s a standard many behavior analysis journals don’t meet, and in the past behavior analysis journals have been flagged by journal indexing services for their high self-citation rates. Fairly recently one of our flagship journals was in danger of being de-listed from one of those services as a result.
Another thing to consider is that a journal may, on average, attract a lot of citations, but due mainly to a few impactful articles, with the modal article going relatively unnoticed. A simple, if imperfect, way to examine this is to see what proportion of a given journal’s articles are ever cited. In general, the “better” the journal, the greater the proportion of articles that get cited. With that in mind check out this figure from one of my previous posts, comparing the number of articles published by each of several journals that did (green) and did not (pink) get cited over a three year window. For some journals, there’s an unsettling amount of pink.
More on This Analysis
I called this analysis is imperfect. hHere’s one reason why. Data from SCImago address citations occurring in the most recent few years, whereas the citation half-life (how long it takes for half of an article’s citation to accumulate) for behavioral journals can run to 10 or 20 years. You might project, therefore, some some the pink/uncited articles will eventually attract citations. That being said, what we’re concerned with right now is variations across journals and it’s clear in the figure above that uncited articles are more of a concern for some journals than for others.
Finally, I’m not a simpleton. I know that citation data don’t tell us everything we may want to know about a journal. For instance, in assessing the health of a given journal we may also wish to take into account how it contributes to the relationship between science and society. Thus, for example…
- Some journals lack citation impact (attention from scientists) but demonstrate dissemination impact (attention from non-scientists). Behavior Analysis in Practice is one such journal. Both kinds of impact are important (and they’re only weakly correlated).
- In behavior analysis, we have been concerned with assuring that women and non-U.S. authors are adequately represented in our journals. We might also ask how well a journal is addressing topics of particular societal importance, say, those concerned with sustainability. See Postscript 2 for how our journals rank on some of these kinds of dimensions.
Quantity and Quality of Journals: The Big Picture
It should be obvious that some behavior analysis journals are underperforming in terms of scholarly citations and some other objective measures (Postscript 2). That’s exactly what we’d expect when the number of journals is too big for the supply of good scholarship. We can cure what ails our journals by publishing fewer of them.
Take an analogy. Say that in your city there are a few really top notch artists, but a LOT of art galleries. Those galleries have limited choices on how to deal with this. They can sponsor exhibits only when they have enough high quality art for them — which will be rarely. Or, they can sponsor exhibits often but pad them with lesser art. Objective measures, like exhibit attendance or the volume of art sales at exhibits, can track the health of the galleries. In either case, we might expect poor numbers for at least some galleries. Infrequent exhibits by definition mean limited attendance. But if mediocre art is displayed in order to boost the number of exhibits, people may be turned off and pass on the exhibits.
Similar dynamics may affect behavior analysis journals. If we had Ultimate Power and could simply will fewer journals into existence, simple math says that manuscript flow to the remaining ones would increase, allowing those journals to publish more articles. This matters in part because a big predictor of scientific impact is a journal’s publication frequency — — in general, journals that come out more frequently are cited more. In one study I read, the highest citation rates in medical were from journals publishing 6-12 times per year. The basic idea: When a journal’s issues appear many months apart, that journal falls off people’s radar in the interim. People forget to seek out its content and don’t think of it first when deciding where to send a manuscript.
Many of our journals come out 1-3 times a year only (and almost none meet the standard mentioned above of 6-12 issues a year). In addition to being lightly cited, some struggle to get enough submissions to meet even a very unambitious publication schedule. Obviously you can’t publish more issues per year without more manuscripts. But scholarly attention is a sort of meritocracy, because the scholarly audience is wise enough to tell better papers from worse ones. A nice side effect of robust manuscript flow is that you can be picky about what you publish. I have seen a number of our journals from the inside, and you might be surprised at how often some of them scramble just to find enough articles to fill out the upcoming issue. Under conditions like that, you sometimes have to bend your quality standards — which might explain some of the pink swaths shown above.
Consolidation
So here’s the proposition: We need to reduce the number of journals that serve our discipline — arbitrarily, I’ll say, by about half. A straightforward approach would be to fold several journals into one omnibus journal that can come out often and cover perhaps a broader range of topics than some journals do now.
But which journals ought to be consolidated? A starting place is with journals that are essentially redundant, that is, exhibit considerable thematic overlap. Those that aren’t performing well could be folded into better-performing journals. A key first step, then, is to identify which journals do similar things.
As a starting point in defining affinities among journals, I used citation data from Web of Science to determine, for each behavior analysis journal it indexes, the % of the journal’s citations that came from two groups of journals as defined by tradition and journal mission statements: four with an unambiguously basic/theoretical focus (JEAB, TPR, POBS, B&P), and eight with an unambiguously applied focus (JABA, BAP, JPBI, JOBE, BM, ETC, BI, JOBM). Below, two clusters of journals fall neatly out.
Basic and Theoretical Journals. Let’s examine those two clusters more closely, starting with basic/theoretical journals. The SCImago web site has a cool tool for mapping inter-journal citation relationships, including direct citation links between target journals and indirect links defined by other journals to which the target journals are closely related. In the plot of more-basic journals below, the distance separating two journals is inversely proportional to the strength of their mutual citation relationship.
Several of the journals appear tightly linked, and based on this empirically-defined suggestion of redundancy, and would make good candidates for merging.
Applied Journals. For applied journals, I started with two trios of journals defined by similar mission statements: three with ABA’s traditional focus on developmental disorders and severe behavior problems (JABA, BI, BM) and three with a school/education focus (JPBI, JOBE, ETC). It turns out, however, that these are all pretty interconnected, suggesting one omnibus journal could encompass all six. The possible outlier is JPBI. Careful inspection of the figure shows that a number of indirect citation relationships link it to other ABA journals (see all of the grey dots between it and ETC/JOBE?). More on that next.
You have to understand that the scaling in these plots is relative. The affinity between two journals depends on the other journals in the “network.” To illustrate, here’s a plot including JPBI, JOBM, (which is topically very distinct from other ABA journals), and JCBS (which tends not to be cited much by other behavior analysis journals). I threw in JEAB, JABA, and POBS as familiar touchstones. In this context JPBI, JOBM, and JCBS all are outliers. JEAB, JABA, and POBS have a lot to do with each other, and relatively little to do with those other three journals.
The relationships shown so far might make more sense in the context of journals that don’t fit so well anywhere, either because there they have little citation connection to other behavior analysis journals (JCBS) or are topically distinct (JOBM). As anchors I included the flagship ABA journal (JABA), the flagship EAB journal (JEAB), and the main journal devoted to theoretical issues (POBS). Those three are tightly interconnected, while JPBI, JCBS, and JOBM stand apart. The practical implications of that bear some reflection, however.
However, a case could be made for merging JPBI with two other education-focused journals. I did not uncover any similar synergies regarding JOBM or JCBS, suggesting they really do stand more or less alone in terms of citation relationships.
The Take-Home
This is all food for thought, illustrating that (a) if we had the Ultimate Power to force journal consolidation, objective data exist to guide those decisions; and (b) those decisions can and should be multiply-determined (based on no single consideration).
Models exist for what the resulting omnibus journals could look like. Some journals have independently edited topical sections, which provides a means to both highlight topics of interest and to assure that manuscripts on those topics are peer reviewed by people with suitable expertise. Imagine, then, that several ABA journals were combined into one. A whole host of options would exist for defining topical sections. For instance, some sections could be defined by populations or behavior problems of interest (e.g., autism, feeding, social skills, psychological flexibility, whatever). Some might be defined by settings of interest (schools, prisons, workplaces). Whatever sections are chosen, each would appear only in issues for which new articles are available, thereby eliminating the logjam that happens in more specialized journals that don’t always receive enough manuscripts to hew to a publication schedule. Oh — and because this journal will be bigger and broader than any current ABA journal, more people will see it, meaning that each article (even one in a relatively obscure research area) will have an enhanced opportunity to find an audience.
All of this is 100% do-able… save for one little detail. Behavior analysts would never let any of this happen, because consolidating journals would threaten those silos that behavior analysts have worked so hard to build. An old adage says that if there’s a lot of some behavior, there must be a reinforcer. And behavior analysts built a lot of silo. They do that because they get a lot out of silos. Where journals are concerned, some folks may simply feel happy knowing a cherished topic has its own silo-ed title (never mind that almost nobody reads that journal or submits manuscripts to it). Many people get professional reinforcers for being involved in running silo-ed journals, or from having their papers published in them. And organizations may profit financially from owning certain silo-ed journals.
So, a dense set of reinforcement contingencies helped to create our journal problems. Those problems could be addressed head on by consolidating existing weak journals into a few strong ones, which is such a good idea that it is bound to be unpopular. That is to say, the same contingencies that created our problems probably undermine any logical solutions. Check back in 5, 10, 20 years… Barring some financial catastrophe, I doubt you’ll find fewer behavior analysis journals. Probably there will be more. And however many there are, some will have not have the objective accomplishments to justify their existence.
Still, we claim to be a data-driven discipline. The purpose of this post is to ask if we are managing our journals accordingly.
Postscript 1: The Article Influence Score
The AIS is determined in three steps. From Web of Science:
(1) The Eigenfactor Score is a reflection of the density of the network of citations around the journal using 5 years of cited content as cited by the Current Year. It considers both the number of citations and the source of those citations, so that highly cited sources will influence the network more than less cited sources. The Eigenfactor calculation does not include journal self-citations.
(2) The Normalized Eigenfactor Score is the Eigenfactor score normalized, by rescaling the total number of journals in the JCR each year, so that the average journal has a score of 1. Journals can then be compared and influence measured by their score relative to 1.
(3) The Article Influence Score normalizes the Eigenfactor Score according to the cumulative size of the cited journal across the prior five years. The mean Article Influence Score for each article is 1.00. A score greater than 1.00 indicates that each article in the journal has above-average influence.
A similar metric is the SCImago Journal Rank Indicator.
Postscript 2: Three Non-Citation Metrics of Journal Health
All data in this Postcript are from SCImago, and journal abbreviations are as employed above. The measurement window isn’t clear to me but most SCImago metrics cover the most recent two or three years.
(1) Percentage of female authors. From SCImago:
To determine the gender of authors, a method based on given names was employed, grounded in “cultural consensus theory” (See Van Buskirk et al. (2023)).
(2) Percentage of articles with international (2+ countries) author teams. SciMago does not provide norms, but note that the Top 10 journals overall in terms of citation impact average about 25% international authors.
(3) Sustainability. From SCImago:
Number of documents related to the Sustainable Development Goals defined by the United Nations according to the Elsevier 2023 Sustainable Development Goals (SDGs) Mapping.
