To the question: "But why academia was unable to engage in a completely obvious and totally plausible sort of collective action to create digital platforms via a massive consortial infrastructure that would completely cut out the expensive middleman?": physics more or less has, with what is now arXiv.org, which is now 32 years old, having been launched contemporaneously and independently of the web. These pieces by its creator Paul Ginsparg marking the 20th (https://www.nature.com/articles/476145a) and 30th (https://www.nature.com/articles/s42254-021-00360-z) anniversaries offer some insight into the change that has been achieved and the unexpected challenges of running such a service.
Of course it hasn't killed the journals, of which there are several more than when arXiv began. One thing that struck me the last time I looked at the financial part of the American Physical Society's annual report is that its journals (which are high-prestige) are the only money-maker in the budget: the conferences it runs are more-or-less break-even, and membership dues don't even cover the regular member services functions. But the publications bring in about $11M more per year than they cost to produce. So there is a way of looking at it in which the APS has arranged for university library budgets to subsidize e.g. its advocacy and outreach activities.
There hasn't been a similar open preprint archive in chemistry, which would nominally seem to be a similar discipline. In fact, the chemistry journals are historically far more restrictive about sharing articles and preprints. The main difference, as I understand it, it that there is no physics industry the way there is a chemical industry, which pays whatever the American Chemical Society asks for its journals.
arXiv.org is the 1st wonder of the new world, really. It's the only remnant sign that someone was paying attention enough to respond. But it should have been across the whole of our industry and it should have been an obvious and unanimous impulse.
"I only know that the opportunity is now long in the rear-view mirror. "
I, like Tom, came here to say that this is wrong. The good fight on this issue is now. bioRxiv is relatively new, for example, and so are things like EU open access mandates and efforts to create things like overlay journals and other efforts to not just produce freely available scholarship but eliminate the absurd overhead that journals extract from us.
What I fixate on though is the plausibility of a comprehensive solution--I can see when it could have been done, I can see how it was affordable for all if enough institutions had joined in (and how much money it would have saved), and I can see where we'd be now--which is not working desperately hard to maintain a handful of disciplinary-based publication platforms. And I don't think that comprehensive solution can happen now, it's up to each discipline or at least clusters of disciplines, to build the alternatives. And for me the question is "Why was this not even a thought entertained by most academic leaders at the moment when it was readily possible?" I'm sure you've seen me pitch this concept before, and it wasn't just something I repeated on the blog--I said it any time I got within hailing distance of provosts and presidents, I said it when I was on a panel at Google, I said it when I was speaking to research librarians. The leadership was always just amused by the idea, at best; much of the time, it was obvious that they didn't think of libraries as anything but a budgetary problem. So I think that's a big part of it--the shift in academic leadership away from caring much about the basic values or mission of academic institutions meant that the idea of working consortially on a massive scale to free themselves from dependency on for-profit publishing and to disseminate scholarship to the global public was in most cases "and why would we care about that?"
I am deeply, deeply skeptical that the "comprehensive solution" you're imagining was every remotely plausible. The international scope, decentralized nature, and differentiated roles mean that even conceptualizing the problem accurately was actually harder then than now, let alone solving it. In particular, the vision of academic publishing as a group of university libraries that pays people at university presses to publish the work of those same universities' academics that has since been taken over by for-profit entities is only applicable in humanities/social science, which is a fairly small fraction of the total academic publication volume.
Depending on which index you look at, humanities + social sciences are somewhere between 25-35% of the total volume of journal titles, which I think is bigger than "fairly small", but it's definitely those journals which most urgently ought to have been transferred to a large-scale non-profit publication platform at an optimal date, and they're also the journals whose editors and authors have a lot less weight to throw around now. (On the other hand, they also don't have some of the absurd overhead costs that are built into the first-tier science journals.)
On the other hand, it's also the basic orientation of a lot of humanities and social sciences faculty that makes any kind of comprehensive open-access strategy difficult now. Though I have colleagues that point out that collaboration is not as rare as we think in those areas, it's still much less the norm than in the sciences. As a result, all collective action is more difficult in multiple ways to coordinate when it involves those disciplines.
To the question: "But why academia was unable to engage in a completely obvious and totally plausible sort of collective action to create digital platforms via a massive consortial infrastructure that would completely cut out the expensive middleman?": physics more or less has, with what is now arXiv.org, which is now 32 years old, having been launched contemporaneously and independently of the web. These pieces by its creator Paul Ginsparg marking the 20th (https://www.nature.com/articles/476145a) and 30th (https://www.nature.com/articles/s42254-021-00360-z) anniversaries offer some insight into the change that has been achieved and the unexpected challenges of running such a service.
Of course it hasn't killed the journals, of which there are several more than when arXiv began. One thing that struck me the last time I looked at the financial part of the American Physical Society's annual report is that its journals (which are high-prestige) are the only money-maker in the budget: the conferences it runs are more-or-less break-even, and membership dues don't even cover the regular member services functions. But the publications bring in about $11M more per year than they cost to produce. So there is a way of looking at it in which the APS has arranged for university library budgets to subsidize e.g. its advocacy and outreach activities.
There hasn't been a similar open preprint archive in chemistry, which would nominally seem to be a similar discipline. In fact, the chemistry journals are historically far more restrictive about sharing articles and preprints. The main difference, as I understand it, it that there is no physics industry the way there is a chemical industry, which pays whatever the American Chemical Society asks for its journals.
arXiv.org is the 1st wonder of the new world, really. It's the only remnant sign that someone was paying attention enough to respond. But it should have been across the whole of our industry and it should have been an obvious and unanimous impulse.
"I only know that the opportunity is now long in the rear-view mirror. "
I, like Tom, came here to say that this is wrong. The good fight on this issue is now. bioRxiv is relatively new, for example, and so are things like EU open access mandates and efforts to create things like overlay journals and other efforts to not just produce freely available scholarship but eliminate the absurd overhead that journals extract from us.
What I fixate on though is the plausibility of a comprehensive solution--I can see when it could have been done, I can see how it was affordable for all if enough institutions had joined in (and how much money it would have saved), and I can see where we'd be now--which is not working desperately hard to maintain a handful of disciplinary-based publication platforms. And I don't think that comprehensive solution can happen now, it's up to each discipline or at least clusters of disciplines, to build the alternatives. And for me the question is "Why was this not even a thought entertained by most academic leaders at the moment when it was readily possible?" I'm sure you've seen me pitch this concept before, and it wasn't just something I repeated on the blog--I said it any time I got within hailing distance of provosts and presidents, I said it when I was on a panel at Google, I said it when I was speaking to research librarians. The leadership was always just amused by the idea, at best; much of the time, it was obvious that they didn't think of libraries as anything but a budgetary problem. So I think that's a big part of it--the shift in academic leadership away from caring much about the basic values or mission of academic institutions meant that the idea of working consortially on a massive scale to free themselves from dependency on for-profit publishing and to disseminate scholarship to the global public was in most cases "and why would we care about that?"
I am deeply, deeply skeptical that the "comprehensive solution" you're imagining was every remotely plausible. The international scope, decentralized nature, and differentiated roles mean that even conceptualizing the problem accurately was actually harder then than now, let alone solving it. In particular, the vision of academic publishing as a group of university libraries that pays people at university presses to publish the work of those same universities' academics that has since been taken over by for-profit entities is only applicable in humanities/social science, which is a fairly small fraction of the total academic publication volume.
Depending on which index you look at, humanities + social sciences are somewhere between 25-35% of the total volume of journal titles, which I think is bigger than "fairly small", but it's definitely those journals which most urgently ought to have been transferred to a large-scale non-profit publication platform at an optimal date, and they're also the journals whose editors and authors have a lot less weight to throw around now. (On the other hand, they also don't have some of the absurd overhead costs that are built into the first-tier science journals.)
On the other hand, it's also the basic orientation of a lot of humanities and social sciences faculty that makes any kind of comprehensive open-access strategy difficult now. Though I have colleagues that point out that collaboration is not as rare as we think in those areas, it's still much less the norm than in the sciences. As a result, all collective action is more difficult in multiple ways to coordinate when it involves those disciplines.