Your post rhymes with James Scott's "Seeing Like a State". The technocratic powers that be share Scott's description of "High-Modernist Ideology" which is a form of your quantification trap. A quasi-religious faith that scientific planning and centralized design can optimize every facet of human life. Given that ideology, the powers that be feel justified in pursuing authoritarianism to realize their utopia by steamrolling local/contextual wisdom and practice.
Integrating local/distributed participation is a obviously a good thing, but maybe even more important is a recognition of the failures of architectures of legibility and the slippery slope to authoritarianism (Scott's example of 'scientific forestry' is sobering). A strong dose of humility among technocrats will get them to think about smaller, reversible interventions that can learn and benefit from (hopefully locally sourced) feedback.
You have already made this point implicitly (or explicitly and I might have missed it). Quantification seems like a way to reduce complexity when making decisions, constraining both the space of information you consume and choices you make downstream. This is bad on a human level as you often point out.
However, it also allows you to benefit from decision making at scale. This seems both (via vague analogies) true for bureaucracies and for e.g. deep learning. For example, it seems like you could indeed design much more efficient solutions if you look at specific problems (and expand your information choices) but to enable general purpose learners, perhaps its useful to reduce the complexity of the feature space (all text!).
think an idea like this should also be accompanied by companion reform of architechtures of legibility. Otherwise you end up with the same problem where participation becomes a malformed optimized target.
The standards we create or simply rely on within architectures of legibility ought to serve coordination that is making the concept of 'the decision maker' possible and not decision makers who ultimately are setting the standards
Thanks Ben. A couple of comments supporting your concerns. Firstly a couple of weeks back I was at a conference on the environmental dimensions of antimicrobial resistance (AMR). Many managers e.g. water industry, are sympathetic to AMR mitigation. But what should they do? One problem is the shear scale, complexity and rapid evolution of this phenomenon. Currently there simply are no metrics validated sufficiently for managers/engineers to justify spending the huge resources likely needed so they might tick their KPI boxes. And I suspect there never will be with absolute certainty demanded by accountants.
Following on from this is the more problem of how decision makers develop operational policies and make decisions in complex, rapidly evolving fields in the first place. This is the cringe-worthy world of Managerialism. My personal "favourite" style is 'New Public Management' which claims managers/decision makers need absolute no expertise whatever to make rational decisions (lest they go 'rogue' and sympathize). What? True its not the only style out there. Here are 6 others (thks Gemini) Technocracy, Professional Autonomy, Neo-Weberian Bureaucracy, Public Value Management, New Public Governance, Democratic Management with varying demands for expert managers. I have some sympathy with such people as the task they have to deal with is rather like herding cats. On the other hand we are also getting fiascos like https://en.wikipedia.org/wiki/Robodebt_scheme and https://en.wikipedia.org/wiki/British_Post_Office_scandal which potentially offer models for the shape of things to come.
Wikipedia is actually an interesting example here. While Wikipedia is indeed a huge collection of information, in some topics this information is highly biased and distorted. For example, it is no secret that non-Western governments and organizations are paying editors to make Wikipedia fit their Geo-political and historical narratives. A few random articles about that include:
As Wikipedia is often the source used by many existing AIs, this distortion is already a huge problem. We certainly do not want AIs teaching Russian propaganda to our kids. It is unclear if making AI a public good, as the post suggests, will make this problem any better or worse.
Thanks for this. Your criticism of pro forma quantification echoes the Frankfurt School’s critique of instrumental reason. I also really like your “architectures of participation” as an alternative: asking how people can help define the problem, contest the terms, and shape the outcome.
Your post rhymes with James Scott's "Seeing Like a State". The technocratic powers that be share Scott's description of "High-Modernist Ideology" which is a form of your quantification trap. A quasi-religious faith that scientific planning and centralized design can optimize every facet of human life. Given that ideology, the powers that be feel justified in pursuing authoritarianism to realize their utopia by steamrolling local/contextual wisdom and practice.
Integrating local/distributed participation is a obviously a good thing, but maybe even more important is a recognition of the failures of architectures of legibility and the slippery slope to authoritarianism (Scott's example of 'scientific forestry' is sobering). A strong dose of humility among technocrats will get them to think about smaller, reversible interventions that can learn and benefit from (hopefully locally sourced) feedback.
You have already made this point implicitly (or explicitly and I might have missed it). Quantification seems like a way to reduce complexity when making decisions, constraining both the space of information you consume and choices you make downstream. This is bad on a human level as you often point out.
However, it also allows you to benefit from decision making at scale. This seems both (via vague analogies) true for bureaucracies and for e.g. deep learning. For example, it seems like you could indeed design much more efficient solutions if you look at specific problems (and expand your information choices) but to enable general purpose learners, perhaps its useful to reduce the complexity of the feature space (all text!).
think an idea like this should also be accompanied by companion reform of architechtures of legibility. Otherwise you end up with the same problem where participation becomes a malformed optimized target.
The standards we create or simply rely on within architectures of legibility ought to serve coordination that is making the concept of 'the decision maker' possible and not decision makers who ultimately are setting the standards
Thanks Ben. A couple of comments supporting your concerns. Firstly a couple of weeks back I was at a conference on the environmental dimensions of antimicrobial resistance (AMR). Many managers e.g. water industry, are sympathetic to AMR mitigation. But what should they do? One problem is the shear scale, complexity and rapid evolution of this phenomenon. Currently there simply are no metrics validated sufficiently for managers/engineers to justify spending the huge resources likely needed so they might tick their KPI boxes. And I suspect there never will be with absolute certainty demanded by accountants.
Following on from this is the more problem of how decision makers develop operational policies and make decisions in complex, rapidly evolving fields in the first place. This is the cringe-worthy world of Managerialism. My personal "favourite" style is 'New Public Management' which claims managers/decision makers need absolute no expertise whatever to make rational decisions (lest they go 'rogue' and sympathize). What? True its not the only style out there. Here are 6 others (thks Gemini) Technocracy, Professional Autonomy, Neo-Weberian Bureaucracy, Public Value Management, New Public Governance, Democratic Management with varying demands for expert managers. I have some sympathy with such people as the task they have to deal with is rather like herding cats. On the other hand we are also getting fiascos like https://en.wikipedia.org/wiki/Robodebt_scheme and https://en.wikipedia.org/wiki/British_Post_Office_scandal which potentially offer models for the shape of things to come.
Wikipedia is actually an interesting example here. While Wikipedia is indeed a huge collection of information, in some topics this information is highly biased and distorted. For example, it is no secret that non-Western governments and organizations are paying editors to make Wikipedia fit their Geo-political and historical narratives. A few random articles about that include:
- https://www.fdd.org/analysis/2026/05/28/how-anonymous-wikipedia-editors-influence-global-narratives-and-ai-systems/
- https://www.adl.org/resources/report/editing-hate-how-anti-israel-and-anti-jewish-bias-undermines-wikipedias-neutrality
- https://www.inss.org.il/publication/wikipedia/
As Wikipedia is often the source used by many existing AIs, this distortion is already a huge problem. We certainly do not want AIs teaching Russian propaganda to our kids. It is unclear if making AI a public good, as the post suggests, will make this problem any better or worse.
Very sharp thoughts, Ben. Thank you
Three cheers for nationalizing the commanding heights of the economy.
Thanks for this. Your criticism of pro forma quantification echoes the Frankfurt School’s critique of instrumental reason. I also really like your “architectures of participation” as an alternative: asking how people can help define the problem, contest the terms, and shape the outcome.