Discussion about this post

User's avatar
Jessica Hullman's avatar

Hi Ben,

To be clear, my comments to Anoneuid should in no way be taken as advocating for mindless application of frequentist statistics nor blanket reinforcement of reform heuristics that someone has decided must be applied -- I would hope this is obvious, as it would go against so much of my prior research and all of my blog posts on how science reform being misguided. I think the NeurIPS checklist is a mess (as it seems you do).

What motivated my responses to Anoneuid (as should be clear from reading them) was my difficulty with the logical implication of your dismissal of statistics in ML that authors should be absolved of having to match their evidence to their claims. Insisting on certain norms for communicating error is silly. However, presenting data from some experiment you ran without explaining enough about the process for generating those results for someone to evaluate your claims is also silly. At that point, the experiments are simply performative. In which case I'd prefer not to see them at all.

Expand full comment
Onid's avatar

Working at one of these industry labs, I’ve had the experience multiple times of being told that an interesting algorithmic idea could not be published because it was being used (or might be used) in our production system.

You talk about how openness is a necessary principle for scientific progress, and I strongly agree. But the problem is that these industry labs aren’t optimizing for scientific progress - that ended the moment their research programs stopped being about prestige and started being about profit and competition. It’s a sad dynamic but that’s where we are right now.

Expand full comment
5 more comments...

No posts