Hi, maybe this is a second order issue at this stage, but did you consider that people could team up to (mis)report model behavior as problematic if they have the incentive to do so?
oh yes absolutely!! I think it's a super interesting question and there is existing work in different contexts from both empirical/theory communities on this... yeah, somewhat "second order" right now but still worth thinking about I think
This sounds really interesting, and potentially very powerful for identifying the impact these tools are having in the wild. I am generally fascinated by the ways we (humans, especially Americans, especially tech culture) navigate the intersection between individual and collective reality.
Are you familiar with Callisto? (https://www.projectcallisto.org/) It's a tool built for college campus environments, where a person can file a report of sexual abuse or assault that will only be shared if at least one other person files a report about the same perpetrator. It's trying to get at repeat offender dynamics, and lessening the isolation that often comes with reporting. (Caveat: I am not saying this is a perfect idea with no potential for flaws or other abuse it's just an interesting approach etc etc etc) Anyway not exactly the same thing but your proposal made me think of it.
I would love to read more about possible ways to implement and experiment with your idea for some of the big LLMs. Or honestly for other giant tech products.
What about privacy? To identify harms affecting specific subgroups from individual reports, you have to collect those covariates (financial status, gender, sex, race, ethnicity, religion, veteran status, etc etc). Broadly speaking, this has two problems:
- you may be subject to privacy rules which forbid you from looking at this characteristics directly
- your end users may be very wary of providing so much information about themselves
I think these two points go some way towards answering the question "why we don't do it in the industry today".
thanks for reading & sharing your thoughts! not a privacy expert but my first thoughts are that (a) from a legal perspective, voluntary data sharing from reporters is different from companies tracking users directly (b) yeah maybe that's a possibility 🤷♀️ that said there are also lots of people are very happy to share tons of info about themselves too.
fwiw i am deeply skeptical that any industry org really truly cares about privacy beyond liability concerns; in fact privacy is almost always a really convenient excuse for companies to remain more closed (as ben discusses a bit in this post - https://www.argmin.net/p/the-closed-world-of-content-recommendation)
Thank you for replying :) Firstly I wanted to say that the work you presented is very interesting and useful, and my remarks are not meant to diminish it, but just to provide an "industry" perspective, especially on the application of it.
Re my first point: in a large org you may have one team which wants to collect this data and another team responsible for privacy & compliance and doing such collection may require a sufficiently onerous internal process that the first team is discouraged. Also regulations like GDPR impose additional cost on anyone storing private data. Compare the two situations:
- I provide a paid service and store the absolute minimum abt my users needed to provide it => less GDPR-related work such as handling data deletion requests
- I collect additional information to carry out such analyses as above => now I get additional compliance tasks which someone has to pay for.
Additionally, you may have customers who request you not to use their data to improve your services to other customers. Then you can't really aggregate the data between customers to improve your product. Maintaining a separate model per customer may be too costly.
As you said, companies are driven by legal liability concerns, but also reputation concerns and financial concerns (cost). My fear is that since privacy violations carry a stronger regulatory penalty than fairness violations, they will err on the side of respecting privacy (don't collect the data unless absolutely necessary).
Re my second point, I have a technical question: if you only collect data from people who aren't bothered by providing it and have the time to fill out the necessary surveys, is this a good, representative sample of the user population? Can your procedure account for such biased sampling by eg reweighting it?
This last point you raise is at the crux of why I think there's really interesting work to be done here! Reports are inherently not going to be a representative sample, and I think the act of reporting itself holds some amount of signal. But yeah, as I discuss in the papers, understanding the dynamics of reporting behavior and correcting for them is one of the big challenges
First
Hi, maybe this is a second order issue at this stage, but did you consider that people could team up to (mis)report model behavior as problematic if they have the incentive to do so?
oh yes absolutely!! I think it's a super interesting question and there is existing work in different contexts from both empirical/theory communities on this... yeah, somewhat "second order" right now but still worth thinking about I think
This sounds really interesting, and potentially very powerful for identifying the impact these tools are having in the wild. I am generally fascinated by the ways we (humans, especially Americans, especially tech culture) navigate the intersection between individual and collective reality.
Are you familiar with Callisto? (https://www.projectcallisto.org/) It's a tool built for college campus environments, where a person can file a report of sexual abuse or assault that will only be shared if at least one other person files a report about the same perpetrator. It's trying to get at repeat offender dynamics, and lessening the isolation that often comes with reporting. (Caveat: I am not saying this is a perfect idea with no potential for flaws or other abuse it's just an interesting approach etc etc etc) Anyway not exactly the same thing but your proposal made me think of it.
I would love to read more about possible ways to implement and experiment with your idea for some of the big LLMs. Or honestly for other giant tech products.
Oh yes I think Callisto is super cool, def related!! Re your last comment... working on it :)
What about privacy? To identify harms affecting specific subgroups from individual reports, you have to collect those covariates (financial status, gender, sex, race, ethnicity, religion, veteran status, etc etc). Broadly speaking, this has two problems:
- you may be subject to privacy rules which forbid you from looking at this characteristics directly
- your end users may be very wary of providing so much information about themselves
I think these two points go some way towards answering the question "why we don't do it in the industry today".
thanks for reading & sharing your thoughts! not a privacy expert but my first thoughts are that (a) from a legal perspective, voluntary data sharing from reporters is different from companies tracking users directly (b) yeah maybe that's a possibility 🤷♀️ that said there are also lots of people are very happy to share tons of info about themselves too.
fwiw i am deeply skeptical that any industry org really truly cares about privacy beyond liability concerns; in fact privacy is almost always a really convenient excuse for companies to remain more closed (as ben discusses a bit in this post - https://www.argmin.net/p/the-closed-world-of-content-recommendation)
Thank you for replying :) Firstly I wanted to say that the work you presented is very interesting and useful, and my remarks are not meant to diminish it, but just to provide an "industry" perspective, especially on the application of it.
Re my first point: in a large org you may have one team which wants to collect this data and another team responsible for privacy & compliance and doing such collection may require a sufficiently onerous internal process that the first team is discouraged. Also regulations like GDPR impose additional cost on anyone storing private data. Compare the two situations:
- I provide a paid service and store the absolute minimum abt my users needed to provide it => less GDPR-related work such as handling data deletion requests
- I collect additional information to carry out such analyses as above => now I get additional compliance tasks which someone has to pay for.
Additionally, you may have customers who request you not to use their data to improve your services to other customers. Then you can't really aggregate the data between customers to improve your product. Maintaining a separate model per customer may be too costly.
As you said, companies are driven by legal liability concerns, but also reputation concerns and financial concerns (cost). My fear is that since privacy violations carry a stronger regulatory penalty than fairness violations, they will err on the side of respecting privacy (don't collect the data unless absolutely necessary).
Re my second point, I have a technical question: if you only collect data from people who aren't bothered by providing it and have the time to fill out the necessary surveys, is this a good, representative sample of the user population? Can your procedure account for such biased sampling by eg reweighting it?
This last point you raise is at the crux of why I think there's really interesting work to be done here! Reports are inherently not going to be a representative sample, and I think the act of reporting itself holds some amount of signal. But yeah, as I discuss in the papers, understanding the dynamics of reporting behavior and correcting for them is one of the big challenges