Audit finds NYPD spent thousands of police hours on unconfirmed gunshot alerts
comptroller.nyc.govAs an adjacent observation: the tone in NYPD’s official responses is a clear signal that (1) they don’t particularly care about what the city’s primary financial official has to say, and consequently (2) that their budget is insufficiently controlled and overseen by the city’s democratically elected leaders. It’s remarkable (but not particularly surprising) to see so much open disdain for basic civic interest, including the open claim that the people of NYC are too stupid to understand the value of a service the NYPD pays for with their tax dollars.
I'm coming to realize that the problem with technology hype men - peddlers of unnecessary complexity of all kinds from cloud and kubernetes everything, to microservices and other architectural fetishes, agile/scrum rituals, and now LLMs, generative AIs etc... They all forget one big thing. For all the 'entropy lowering' powers of whatever they are selling, they always hide the entropy exhaust.
Thermodynamics ought to have taught us that we can only move entropy around. And all these silver bullets have giant gaping entropy exhausts that we get to find out about once they've been paid for.
We will see backlash like this, and justifiably so, as we continue to iterate AI/ML as product.
A better use of the tech, though perhaps below the justification threshold for purchasing it as SpotShotter, would be to use it as confirmatory forensics after an instance versus investigating every shot.
Reasoning: if a loud noise happens and no one calls the cops, no harm no foul. But if someone calls the cops or a crime with a weapon is noted, such a murder, wounding, or other violence, this could help narrow down the timeframe the shot may have been observed.
I was hoping the backlash would be from NYPD installing a network of microphones on sidewalks all over the city. I don’t care what they’re using them for it’s creepy.
Yeah, hypersurveillance is also bad!
ShotSpotter isn't really related to current AI/ML products. The company has been operating for almost 30 years.
Correct, it isn’t AI. It is however massively overrated, has questionable contract practices, and is a false alert generator… for over 30 years.
It's filling the same role people are wanting to use AI to fill, automation of information collection and identification of signal in the noise.
You've got it completely backwards. If you have a piece of evidence with a high signal-to-noise ratio, you don't validate that with a mechanism that has an extremely low signal-to-noise ratio. That's completely scientifically and statistically unsound.
What GP is implying is using the high signal to noise ratio evidence to give you a better way to screen the low signal to noise ration evidence and check for extra information. Not using the former to 'validate' the latter.
Shocker that NYPD rejected every single recommendation other than, pay your bills on time, and enforce existing contractual requirements.
No willingness to evaluate the product, discuss or prove efficacy, or even consider new renewal contract terms.
It seems kinda reasonable to me? If someone proposed that we should turn off compiler warnings because only 13% of them are confirmed to be bugs, I'd give basically the same kind of response the NYPD gave here. The proposed metrics don't meaningfully evaluate what the system is trying to do, and we don't really have the time to drop other priorities and produce a comprehensive report about it, so let's keep using the tools that the people doing the work feel are useful unless we have a more compelling reason not to.
Does this analogy consider the costs involved with sending out the police? When the compiler is wrong, do armed agents of the state with broad discretion to employ violence in the commission of their duties get dispatched? What is the potential opportunity cost of inefficiently using the compiler vs the police choosing to respond to a shotspotter alert over a potently more accurate signal?
To be really clear, I think the risk profile of a false positive in terms of a compiler vs police dispatch is the difference between the potential waste of pennies compared to the potential waste of human lives.
I'm not sure what you're referring to, since the audit described in the source article doesn't seem to mention any such costs.
Perhaps you're coming from a perspective where policing is presumptively bad, and we ought to avoid having police around unless we're confident there's a specific benefit in doing so? I know that many people think that way, and I'm not sure how to engage with it other than to say I don't agree. I don't think the police are perfect by any means, but I live in an area covered by ShotSpotter, and I feel safer knowing they're detecting and investigating anything that sounds like a gunshot in my neighborhood.
Perhaps you're coming from a perspective where policing is presumptively bad
In much of the US, this is a fair assumption. The police have no legal obligation to assist. They carry guns and are authorized to use them. They receive less training than many of their European counterparts. And the entire justice system is constructed to allow minimal accountability in the face of malfeasance. It's a recipe for not only wasted dollars, but also excess violence and wasted lives.
Again, I'm not sure what to say. I agree with many of your criticisms, but it's hard for me to see why they imply that a tool to help police detect and investigate every gunshot noise in my neighborhood is bad. I'd much rather have an undertrained cop investigate than nobody at all, and training cops to a European standard would require large investments in police funding that I'm not sure are politically feasible today.
The problem is the tool is ineffective (to the point several cities have stopped using it). And anything that causes an erroneous/unnecessary police response should be viewed with extreme suspicion, given the massive flaws in our police model.
> the audit described in the source article doesn't seem to mention any such costs.
Not sure I understand what you're saying - it's not mentioned in the article, which means it does not exist? Anyway, have a nice day.
Yeah I'm not sure this report says much. They recommend the NYPD look into it more, but it could realistically be a good investment of resources. Some rough math:
(32*7262)/60/8/365 = 1.3 officers a year are wasted. There are like 30k officers in NYC. This seems like it could be a worthwhile investment even with only 13% true positives.
My understanding is that cop car driving down the block and making a visual inspection after a likely gunfire event is a reasonable standard. I would expect that 90% of the time (in fact, 87% here) they won't find anything. It is easy to take a gun indoors after firing it, or even hide it under a jacket. That doesn't mean that a cop car driving by in the next 5 minutes after any gun shot event won't have a positive impact.
Perhaps this is racially biased policing. Gun shot firing events may track racial neighborhoods demographics. In that case, we should consider whether "racial bias" is a useful dialectical tool here.
> I would expect that 90% of the time (in fact, 87% here) they won't find anything
Not opposed to the hypothesis, but we're talking about the dispatch of armed agents of the state who are permitted to use violence in the discharge of their duties. It's serious, and the facts that no party disputes are that 87% of the time, the destination is indistinguishable from one where no shooting occurred.
Now, the hypothesis that shots are in fact being correctly detected, however frequently there is no observable evedence is plausible and should be considered. This is something that can be investigated with double blind procedures to determine things like the false positive detection rate, or the rate at which police can accurately determine if a shooting occurred after the fact. I suspect that both of these rates are known and do not support the use of this product, but I am very open to saying I'm wrong if presented with more information.
This all just looks like your bog standard bullshit-product-sold-to-the-government con. Government officers (frequently police and the military) buy into a bullshit product because it justifies their operations and confirms their worldview. Up to the early 2000, the big one was a magic explosives and contraband detection wand that claimed to alert the operator via metaphysical energy when it was placed near any kind of explosive or contraband material. These products are largely psychological aids that help the operator justify whatever it is they are doing.
>but we're talking about the dispatch of armed agents of the state who are permitted to use violence in the discharge of their duties.
You could say the same thing about a police officer being sent to direct traffic at a busy intersection, or going to a school for career day. Just because the police officer is an armed agent of the state permitted to use violence doesn't mean the police officer shouldn't do things that aren't very serious.
These news reports quote responses from other interested groups like the New York Police Department itself, the New York Civil Liberties Union, and ShotSpotter, Inc.:
https://www.cbsnews.com/amp/newyork/news/nypd-shotspotter-re...
https://www.nytimes.com/2024/06/20/nyregion/nypd-shotspotter...
Note that the NYPD's leader, the New York City police commissioner, is appointed by and answerable to the mayor. Mayor Eric Adams supports the ShotSpotter gunshot alert system, which may explain the NYPD's position.
Also note that this audit is published by the New York City Comptroller's office. Both the mayor and comptroller are directly elected by the people, meaning both Adams and Comptroller Brad Lander are politicians as opposed to nonpartisan bureaucrats/civil servants. That may have something to do with Lander's framing of his office's report.
ShotSpotter's reputation is terrible. Chicago ditched it a few months ago: https://www.npr.org/2024/02/15/1231394334/shotspotter-gunfir...
The most shocking story I've seen about it was from 2021: Police Are Telling ShotSpotter to Alter Evidence From Gunshot-Detecting AI https://www.vice.com/en/article/qj8xbq/police-are-telling-sh...
ShotSpotter is a hot mess. Their data can't be trusted if they collude with police to alter their data after the fact to cover for police misconduct.
An irrelevant complaint until the cited stats are, first, confirmed with a research critique and then compared to parallel stats in the context of ultimate successful criminal prosecutions without (compared to with) shot spotter.
Because the real goal is successful prosecutions, not to eliminate false calls or to reach accuracy parity with 9/11 calls. To wit, no one would want to eliminate all false calls at the expense of a significant number of real calls; especially if a portion of the real calls would not have been reported via 911. False calls are fine if the number of successful identifications and prosecutions is above the rate without shot spotter.
Moreover, we'd need a methodology that looks at the total amount of time worked for successful cases with and without shot spotter, even including false call time. Being frustrated at "unconfirmed" shooting responses is ridiculous if the total time worked build a case, including false calls, is par or better for the same number of cases that don't involve shot spotter. Efficiencies, and conversely wasted time, can be hidden everywhere.
At what point does the city start ignoring shot spotter critiques because logic incompetent critics generate more noise and busywork than they are worth? After all, they aren't even self-careful of their own perspective and respectful of everyone else enough to have a cheap grad student edit their research review for public presentation. If they don't care, why should everyone else?
The cherry on their BS Sundae. This person has zero data to show that the ultimate shot spotter results are substandard. Fire them and hire someone who can think and therefore doesn't embarrass The City:
>“The NYPD’s response to these audit findings is disappointing and reflects a disinterest in using data, effective performance metrics, and transparency to improve public safety. With a thorough evaluation before deciding whether to renew this multi-million-dollar contract, better performance standards, and more transparency, the NYPD could deploy its resources – especially its officers’ time – far more effectively.”
Sounds like a classic case of Algorithm Aversion to me.