Wyze security incident update
forums.wyze.com> We’ve identified your Wyze account as one that was affected. This means that thumbnails from your Events were visible in another Wyze user’s account and that a thumbnail was tapped. Most taps enlarged the thumbnail, but in some cases it could have caused an Event Video to be viewed.
Kudos to Wyze for doing the things noted in the thread like being honest and prompt with notification etc, but "thumbnails from your Events were visible... and that thumbnail was tapped" is a pretty mealy-mouthed way to say "another person saw your private pictures and videos taken with your Wyze cameras"
Well obviously, and I think people can figure that out.
I actually appreciate the more specific details on how the private pictures and videos were actually viewed using terminology from the Wyze app.
"like being honest and prompt with notification etc,"
On the Wyze subreddit, people have been griping about the fact that it took them days to even acknowledge something happened. When I read the Wyze email about it, there was no new info in it, it had pretty much all already been discussed online.
So 30 min of me sitting on my ass are on the web. Ok… enjoy viewers.
In the meantime, Wyze has rolled back RTSP support, where it was possible to use their devices locally:
https://support.wyze.com/hc/en-us/articles/360026245231-Wyze...
A good response to this might be to put it back, and to extend other devices to be dual-use (Wyze Cloud or HA).
For those that want their Wyze cams 100% local (with RTSP), you can use wz_mini_hacks[0]. I've set up v2 and v3 cams using this, and they've never touched the internet or wyze app.
I've use the official RTSP firmware in the past on some v2 cams, but I remember it having some problems and not being as good as this solution.
Thanks for this. I was using dafang hacks for v2 but didn't have solution for v3.
Another in a long line of reasons to avoid low price, off-the-shelf, unauditable, cloud-enabled cameras.
I continue to be amazed that there is not a reasonably priced, open source, audited, local-first solution, which doesn’t require a significant personal investment of time to install and maintain.
I swear sometimes you people post this ignorance bait to get product suggestions.
Local-first cameras are super cheap and easy to find. Most just run their own local RTSP server, which you can connect to live with VLC, homeassistant, whatever. OpenIPC is like ddwrt for ip cameras, here is the supported devices page: https://github.com/OpenIPC/wiki/blob/master/en/guide-support...
But unless you're Richard Stallman, go with the closed source Reolink RTSP camera, which is about $100 and used by big corporate installs. It can integrate with the cloud, but you can set it up to just have each camera run a RTSP server with user/pass auth. You have to secure your own network.
But there isn't a really great open source platform for the kind of multi cam security that businesses might need. You have to do your own storage. But grab four reolinks, send the feeds to homeassistant, and most homeowners will be fine.
You yourself just admitted that there "isn't a really great open source platform", so what exactly was ignorant about my request?
I have spent days searching for an affordable solution that meets my requirements. I know there are a lot of people that want the same thing, have done their own research, and have reached a similar conclusion. What we want simply does not exist.
I will stop posting this kind of comment when someone finally gives me a straight answer, but no one has done that yet.
Was this response just to call others ignorant and then suggest your favorite brands?
At least with my setup I prefer Amcrest over Reolink (based on compatibility and setup and plugins with the software I use). An open source option that just requires cameras, a computer, and a network is Scrypted + Frigate (+optional Google coral). Amcrest plugin in Scrypted to create a rebroadcast point, then capture it in Frigate for a birdseye view and other security features. If you have Google coral frigate can also do AI object detection (it can do it without but slower). Since Scrypted is the rebroadcast point I also have it hooked up to Apple home kit since I have an Apple TV 4K. Pretty happy so far.
Edit: Another comment mentioned https://gitlab.com/Shinobi-Systems/Shinobi which I haven’t explored but am definitely going to also try out now.
Personally I'm using Shinobi[1], but I've heard good things about frigate too if you're more into the ai detection and HA integration. There's a proxmox helper script one liner guided installation of you're into that [2]
I cringe every time I browse Amazon and see the random cam/cloud service/Play store apps that I know so many people are installing without questioning anything about them.
I would really like a nice consumer friendly product that either didn't use cloud storage at all or if it did, had end to end encryption of the video.
I don't have time to mess around with hacking a camera to do RTSP and then figure out how to set up and use something unfriendly like ZoneMinder.
Not open-source/audited, but Apple HomeKit Secure Video is light years ahead. Encrypted in-house on your own hardware (ATV or HomePod), they don't have the keys, excellent UX, hosted service, super easy to use. I'm all about the self host, but assuming Apple isn't straight up lying, they have build something that's too good and too easy. I buy cheapish HKSV cameras, and block them from accessing the WAN, so there's minimal Apple tax outside one hub.
NAS companies like synology have an offering too. The main problem is that they still live in 2012 when it comes to cpu power and sell vastly underpowered boxes
Price. I don't need CPUs to do rendering on my NAS, and wouldn't pay the premium for it.
The margins on these devices are already pretty high because they're enterprise and business focused - I "buy once cry once"'d when I bought my <vendor>* drive for the home. I would have bought a different vendor if I was forced to pay an additional premium for a render capable CPU.
(* - I'm probably tin foil hating here, but realized from a security posture perspective I don't want to publicly state the vendors I use in my network.)
This is one of the things Apple does right. HomeKit working local is a pretty great setup and just works. I put my HomeKit cameras on a VLAN without internet and device isolation and they still work seamlessly. The hard part is getting cameras that are wireless. I use scrypted but even then, getting ONVIF or RTSP isn’t as straight forward nowadays. I also have a local frigate backup which works great too. You can pipe in the detection to scrypted with MQTT.
I'm looking forward to the day HomeKit supports 2K and/or 4K video recordings.
Scrypted does this. I actually have no clue how it works but you get 2K streaming and recording.
Not my project but I have had great success with https://github.com/gtxaspec/wz_mini_hacks & V3 model.
The V3 models need to be downgraded to a specific firmware first and patching it exposes RSTP streams using https://github.com/AlexxIT/go2rtc. Everything doable without ever installing Wyze app on an environment air gapped environment with no internet.
I'm having great success with half a dozen v3's in tandem -- for $30 a camera, the quality is really unbeatable -- setup / notes below.
1. all cameras (firmware v4.36.9.139) have 64gb+ micro SD cards and record to local storage -- many people seem to have issues with anything greater than 32gb in v3's but I've found that this Verbatim tool [0] formats FAT32 at high capacity with no problems
2. all cameras have wz_mini_hacks [1] on the SD card with RTSP enabled
3. all cameras are connected via ethernet instead of wifi using this adapter [2] and wz_mini_hacks config
4. network blocks all outgoing internet connections for all cameras to keep them LAN-only -- this means I have to connect to VPN to review video when outside the house, but I'm cool with that
5. all RTSP streams are also recorded over the network via Agent DVR [3] to a NAS
6. the Wyze app (free tier, not paid) works normally with all of the above in place -- I find it much more intuitive to review recent videos in-app (streamed off the SD card), and then review the very occasional older video from a computer off the NAS (scrubbing through in VLC on a computer)
For what it's worth, I don't use them like a Ring camera where you're responding to realtime video events / talking through the camera to a delivery person -- this is mostly just for 24/7 recording. I have all object/motion detection events turned off, just a straight uninterrupted feed recording local and on the network.
Links:
0. https://www.verbatim.com/index/search.php?words=fat32+tool
1. https://github.com/gtxaspec/wz_mini_hacks
Something else I forgot to mention -- when you add the cameras to your network, give them a static DHCP lease by MAC address in your router (separate ones, for both wifi and ethernet if applicable)
If you don't and IP addresses cycle, your RTSP stream URLs will change and your recording software will just consider the camera disconnected. The Wyze app sometimes will get confused too (with ethernet in particular, if IP addresses change), and rebooting the cameras sometimes means physically cycling power if you don't know where it's at in your network. Better to just reserve a specific IP up front, write it down, and never think about it again.
I also use a lot of these types of crappy smart plugs [0] throughout the house (as well as starting to replace some with actual wifi outlets [1] for aesthetics) and it's actually really useful to be able to force a power cycle on certain things from my phone.
Links:
0. https://www.amazon.com/gp/product/B09LXGHR5X
1. https://www.amazon.com/KP200-Outlet-Required-Control-Certifi...
This is really awesome. Thanks for the links. I'm done with everything iot cloud, even though it takes a bit more work on the home server upkeep side.
This is highly helpful, thank you! Was wanting to build something like this.
Do you have any Wyze cams outdoors, but under cover of a porch? Or are they indoor only?
They’re all outdoors, although at least partially covered by something (roof overhang, gutter, etc). The cameras themselves aren’t truly waterproof and do have a “grill” cut into the back panel for the internal speaker, but since I don’t use them for 2-way audio, you can seal that off with rubber cement, flexseal, rubber tape, etc. Mine have been fine in all four seasons.
That said, all the modifications and tinkering is definitely less turn-key than some people prefer, but I’m a glutton for punishment haha
I have some V3s outside, somewhat covered by a rain gutter. They definitely get wet when the rain comes down sideways. Never had a problem.
I did this for a long time and then realized I was dealing with a lot of bugginess where I'd have to restart cameras, things were wired to wall warts, etc, all to save just a few bucks. I got some cheap poe cameras off Amazon with built in rtsp and its been great.
I finally gave up trying to use the mini hacks to make RTSP work reliably. I ended up using Wyze Bridge [0] instead, and it has been far more stable. Using Frigate for the web UI. It doesn't make for a local-only solution, but I don't use my cameras to record anything that would bother me if other people saw it.
As time and motivation permit, I've been converting the cameras I care about over to POE. But having to run cable across the house for each one means I haven't done them all.
This is one of the reasons why you want end-to-end encryption wherever possible.
Even a bad implementation with cloud-synced encryption keys (which defeats most of the benefits of e2e) would have stopped this.
The response in this case (notifying customers and specifically stating whether they were affected or not) is excellent, but this seems to be a repeat of a previous incident from September 2023: https://www.theverge.com/2023/9/8/23865255/wyze-security-cam...
Wow. That is decidedly not a good trend. Once is definitely bad enough, but twice…
There was another, different incident reported in 2022 spanning 3 years, and a data breach in 2019.
https://www.consumerreports.org/home-garden/home-security-ca...
I wonder how many people would continue to so casually use these services if they understood that, for the most part, there is rarely proper end-to-end encryption of their data with these services. It is awfully disingenuous when these companies' marketing materials describe their services as "encrypted" when it usually just means there are two independent TLS pipes, which both terminate in their "cloud"; this surely gives a false sense of security to end-users who may not understand the implications of such a setup.
Too many. I've consulted with friends who installed "smart" security cameras and other IoT devices. I really spelled it out, saying that there's a very real possibility that one day they'll find out someone's been listening in on all of their private conversations (audio) or watching them through their own cameras.
Responses typically range from "I'm not that interesting" to "I really don't care". I think it's too abstract of a threat for most people to take seriously before it happens to them.
Where do you charge your cell phone?
I totally agree with you, but then I put my phone on a qi charger on my nightstand and go to sleep. It's a device with both quality cameras and microphones, so I feel a little hypocritical given that there is a non-zero chance that someone could be listening or watching through my phone.
That's a possibility, but that would require an exploit and smartphones are far more secure and actively updated. I just keep on top of security patches and hope that's enough.
With IoT there often aren't any security patches and your audio & video are just being live streamed to the OEM's cloud waiting for someone to listen in, it doesn't even require a security exploit.
It's easily abused by employees, it even happened at Tesla where they watched their customers through the onboard cameras, taking screenshots of them walking around naked, and sharing them on company Slack channel for laughs.
That's why I find it so mind boggling, the company could incidentally hire a pervert and now you find yourself being watched in your own home by someone who knows your home address. I find this scary because it doesn't require a security exploit, just a deranged mind and those are dime a dozen.
So your issue is with the quality of the firmware on the devices and not the fact that it is a camera in a private place which is connected to the internet?
I agree with everything you're saying, but you may be overstating security patches. Until recently, most Android phones only had a few years of security updates.
I guess what I'm getting at is that if I truly believed in keeping Internet connected cameras outside of private areas I wouldn't have a smart phone at all.
The problem with Teslas wasn't the firmware on the cameras, but rather the infrastructure behind it. Ideally the data would be encrypted on servers and decrypted locally when needed. This doesn't pair nicely with services that perform analytics on video streams, of course, but it's a better option for privacy.
At the end of the day I share your concerns, and I want only devices which are controlled locally. I have been making efforts to make this a reality.
> So your issue is with the quality of the firmware on the devices and not the fact that it is a camera in a private place which is connected to the internet?
I'm just making a distinction between "connected to the internet" and "streaming private data to the cloud 24/7".
Most of us use a smartphone under the assumption that nobody else has access to it, and that it's not going to send all of our data to some cloud. If someone gains that kind of access to my device, I'll have bigger problems to worry about than someone listening to my conversations, like locking down bank accounts, investment accounts and changing dozens of passwords.
> Until recently, most Android phones only had a few years of security updates.
Tell me about it, I begrudgingly buy a new device when the old one runs out of security updates. I'm not a fan of Samsung or Pixel line (which now offer longer support) so I was planning to switch to an iPhone after my current Android device is made obsolete, but I changed my mind with Apple's latest EU meltdown.
“Don’t use Wyze” seems like the wrong takeaway from this.
I’d go with “don’t put internet-connected cameras in your house if you don’t want those images on the internet”. I’ve got a Wyze in my garage looking over my mountain bikes, and for $35 I don’t really care if somebody else sees that image. But I’d never put one in my living space, regardless of their security track record.
I have mine pointed at a mouse trap. It's been super useful, honestly, but I would never point a camera like this into a living space.
What is the point of filming your mountain bikes? Do you watch them from your office with fondness of your most recent ride? Will that prevent them from being stolen? I doubt so.
The primary use case for a camera watching something that might get stolen is to provide proof of theft for insurance. In some cases it can provide clear evidence of what exactly was stolen, and in some cases information about who stole it.
Unless the camera has lasers, it isn't very good at stopping said theft. ;)
Why would your insurance require a proof of the theft? That is not how insurances work usually.
My home insurance requires proof of theft.
Often you use a police report. My local police department is on an unofficial slowdown strike because they don't feel appreciated post-BLM. Possibly also because if they "forget" to file police reports or they "get lost in the system" then the official crime rate goes down. It took me over a month to get a police report I could send to insurance for a simple break in, and spent more of my labor by hourly wage trying to get that report than the cost insurance reimbursed.
Doing a "job action" like that because people have lost trust in your profession isn't going to restore that trust.
The only thing that will is to stop being trigger happy menaces to public safety.
Similar, but 6 or 7 years ago and with no discernible reason. The process of filing a police report and then compiling all the documentation required by insurance took longer and cost more by wage than I ultimately recovered through insurance.
I wish I could say it was still worth it because filing the report and providing the information about the theft gave the police additional info to use if they ever caught the thief, but nope. They didn't even want a copy of the video from the security cameras at the store in whose parking lot it happened. They made it acutely obvious that I was wasting their time by being there and that they had no interest whatsoever in doing what I naïvely thought was their job.
What an absolute waste of time.
> Similar, but 6 or 7 years ago and with no discernible reason. The process of filing a police report and then compiling all the documentation required by insurance took longer and cost more by wage than I ultimately recovered through insurance.
How could it cost more? Are you counting the time spent compiling the documentation?
Of course you need to provide proof of theft. I can't simply go to my insurance, file a claim for a high-value item, and expect them to not ask any more questions. This would be rife with fraud (read: not a reasonable business model) otherwise.
> Of course you need to provide proof of theft.
Generally filing a police report will suffice, if I’m not mistaken. Could you lie? Of course, but (1) it’s generally not in your best interest because the insurer might raise your rates and/or discontinue coverage, and (2) you’d be making a decision to make false statements to the police for purpose of fraud, which most reasonable people won’t do.
Other than perhaps using cameras as a means to deter thieves, I’m not sure that low-value (under USD 5000) items like bicycles are worth the time and effort for insurers to launch full investigations over.
"Other than perhaps using cameras as a means to deter thieves, I’m not sure that low-value (under USD 5000) items like bicycles are worth the time and effort for insurers to launch full investigations over."
I would agree, yet I have seen it happen. It is a somewhat difficult to predict path. I have seen some smaller claims (~$2000) take significantly longer and with more investigation than other very large (>$50,000) claims. I would assume there are certain metrics and algorithmic methods that effect how this plays out, so it is possible that someone who has had multiple small claims my get more attention than a single large claim.
Either way - The concept of the camera is simple in providing evidence in cases where that evidence can help you.
> I would agree, yet I have seen it happen.
You've seen insurance claims denied because someone didn't have an unverified video that allegedly showed a theft taking place, even with a police report in hand? (Seriously: How would an insurer know that you hadn't arranged to have your brother-in-law pretend to steal your bike for the camera?)
You said "...bicycles are worth the time and effort for insurers to launch full investigations over."
and I said:
"I would agree, yet I have seen it happen."
I was precise in answering your precise statement.
I have seen claims for less than $5000 items create significant investigative and delay-inducing efforts from an insurer. A video of the theft would have, in some of those cases, reduced that effort. Having video evidence is in almost no circumstances going to increase the time it will take to get an insurance claim paid. By you having the video evidence, you have the choice if you want to disclose it. It is just upside that you control.
If you do not want to put a camera in your garage pointing at your mountain bike, by all means do not.
Just like there's an imperfect but effective barrier to lying to the cops for a police report, there's an imperfect but effective barrier to staging a theft on a video camera.
You don't need a proof but a police report.
There is tremendous variance with what will be 'required' to complete an insurance claim. In some cases a simple statement will suffice. In others, a police report is sufficient. In some cases the insurance company will want to 'investigate'. In many cases the underwriter will have some additional requests.
However it is always good to recognize the primary goals of the insurance parties - which is to not pay. They have strong incentives in that direction, and you providing proof of theft reduces the opportunities for those incentives to slow or reduce the payment.
It is especially true in cases of undeclared specific items, like an expensive mountain bike that you do not have a dedicated policy on.
Yeah all they require is a police report.
This actually looks like a concurrency bug in their request handling code that may have stored the user id and camera id in shared variables, under load the wrong camera id is seen by a user. At least based on the description of what they say happened.
Yeah sounds like an issue with garbage collection/freeing up memory.
These are the kind of security gimmicks that I am terrified I will never fully internalize and, as a result, the code I might be proud of may in fact be a ticking bomb.
I would have liked them going more into the details of the caching issue. It sounds like they think the cache library was responsible for the issue, but a more technical analysis of what exactly went wrong within that library would be great.
My work encountered this same sort of thing after an outage. Our Redis instance or client got confused. If a=b and c=d in our cache, a request for a returned d randomly.
We quickly realized that cache is fast but not infallible. Use proper security on all your resources. Don’t rely on UUIDs to obfuscate your data as security.
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
That seems like enough of a line of bullshit to steer me away from ever using wyze.
Do you think the issue was something else? "People randomly see other people's content" is an issue that would immediately make me think some issue with caching is the culprit.
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
> Do you think the issue was something else?
No, I'm not questioning whether or not it was a caching issue. I'm taking exception to the lack of accountability. They chose the library. They (probably) chose to ignore a documented or common failure mode of caching systems through either poor choice of key or lack of synchronization. They've obviously designed their infrastructure in a way that isn't resilient to its current level of usage (cold start is a normal part of software's lifecycle).
They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
It's not even that: the quoted language doesn't even blame the library - it appears to blame increased load.
"As a result of increased demand, it mixed up device ID" - no, it mixed up IDs as a result of some sort of a concurrency bug. I don't understand the point of deflecting this far.
Likely to be a multi-threading issue; my bet is the cache client wasn't thread-safe. I've seen this in some apps before and the solution was to turn off multi-threading while we debug the library that was causing the issue.
This is the answer!
> I'm taking exception to the lack of accountability.
I'll bet money that their statement was run through legal and stripped of all possible blamey statements.
> They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
I agree. Companies need to own up to their fuckups, even with legal tells them that it can hurt. Because all companies will fuck up; how they handle it is the differentiator.
> I'm also curious what the actual bug was
Hardware. Rowhammer-type effects occurring accidentally under sudden load spikes. The hardware has just got too dense.
(I should clarify this is speculation, but reading the recent article included here on sudo using special maximum-distance bitfields to hold state internally (https://news.ycombinator.com/item?id=39165342)... it must be a problem that's being observed in the wild)
I can't imagine that happening with a sufficient frequency. A system making such mistakes so often would just be too unstable to keep an uptime >1h.
With the 'cattle not pets' mindset that pervades modern development is the lifespan of ephemeral cache VMs that closely monitored? They get spun up and down on demand in most architectures. I can see this being an edge case failure when the system is trying to scale up, the existing VMs are getting absolutely hammered, the hypervisor is trying to start up new ones, memory pressure and iops on the existing ones are maxed out...
It just seems like the most obvious root cause to me, a single bit-flip in a hashed value is going to give you the wrong result data without any other error because the hash value is already essentially heavily compressed, meanwhile the hash table is almost certain to be 100% stored in memory and very heavily accessed from multiple directions in a read/write manner.
No. That's not how that works.
How so? I've seen caching clients exhibit some really weird behaviour under heavy load. It's not beyond the pale that, eg, the caching library doesn't do proper locking before writing, resulting in writes stomping all over each other.
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
Yes but either the library is seriously bugged (like, expecting writes to be ordered and screwing up things if it gets too many writes for different objects at the same time) or there was some serious bug in their implementation. Anyway the attitude and the message passed in the communication seems like handwashing to me. I might be too cynic, though.
How else would you say a 3rd party library had a bug under heavy load? 1. You don't want a defamation lawsuit your way. 2. If it was vendor code, you have a contract that may be under a NDA. 3. If it was a vendor, lawyers, lots and lots of lawyers, they likely had to say the minimal amount. The fact they sent out communications for each type of incident in such a short time was great.
The problem is how much they're pointing fingers at the library in the first place.
I might be splitting hairs, but they say that the incident was "caused by a third party library" when in fact, the incident was caused by insufficient testing on their part.
It sounds like they're trying to shift blame for the incident but then they try to pat themselves on the back for all the effort they put into security. It comes across as dishonest.
Technical details are appreciated but they should've emphasized that this is their own fault. Bonus points if they commit to at least consider E2EE which would sidestep the issue.
Yeah I would very much likely to know what caching library has a failure mode of returning content for the wrong keys, that seems pretty bad if not a highly suspect explanation
Same thing happened to OpenAI. Will you steer clear of OpenAI forever as well?
... Yes?
The whole thing points at everyone but themselves... "Originated at AWS" then "caused by a caching library"
Very little ownership on Wyze's side.
There are only two hard things in Computer Science: cache invalidation and naming things.
-- Phil Karlton
And off-by-one errors
...and off-by-one errors!
I would at least want to know the client library so I can never use it for anything. Also never trust the client and I hope they don't mean the actual client app such that it can access other user id without server validation...
Coincidentally, I just cancelled my wyze service because the product and support are so terrible. I wanted a simple way to see if there was a package on my doorstep but instead I got something that alerts me when any dog, person, or vehicle goes down my street, and all I’ve gotten from support is robotic responses suggesting I update my firmware and ignoring my direct questions, running out the clock on my ability to return the thing. At this point I’m not surprised their engineering is bad and amused that it’s caused two different security incidents.
Device id and user ids are non unique?
hash collisions?
I'd bet it's this, plus something even stupider like hashing a connection timestamp millisecond as the "uniqueness" of the hash. I've seen a lot of terrible code implementations that assume that there will never be two clients connecting in the exact same millisecond
This sounds like a pretty decent guess to me. I bet you are right.
Sounds like its redis-py again...
sounds like a hashing function with insufficient entropy. "increased demand" would lead to a higher likelihood of hash collisions.
Not sure I follow. Hash functions don't require entropy, and a hash collision in a hash map shouldn't cause incorrect data to be returned (it just makes them less efficient).
I think that they are saying that the output space, i.e. the list of all possible hashes, is too small. Thus, IDs 1234 and 5678 lead to the same hash.
The collision is not in the insertion into the hash map but rather in the look up.
There's a bunch of things in here that don't really make sense:
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
What? How does load on the system affect correctness?
> The outage originated from our partner AWS
What does this mean? Was there an AWS outage for a service they use, or was this just a normal loss of an instance?
It's interesting that they blame external entities for the root causes of the incident and don't take responsibility for what is ultimately on them.
I assume the code was always incorrect, but only exhibits the problem in practice under high load. This could be a race-condition/data-race, or treating short hashes as unique.
It’s just a WAG, but I bet someone used a timestamp as a unique key, or at least part of one, so you were unlikely to get collisions except under load.
> What? How does load on the system affect correctness?
Seen this happen quite often with code that is not multi-thread safe, especially in languages like c# and java, such as using a static class property for data that should be request-scoped, or not using the appropriate concurrent collection classes etc.
Nobody should ever be surprised that sending video to someone else’s computer (ie “the cloud”) results in third parties viewing that video.
It’s 2024. Everything is connected to the internet. Dropbox, Google, and Apple all offer multiple terabyte level plans. The default today is to store in the cloud. We are all storing data in someone else’s computer.
Instead of blaming the users, we must hold the companies responsible. Data privacy laws must be stricter and these incidents must be taken more seriously.
And if the companies do not want to be responsible, empower the users to run things on their own without lockin.
You can indeed make laws to hold the companies responsible, but that's not going to change the situation where it's dumb to store private information on systems that aren't under your control.
You might be able to cause consequences after the fact, but your data will still get leaked first. You can't undo a privacy violation with tort law, and there won't ever be criminal penalties.
If you want your information to stay private, don't store it on other people's computers. IDGAF what "the default today" is. (Also, that's wrong - everyone that is serious and actually wants their data to remain private doesn't store it in the cloud. This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.)
A main draw of these “security” cameras is to be able to remotely monitor the locations where they are, including being able to events in the event the devices are stolen. Should I build my own off site, redundant, data centers to make this possible without using a cloud service?
> Also, that's wrong - everyone that is serious and actually wants their data to remain private doesn't store it in the cloud. This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.)
Do you think that “everyone that is serious and actually wants their data to remain private” is the default? In a random sampling of 100 people, how many do you think fall into this category?
> This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.
Is your threat model the same as Langley’s? Or might there just be different levels of what people’s needs are?
There’s ideals and there’s practicality. It’s impractical in today’s world to completely avoid cloud services. If you can do it, congratulations, more power to you.
Or buying cheap, no-brand or upstart brand cameras with cloud capabilities.
I had a heck of a time finding a proper POE recording DVR camera system for my mom's house without online or cloud bullshit, but still I isolated it on the network to not take any chances of UPnP port opening or dial-home crap.
The only system I would trust would be one that laid out their security model, source to their apps, and had a self-hosted server DVR option. The captological signals of 99.9% of security system websites do not instill confidence in my mind.
Got any brand/model recommendations? I also want to run full local.
I have 30 POE IP cameras connected to BlueIris (Running on a server). It records based on activity, and is entirely local (including being on a dedicated VLAN).
The cameras are a wide variety of IP POE cameras which is helpful as I am not stuck to a single brand.
Wyze cameras can actually be used very securely, as long as you bother to jump through some hoops.
First of all, google "Wyze RTSP firmware". It's the official firmware from the vendor that enables the RTSP protocol. Now you can enable RTSP via the app and give the camera a fixed IP address in your DHCP server.
RTSP is a pretty standard protocol, so you can now view the feed via VNC player, record it 24/7 via ffmpeg, use tools like motion, etc.
The camera will still try to connect to cloud, but you can move it to a local-only Wi-Fi network, or outright block it from reaching the outer world on the router side.
And if you want advanced stuff (multiple streams, organized recording, etc), there is a plethora of free/open-source security camera tools (iSpy for instance). It all takes time to learn and configure, but you can have your own fully closed-circuit surveillance network, while still using the Wyze's rather cheap hardware.
Instead of patching, you can also just use PoE cameras that are designed for this use case (local RTSP) and are only a little more expensive than Wyze. I’ve installed an Amcrest doorbell that works well with Scrypted and HomeKit, and plan on adding some Amcrest cameras like these soon: https://www.amazon.com/dp/B083G9KT4C
lazyweb: https://github.com/koush/scrypted (regrettably the licensing is "it depends" https://github.com/koush/scrypted/blob/v0.93.0/LICENSE.md#li... )
and don't overlook that user's other repos, as seems like there are quite a few fun things in there: https://github.com/koush?tab=repositories
> Wyze Update 04/05/2022: RTSP was considered a beta feature and we are currently assessing the path forward as the firmware versions have aged quite a bit. Wyze has removed the firmware files for these versions for now and will update the pages when plans are finalized. Please note that firmware files take a while to work on and test so you may not see an update in the near future. Wyze apologizes for the inconvenience.
>The outage originated from our partner AWS and took down Wyze devices for several hours early Friday morning. ... As we worked to bring cameras back online, we experienced a security issue. Some users reported seeing the wrong thumbnails and Event Videos in their Events tab. ... The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
As an software engineer who's dealt with caches for large high throughput services, this does not make sense to me why they are blaming a caching client. It's your own code that will decide what is the cache key, and what value to pass as the cache key. Did the caching library have a bug where when you ask for a given key, it returned results for a different key? Or more likely did your own code have a bug where you mixed up the keys? I think we need more details on what went wrong in here.
Lot of blaming others - for architecture, stack, configuration, and operational choices that are likey/should be own decisions that should come with taking ownership.
But they've built dashboards! They probably even bought products with a single pane of glass!
This company non stop spams me, might just have to ditch the cam. I have grown used to throwing away hardware due to infinite fees, self bricking, or hacked out of the box. Consumer electronics have taken a painful dive in quality control. And hey Wyze unsubscribe me already !!!
Looks like they had a similar issue in 2023. https://www.nytimes.com/wirecutter/blog/wyze-security-breach...
Pretty much the same thing that hit OpenAI, I wonder if it was the same redis bug.
> Wyze blamed "a third-party caching client library that was recently integrated into our system" for the trouble.
Yes, of course. Blame a third party library which was probably created by an open source maintainer instead of testing your own systems.
Of course, the reason this keeps happening is the infrastructure is designed to let it happen in certain cases. Notice how they explicitly say, they need to fix it in the front end. They can't fix it in the backend because that would break eavesdropping.
That’s a bit tinfoily take
Also "was recently integrated" - by ... magic?
This is the sort of thing that makes me salty that Unifi Protect is basically cloud locked in. No direct IP connection with "local" account support on the mobile app.
I recall seeing setting that this cloud access can be disabled, and you could just VPN to get your “local” only access back.
direct IP connection is coming: https://community.ui.com/questions/Unifi-Protect-Mobile-acce...
but even with this, it's still an security camera app that can't send push notification without cloud access.
I'm wondering about the probability that, out of all the affected customers, at least one had the research skills and social skills to identify another customer and successfully ask to meet. Like, for an essay about "His schnauzer needed a mom. WyZettle: the amazing story of a pivot from a home camera service to a dating app."
A little off topic, but how is it possible that a tech startup named itself “Wyze” and didn’t get sued by Google over the “Waze” trademark? In some accents it sounds exactly the same, and they’re sort of in an adjacent product space.
Am I the only one that saw Wyze and thought Wyse? Guess I'm old.
Right, that’s another one. I get that the trademark space is pretty dense, but still - trademarks exist to stop consumers getting confused, and this naming really is confusing?
Waze could come out with a line of dash cams, and then you'd have Waze cams and Wyze cams.
Trademarks are about confusing names in a similar product market. Just having a similar-ish sounding name doesn't mean it violates the trademark. Self-driving cars are pretty different market to home security cameras.
You're thinking of Waymo :) Waze is a crowdsourced traffic & maps thing. It's vaguely, if you squint, adjacent. I don't think it would be completely crazy, for example, for Waze to introduce a line of dash cams.
I have my cameras blocked from the internet and doing backups in the attic.
Are they wifi or closed circuit?
PoE / Ethernet
3rd party libraries are a serious security problem that nobody wants to talk about, because there is no easy solution. I once lost $300k to a hack related to that issue.
Initially I thought this was about the money transfer company. Curious how there are two software companies with the same name and one hasn’t sued the other.
Initially, I thought it was the turn-by-turn mapping app. Apparently, there are three wise companies.
How the hell these incidents happen?
In the era of cloud and microservices why each user does not have their own dedicated resources?
> How the hell these incidents happen?
Because race to the bottom.
People want incredibly cheap products that are internet connected. Your average home user should not have to worry (and won't) about cybersecurity concerns, so this will continue to happen. The only out I can foresee is government regulation stepping in to make these incidents actually hurt for the companies, but America has basically no appetite for that.
An individual S3 bucket or policy for each user?
Everything user specific. Nothing should operate across users. Why should it anyway. You only need to aggregate logs across users, cloud watch does this.
Well for one there is a hard limit of 1000 s3 buckets per account
Possibly a collision in the caching library. Pretty bad that the video streams aren't properly permissioned
connected cameras have been easily hackable circa 2001 and this isnt changing any time soon especially not with node.js smartcrap
Accidentally mixed the IDs!
Why do people have cameras inside their house anyway?
To make sure my cat feeder doesn't suddenly stop working and kill my cat while I'm away for a few days. Or other variations of similar.
Income