Settings

Theme

Goes-19 weather satellite enters Safe Hold mode

spaceweather.gov

175 points by yabones 2 days ago · 95 comments

Reader

dabluecaboose 2 days ago

Former GOES engineer here. At this point I'd almost be surprised if 19 didn't have something go wrong. We had issues on almost every other satellite. GOES-17 had the loop heat pipe anomaly(Supposedly from someone stepping on it in the cleanroom...), GOES-15 (IIRC) had a micrometeorite strike, and GOES-13 had a fuel tank anomaly right before deorbit.

GOES-16 and GOES-17 are on-orbit spares, so in the extremely unlikely event of a total failure there's at least another spacecraft on-orbit ready to take up station.

That said, I have every faith in the GOES team to get to the bottom of this. They're the best, and I often wish I was back there working with them.

  • Rygian a day ago

    From now on, every time I see the word "anomaly" I will assume it is an euphemism for "someone stepped on it".

  • fishgoesblub 2 days ago

    > Supposedly from someone stepping on it in the cleanroom

    I would be too embarrassed to return to work if I did that.

    • pooloo 2 days ago

      How do they track that? Is there a log book where someone has to write to?

      "Observed EMP555 step on loop heat pipe. Conducted visual inspection of the affected area; no damage found, pipe remains nominal."

      • robszumski 2 days ago

        I'm sure they take detailed close out photos and likely video of the entire handling process.

      • altairprime a day ago

        “Measurements taking of the person: foot and shoe weight, total weight including clothing, height and leg length, estimated velocity of travel at time, estimated duration of load”

      • KennyBlanken a day ago

        Since it went into space with the stepped-on part, clearly nobody DID write it up.

        They really don't care. If the satellite blows up or fails, they get to build another.

        • Alive-in-2025 a day ago

          I didn't but that's the case. More likely there was a limited chance to evaluate the issue and something like they had a limited time launch window, no replacement ready and it tested good. "Just put a little butter on it, it's fine".

  • GTP a day ago

    You expect cleanrooms to be extremely controlled environments where skilled technicians and engineers very carefully handle sensitive equipment... Then someone steps on a component :D

    • krisoft a day ago

      Yeah. Stuff happens.

      There is the famous case of dropping the NOAA N-Prime weather satellite on the ground. They were trying to turn it from vertical to horizontal and they forgot to bolt it to the adapter. Worse: multiple people signed the paperwork attesting that they verified that it was bolted down correctly. Pictures and details here: https://spaceflightnow.com/news/n0410/04noaanreport/

      Or the famous “space aligator” event. There the original plan was to launch a manned space craft and an unmanned module to do a docking test between them. But the shroud protecting the target module didn’t deploy properly. If i remember it right because the man who usually assembled that part had to leave during assembly because his wife was giving birth. Someone put it together but appearantely not correctly. The half opened shroud reminded the astronauts to the jaws of an angry aligator hence the name. Pictures: https://www.nasa.gov/missions/gemini/gemini-ix/gemini-ix-cre...

      Or to not only list American failures: this russian satelite launch failed spectacularly because the IMU was installed upside down. https://youtu.be/ycRVAcZC5R4?si=LRTS7sutKSp6HnGs This was designed to be impossible to do, but someone bent and forced the component to stay in place in the wrong orientation. Then someone else whose job was to check it just signed the paperwork without climbing into the location where he could have checked it.

      I love these cases. Because it shows to me how even though the trappings of high-tech we are all fundamentally just occasionally lazy, occasionally distracted monkeys banging rocks together.

      • fhdkweig a day ago

        > multiple people signed the paperwork attesting that they verified that it was bolted down correctly.

        In my experience, the more people who are assigned to do something, the more those people assume that someone else did it and slack off. Of course, I've never worked on something as difficult to fix as an orbital satellite.

        • superxpro12 a day ago

          The infamous "it fell in-between the seat and the console" problem.

          These happen constantly. It's always important to close the loop on these and make sure someone is assigned to it directly, not left to indirectly assume.

      • Suzuran 18 hours ago

        There was also a test of the Apollo Launch Escape System wherein an unmanned CSM was launched on a Little Joe rocket with the plan being to manually trigger the LES at a specific point in the flight. The problem was that the roll accelerometer in the Little Joe had been installed backwards, so it quickly developed a substantial roll rate once it left the ground and the resulting centrifugal force acting on the innards of the SM turned them into rapidly escaping outards. This tore the SM skin apart and satisfied the automatic abort conditions for firing the LES. The LES operated as designed, the CM was pulled free of the SM, then went on to become stable and reach successful recovery conditions. The result was determined to be a successful test of the LES and the mission did not need to be reflown.

      • ahazred8ta 14 hours ago

        In 1968, a team of Perkin-Elmer technicians were carrying the million dollar Copernicus space telescope mirror to the other side of the lab. They tripped over a piece of scrap lumber. Shaka, when the walls fell.

        • jsrcout 10 hours ago

          That's a new one for me. Do you happen to have any pointers to more info? I found a few mentions, but nothing substantial.

          • ahazred8ta 7 hours ago

            It was mentioned in http://www.terrydunkle.com/glass.php and in a Sky & Telescope magazine article from 1989 which explained why Perkin-Elmer began using very meticulous step by step procedure checklists after that. To prevent any major mistakes. Oh the irony.

      • bagels a day ago

        Upside IMU is my favorite if only for the stupidity and spectacular video that resulted.

      • KennyBlanken a day ago

        > Stuff happens.

        > There is the famous case of dropping the NOAA N-Prime weather satellite on the ground. They were trying to turn it from vertical to horizontal and they forgot to bolt it to the adapter. Worse: multiple people signed the paperwork attesting that they verified that it was bolted down correctly.

        That wasn't a case of "stuff happens." It was a case of systemic issues with defense contractor employees not doing the work they were supposed to be, cutting corners, and "not giving a shit because if it breaks the government will just order another one from us."

        It was basically fraud/grift/incompetence, not "stuff happens."

        > If I remember it right because the man who usually assembled that part had to leave during assembly because his wife was giving birth. Someone put it together but appearantely not correctly.

        And why was the process so poorly documented only one guy could do it right?

        Why didn't someone check the assembly?

        Answer: defense contractor laziness and incompetence.

    • walrus01 a day ago

      NOAA-19 was dropped on its side because somebody forgot to attach the bolts to the transport platform it was riding on.

      https://www.google.com/search?client=firefox-b-d&q=satellite...

  • doright a day ago

    > GOES-17 had the loop heat pipe anomaly(Supposedly from someone stepping on it in the cleanroom...)

    Was this reported anywhere in the news? Sounds like one of those "not even once" kind of mistakes.

dlgeek a day ago

From the latest update (https://www.ospo.noaa.gov/operations/goes/status.html#datafi...), it looks like they're already restoring systems.

``` Update #3: DCS and SAR have returned to service as of 1630Z. Engineers will now work to restore ABI and expect imaging to resume by 1900Z. Image navigation may be slightly degraded for the first hour after imaging starts. The GOES-19 instruments will be restored in the following order:

    ABI
    GLM
    SUVI
    CCOR-1/EXIS/MAG/SEISS

The recovery process to return all GOES-19 instruments to normal operations is projected to take approximately 8 hours.

Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known. ```

ImJasonH 2 days ago

https://www.nola.com/news/hurricane/weather-satellite-goes-1... explains a bit more what this is, and what this means.

> The main NOAA satellite for tracking Atlantic, Gulf Coast hurricanes is out until further notice

> GOES-19 is the main instrument used to identify tropical waves as they strengthen and move over the Atlantic, Caribbean and Gulf of Mexico, providing real-time tracking for forecasting.

dekhn 2 days ago

I love how "safe mode" for a satellite is basically: "extend solar panels, turn self towards sun, don't do anything unnecessary, wait for further instructions".

  • munificent 2 days ago

    My restorative mode is also "turn self towards sun, don't do anything unnecessary".

    • erikig a day ago

      "Turn your face to the sun and the shadows will fall behind you." Māori Proverb

  • pphysch 2 days ago

    They should rebrand it as "Praise the Sun" mode. We are sorry, GOES-19 is temporarily unavailable during a planned solar worship break of indefinite length.

    • dabluecaboose 2 days ago

      The cool thing about geostationary orbits is that they're far enough out that they get 24/7 sun (Except around the equinoxes). We could easily fit a solar worship break in the schedule in between imaging and momentum dumps.

      • Terr_ a day ago

        > a solar worship break

        It'd be easy to optimize with some additional religions too, since from that distance pretty much all possible Earthly [0] holy sites are in the same direction.

        [0] I just stumbled onto a fun grammar question! Is it "earthly" (non-supernatural) or is it Earthly (proper-adjective related to the planet), or both? I submit that it's possible to have an "earthly Martian holy-site", and therefore only the capitalized version works here. :p

      • m4rtink a day ago

        Sure, GEO sats are almost always in full sunlight. But still, if you loose orientation, you could end up with the solar panel wings pointing in the wrong direction or overheating parts of the satellite not meant to be in full sunlight for long time.

    • bee_rider 2 days ago

      Just don’t point the thing at any maggots.

    • farx 2 days ago

      Initiate Sol Invictus mode

Uncle_Brumpus 2 days ago

Interestingly, I noticed this in aproximately real time. I had been checking up on the visible-light geocolor composite images every hour or so to look at the massive plume of Canadian wildfire smoke that was turning the skies in the northeast dark orange yesterday.

I haven't interacted with the GOES site or cared too much about the image output until the last 2 days, and the it immediately broke. Somewhat humorous to me.

ls65536 2 days ago

Looks like they're making progress toward getting things restarted: "Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known." [0]

[0] https://www.ospo.noaa.gov/data/messages/2026/07/MSG_20260716...

tomnicholas1 2 days ago

Anyone interested in accessing GOES data at scale will find this interesting - I created a Zarr index over the 7 billion chunks of data in the GOES-16 archive.

https://www.earthmover.io/blog/virtual-zarr

  • runamuck 2 days ago

    I used to work for NOAA. I think the scientists would flip out over this. Did you share it?

    • tomnicholas1 a day ago

      Yes! But if you want to share it with anyone else that would be great, since we're advocating for fairly radical changes within a big bureaucracy here, as I'm sure you will appreciate :)

venzaspa 2 days ago

As an aside, I'm always surprised how US Gov websites look like they've been made in Dreamweaver in about 2006. Not even seemingly with a emphasis on usability either.

  • dabluecaboose 2 days ago

    While it may not be flashy, I personally find the GOES sites extremely useful. Things are often simply placed at obvious and expected URLs, so scraping or monitoring is extremely easy.

    I wrote the script that provides the GOES NavSum [1] and it pretty much just builds a standardized text file and drops it in the folder. The neat thing is that this makes it really easy to programmatically scrape and parse the data.

    I wrote a personal script at one point that would download the GOES-EAST CONUS image and both EAST and WEST full disk images and composite them into a wallpaper. At one point my server had 500GB of archived GOES imagery. I liked to joke with my former coworkers that I could report image anomalies before they notice because my desktop wallpaper would change every 10 minutes.

    [1] https://www.ospo.noaa.gov/resources/cemscs/navsum.txt

    • ranger207 2 days ago

      Hey, I have a script for updating my background too! I'm not archiving the old images though, but I've thought about it to make some cool animations

      • dabluecaboose 2 days ago

        Hah originally making an animation was my plan, but as so often happens it fell on the backburner and then I ended up with a massive archive. I just deleted it once I realized that A) Better archives exist elsewhere and B) I wasn't going to do anything with it.

        I still have the script somewhere. I should throw an LLM at it and see if I can't sand off a few rough edges.

        • mthoms 2 days ago

          It would be great if an LLM could be trained to generate interpolated images between the 10m intervals of the full disk geocolor product. The animations would be fantastic.

          I've got about 2.5 years worth of that imagery if someone knows a good way to do this on a budget.

          • zanecodes 2 days ago

            I doubt you need an LLM for that, a diffusion model or perhaps even a deep CNN could probably do a passable job. You could train it by taking consecutive triplets of images [A, B, C] and providing [A, C] as inputs with B as the expected output.

    • getpost 2 days ago

      There's an app that updates the desktop https://downlinkapp.com/

    • iberator 2 days ago

      make torrent of it

    • xd1936 2 days ago

      Maybe if the UX was nicer, you wouldn't need to write scrapers and parsers and could just use their site.

      • dabluecaboose 2 days ago

        We don't need a bloated React framework to show a plaintext file with the fuel tank levels. It's NOAA, not Microsoft.

        • irishcoffee 2 days ago

          > We don't need a bloated React framework

          Could have stopped there for 99% of websites

      • jefftk 2 days ago

        They're scraping to automatically update the wallpaper on their desktop. That's not something a website can do, even with fantastic UX.

        • lightedman 2 days ago

          "That's not something a website can do, even with fantastic UX."

          Actually, I used to have a live-updating website AS MY BACKGROUND. Windows 98 and Me, website used AJAX and Comet to make it happen.

          You used to be able to set websites as your desktop background.

          • queuebert 2 days ago

            I'm sure that was 100% secure, too. /s (I did the same thing back then.)

  • kube-system 2 days ago

    The ones that look old are old. The USG has newer design systems that you'll see used on many of the websites that have been redesigned more recently: https://designsystem.digital.gov/

    This admin gutted both NOAAs budget and workforce so a website redesign is probably low priority at the moment.

    • dylan604 2 days ago

      Sites like NASA's APOD have not changed by design. So many third parties have been built up around sites that any change [w|c]ould break so much for no effective gain. Same holds true when people ask why things like NOTAMs and even NOAA's alerts are formatted the way they are.

  • queuebert 2 days ago

    The link OP submitted appears to be a webpage displaying a screenshot of another web page, and the image aspect ratio has been altered. It's so comically bad it had to be on purpose, or someone is doing their web dev in MS Word.

    Edit: I think actually it's a screenshot of a screenshot even, and this appears to be the entire design of spaceweather.gov. What in the holy heck is going on there? This has to be a top 10 worst website designs of all time.

  • bityard a day ago

    Lots of the web still looks like this when you step outside the comfort zone of big tech search engines, content streaming sites, and social media.

  • pdntspa 2 days ago

    Why must everything look and be modern

  • kevin_thibedeau 2 days ago

    You can thank AccuWeather for nerfing any funding for site modernization. I'm surprised the tiled radar map hasn't had the Biden performance fixes reverted.

jubilanti 2 days ago

A safehold is like maintenance mode, shutting down all non-essential systems, after it detects something is wrong. Doesn't necessarily mean it is gone for good, but not a good sign.

ls65536 2 days ago

Very unfortunate timing given the ongoing wildfires and associated smoke spreading across eastern North America in recent days.

qwertox 2 days ago

Better link: https://www.ospo.noaa.gov/operations/goes/status.html

  • lolc 2 days ago
  • isaacdl 2 days ago

    I disagree. That just shows GOES-19 as "green", whatever that means. The OP link is also not very informative, but this link is even less so.

    • longwave 2 days ago

      The outage list at the top is up to date, but the main status page is nearly three months old - the last updated date at the end is April 20, 2026.

    • dabluecaboose 2 days ago

      > Please note: This status information on this website is generally updated on a monthly basis. Recent outages and anomalies on data flow are highlighted at the top of the page.

ck2 2 days ago

status page says

        SAFEHOLD HAS BEEN RESOLVED
> Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known.

* https://www.ospo.noaa.gov/operations/goes/status.html

* https://www.ospo.noaa.gov/operations/goes/status.html#datafi...

kittikitti a day ago

These environmental satellites affect our day-to-day including the accuracy of weather forecasts. I believe the GOES program must receive more government funding because GOES-19 is a single point of failure and there have been numerous funding concerns. This is a national security issue because lack of resources to mitigate emergency weather events can lead to increased loss of life.

ck2 2 days ago

is this old cache?

* https://cdn.star.nesdis.noaa.gov/GOES19/ABI/CONUS/GEOCOLOR/2...

* https://cdn.star.nesdis.noaa.gov/GOES19/ABI/CONUS/GEOCOLOR/G...

  • dabluecaboose 2 days ago

    Looking at the timestamp, that's from yesterday. Nominal product delivery happens ~every 10 minutes.

webdoodle 2 days ago

Invasion of Cuba incoming?

  • edot 2 days ago

    No, you can’t see ships with this satellite. Too small. Besides, Russia and China have way better satellites that they’ll happily share intel from with Cuba.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection