Even when it’s -2 and you’re 6,500 miles from your router, a good IT pro finds a way.
The good ship Akademik Tryoshnikov. Credit: Jen Thomas
Carles Pina i Estany is not what comes to mind when you picture your typical Polar explorer. A native of sunny Barcelona, he works as a Software Engineer at Mendeley—a London-based technology company owned by science publishers Elsevier. Before this year, he had never even slept aboard a ship. But when the invitation came for him to embark on a three-month expedition around the Antarctic, he jumped at the chance.
It all happened rather quickly. Pina i Estany’s partner, Jen Thomas, who had previously worked with the British Antarctic Survey, was working as Data Manager for a research trip led by the newly created Swiss Polar Institute. The SPI connects researchers active in polar or extreme environments, promotes public awareness of these environments, and facilitates access to research facilities in those extreme environments. Billionaire adventurer Frederik Paulsen sponsored the excursion—he even went along for the ride. This was most definitely not your typical office tech support gig.
Having an IT person onboard (in addition to the two maintenance and electronics engineers) was essential. Luckily, Thomas knew a guy. Suddenly Pina i Estany found himself alongside her aboard the Akademik Tryoshnikov, a research vessel set to undertake an ambitious Antarctic loop via Cape Town in South Africa, Hobart in Tasmania, and Punta Arenas in Chile.
During the IT professional’s short tenure as at-sea support, the Akademik Tryoshnikov pushed through bad weather, dealt with sub-par food, and navigated a telecommunications setup that never seemed to work properly. And while some troubleshooting is to be expected on any such trip, the fact that they’d had such little time to prepare the equipment before setting off presented Pina i Estany with a unique set of challenges right from the get-go. (As he optimistically put it, there were “more opportunities for me to solve things.”)
“I am 35 now, and I have been working with computers for over 20 years—hack days, evenings, weekends, personal projects,” he told Ars. “I tell you, in this expedition I used everything I know, even little bits of knowledge I thought were pretty useless. It all becomes useful in the Antarctic.”
When Pina i Estany’s unusual office time wrapped up this summer and he returned to London in late July, he found time to meet with Ars and regale our inner networking nerd with tales of IT at the high seas. If there’s anything to take away from his experiences, it’s that tech support in the most extreme places in the world isn’t so different from tech support in more familiar settings—that is, except for the lack of reliable connectivity and ability to snag parts and equipment from Amazon or your local shop. But the need for help is constant, the emotions can run high, and requests run the gamut from simple e-mail server stuff to things no computer science curriculum will train you for.
Our adventurous IT man gets an unusual assignment—anyone know how to troubleshoot a winch? Jen Thomas
The winch
One evening, about a month after setting sail, a researcher cheerfully approached Pina i Estany with what he called a “new challenge.” Up until then, he dealt with the sort of familiar equipment that was well within his comfort zone: computers, routers, hard drives, and Raspberry Pis.
But this support ticket fell into an entirely different league. A massive winch—which is a hauling mechanism made of a cable and a crank—had started to fail.
Until this point, Pina i Estany didn’t even know what the word “winch” meant (that was about to change, naturally). This particular winch was responsible for lowering the ship’s only CTD device, which collected and analyzed water. The CTD normally would be lowered to depths of up to 1,500 meters, and it proved crucial to most of the 22 separate research teams onboard.
The problem, as it turned out, was a software malfunction. And the error was interfering with the winch’s ability to lower the extensive cabling into the water smoothly. Pina i Estany attempted some debugging, but the manufacturer soon told him that inputting new parameters on the CTD winch computer remotely was impossible. Considering the Akademik Tryoshnikov was in the middle of the ocean, this presented a bit of a problem.
The solution, as it turned out, called for some “semi-hacking skills” in addition to willingness to brave the elements:
“The temperature was around 0 or -2, windy with lots of sea spray; the boat was rocking, and my hands were freezing. People were asking me what I was doing out there with a computer, but the CTD was connected with a very short network cable. So I had to work outside,” he tells me. “So I accessed the winch computer, which operated Windows CE, using my Linux machine. I discovered the IP from the boot screen; using nmap, I found that it had a remote desktop server. I had a moment of joy when I pressed enter and, Yes! I could change the parameters.”
Pina i Estany’s joy was short-lived, sadly; that didn’t quite solve the problem. After the winch manufacturers begrudgingly agreed to let him reinstall the software, he had to wait until they docked at Hobart for resupply in order to download it through the hotel’s Wi-Fi.
“So we reinstalled the software, and it still wouldn’t boot up,” Pina i Estany said. “That was one of the worst moments of the expedition IT-wise. The problem was that the CTD was one of the most essential bits of science equipment onboard, but you know, this is a big winch, and I really had no idea about it… ”
Luckily, this particular story had a happy ending. After reinstalling the update in a different way, the winch finally stopped acting up. Soon enough, all the scientists got their water samples safely collected, and Pina i Estany could finally take his computer back out of the cold.
Many tech support gigs deal will remote access point trouble at some point… Jen Thomas
Hacked networking
In another memorable help request, one day a scientist approached Pina i Estany with a machine she used to measure the reflection of light on the sea. She needed to retrieve data from the device but could only do that by connecting it to a remote access point.
“So I was like, ‘yeah, no problem, where’s the router?’” he recalled. “It wasn’t on the ship at all—it turned out to be somewhere in Australia.”
The extremes of this Cape Town to Punta Arenas loop are easily 6,500-miles-plus away from Australia, and Pina i Estany recalls being roughly 2,000 miles away from South Africa at this time. Not that this stopped him, of course, as he used his laptop as a remote access point to connect to the equipment’s FTP server to get the data. “But that was a bit of a faff, since every time they needed more data, they had to get me and my computer,” he said.
Instead, Pina i Estany looked to a simple yet favorite device for network hackers—a smartphone.
“I figured out a way to hack together a network for her by using an Android phone,” he explains. “With Android, you can set up a hotspot, even with no signal, so the device can connect to their own laptop via this phone. That way, the scientists could retrieve their data whenever they wanted without me getting involved.”
NAS systems brave the Antarctic Loop, too.
Credit: Carles Pina i Estany
NAS systems brave the Antarctic Loop, too. Credit: Carles Pina i Estany
Mail system
Try and cast your mind back—if you’re old enough—to the days of dial-up. You’d spend eons watching that progress bar as it downloaded a single e-mail or page. What was annoying back then is now maddening for those habituated to modern broadband speeds. But when you have a lot of people relying on an unreliable satellite Internet connection for sending and receiving all their e-mails, it’s a recipe for disaster. Things can quickly become tense.
“I have never seen so many people hitting their computers,” recalls Pina i Estany. “It was very painful for me. I can’t stand watching a scientist standing in front of a computer waiting for things to happen. It breaks my heart, seriously.”
Apart from the widespread frustration, a mounting e-mail backlog meant they couldn’t access some crucial expedition permits and documents. At one point, about 100MB of e-mails were stuck in the system, so Pina i Estany set out to retrieve them.
“It would have been easier to sail back to port to get them than to retrieve them at sea via Outlook,” he explains.
Of course, that kind of rerouting wasn’t an option. Instead, Pina i Estany accessed a remote server, downloaded and compressed all the e-mails to it, and then sent those compressed files to the ship using a piece of software called Rsync, which deals very well with unstable connections. He also wrote a script that meant if the program stopped downloading at any point, it would start again from the same place once a connection was re-established.
“So I left this program running for eight or nine hours and then opened this huge file using Thunderbird,” he said. “With that, I was able to get all the wanted e-mails, including the permits we needed.”
Still, the quick fix didn’t provide a solution to the main problem—the ship may have had these e-mails, but the expedition didn’t have a reliable communications system.
“I knew how to fix this, but I needed a proper Internet connection for a few hours to do it,” Pina i Estany said. “During our three-day break in port after leg one, Jen and I went to a hotel, where I set up a new website domain and webmail server, then created users for everyone. It was very popular. Everywhere I went on the ship, people were using the webmail I had deployed. That was amazing to see.”
For things to continue to run smoothly, however, Pina i Estany needed to limit the size of e-mails that could be sent over this system to 200kb, which meant setting up an alternative solution for sending larger files.
“I designed a system where the e-mails were downloaded in chunks,” he recalled. “It was a bit of a hack since e-mail protocols generally do not allow e-mails to be split—they download it all or not. I got around that so that if the download timed out at 20 percent, it tried again to download the rest of the e-mail rather than starting from zero.”
Pina i Estany also managed a queuing system to deal with the slow connection. With this in place, even though it might take five or 10 minutes for someone to retrieve their e-mail, users got a confirmation message straight away and knew that things were in hand.
“I was obsessed with being fair, so I was downloading e-mails in the order they were received,” he said.
The ferry box
Pina i Estany’s final anecdote involved something virtually every IT pro deals with these days: data management. Managing the vast amounts of data the expedition collected in real time was probably the most consistent (and huge) challenge that he applied his IT skills to, and the best example of this was the Ferry Box. This Linux-based machine remained in regular use, constantly measuring properties such as salinity and temperature of the water from the sea’s surface.
Because it took several days for each batch of collected data to be made available, however, scientists were left with little scope for forward planning. For stretches of time, the researchers could initially be unaware of important happenings, like crossing into different water bodies with distinctly different characteristics.
“So [to solve this], I figured out a way to feed that data into a website that the scientists could access in real time,” Pina i Estany said. “That way, they could make timely decisions about where to stop and take samples, for example.”
This functionality effectively meant that the researchers could shift from a reactive to a proactive approach in data collection. It’s not difficult to see how that would impact the science itself. And after a few tales from Pina i Estany, it’s not difficult to see how even the most extreme science relies on a humble IT pro, either.
Alice Bonasio is a technology journalist, author, and consultant. In addition to contributing to Ars on everything from VR treadmills to HoloLens applications, she runs her own Tech Trends blog and regularly freelances for Quartz, Fast Company, The Next Web, Wired, and others. She has a particular interest in Virtual/Mixed reality and is currently writing VRgins, a book about sex and relationships in the virtual age. Find her on Twitter: @alicebonasio.
