Details and Theories about the Station's Loss of Power

[Posted in September/2014 but back-dated to match the date of the email this is largely lifted from.]

Exploding CTDs have a way of distracting you from other issues, and I just wanted to clear up any remaining confusion about the station's status:  I believe it has lost not only communications but also power.  See the below graph of voltage levels, with the day-of-year on the x-axis, beginning at initial switch-on Tuesday afternoon and ending with a final drop in the early hours on Friday morning.  These data points are spaced thirty seconds apart.

Click on the graph to see a larger version.

The thing is, I don't see any sign that the batteries are being charged by the station solar panels.  None.  There is a gap of 1.5 hours on Wednesday morning when Jon had shut down the station, and after that point, voltages resume from a slightly higher point but continue to decline.  Normally I would expect to see *some* kind of up-and-down during periods of daylight and darkness, but here I see nothing.

Here are some possible explanations, none of which is entirely convincing to me:

1. There could be a short in station electronics that is draining the batteries faster than the solar panels can charge them.  I suppose this is possible.  But we removed the exploded CTD and dummy-plugged its cable.  We also removed both BICs and dummy-plugged those cables, since we don't have any more underwater BICs in our inventory.  The only two underwater sensors are the Deep CTD, which continued to report good data through the end of this dataset, and the "groundtruth" CT, which we left ziptied to the station and likewise continued to report data until the end.  If there is a short, I don't know where it is coming from, except possibly from the brain itself.
2. The solar panels might not be correctly connected to the brain.  But I have Jon's assurance that all five solar panel plugs were connected, and both battery plugs, and the grounding wire.  Furthermore, the solar panels are the only wires that use a two-prong plug on our stations, so it is impossible that they could have been plugged into the wrong plugs.
3. There could be something wrong with one or more of the solar panels on this station.  This too seems unlikely, because from the evidence of the recovered dataset we see that the station continued to operate (without communications) in the entire period leading up to this week.
4. There could be something wrong with the batteries.  But these are brand-new batteries that were purchased only a few months ago.  It's possible that in the process of installation the battery cables were damaged or otherwise disconnected, but Jon raised that very concern after installing the first battery and checked voltages on the battery plug after installation.
5. There could be something wrong with the brain hardware.  This is perhaps the most likely explanation, but there are still points of evidence against it.  The central problem was that brain construction was not completed by April 25th as expected, and in fact was still incomplete at the time of our May 16th shipping cutoff.  So the parts of the brain that were finished were included in our shipment, and the rest of the brain was completed on May 23rd, the final lab day before travel.  Those final parts were carried to St. Croix by myself in my personal luggage.  This meant that I had to do some significant "brain surgery" on the evening of Monday, May 26th, in my hotel room, and the final assembled brain was never tested (!) before installation.  [This is also one reason why the CTD batteries had not been connected the night before deployment as is usual, but were connected on the boat the next morning immediately before deployment.]  However, arguing against this theory is the fact that the partial brain was tested for a bit more than a week on AOML's roof, and during that time it was showing every indication of solar-panel charging.  So if there is now a brain-based charging problem, it must be a new problem introduced during brain surgery on the night of the 26th.  This is certainly possible but not entirely likely.

Naturally the failure of communications/power (whatever its cause), coming as it does on top of the CTD explosion and the resulting structural damage, is hugely disappointing.  At this point there is clearly no value to the station as deployed, and some risk of further structural failure.

(signed) Mike Jankulak

Swapout operations, NCRMP experiments, Damage to structure

[Note:  this is a back-dated blog entry, posted in September 2014, to document the events of the station swap-out that led to a moratorium on climbing this station.]

In the week of May 26th - 30th, AOMLers Ian Enochs and Mike Jankulak traveled to St. Croix, USVI, with C-ARMS contractor Jon Fajans for work at the CREWS station near Salt River Bay on the north coast of the island.  There were four goals: swap out all instruments on the station, install a new cellular modem to improve station communications, deploy a number of NCRMP (National Coral Reef Monitoring Program) experiments, and train Jon Fajans as a fully-redundant backup for Mike Jankulak's station-climbing activities.

These activities were interrupted by the unexpected and catastrophic failure of a RDI/Teledyne CTD within 24 hours of its deployment.  The CTD is believed to have exploded due to an internal buildup of gasses from its batteries, although the exact cause of the malfunction is not yet understood.  This explosion was witnessed by the team Wednesday morning and the resulting structural damage to the fiberglass pylon was documented in photos.  A decision was made to cut short all work on the station until the structure could be reevaluated for its fitness to climb, leaving the station with a mixture of instruments (four new, four old, three missing) and with key elements of the training exercise incomplete.  Although the station was left on Wednesday in a functional state, two days later it ran out of power because its solar panels were not correctly charging its batteries.

On the plus side, the NCRMP activities led by Ian Enochs resulted in the successfully deployment of three autonomous reef monitoring structures (ARMS), five calcification accretion units (CAUs) and five bioerosion monitoring units (BMUs).  On the CREWS side, the team was able to recover six months of station data that had not been expected to exist, following the station's apparent loss of power and cessation of communications in December of 2013, which considerably extends the data coverage at this site.

Three photos of the CTD and station, post-explosion, follow (photo credit: Ian Enochs).  Clicking on any of these photos will load the original, larger version of the image file.

(signed) Mike Jankulak

Here the exploded CTD is seen in two pieces connected by some previously-internal wiring.  Two hose clamps which had circled the CTD and the station are holding the CTD up by this internal wire.  The CTD's top cap blew off entirely and is hanging off the cable which connects to the rest of the station, while shards of the CTD's hard plastic shell have fallen to the ocean floor below (not shown).  The CTD was deployed after being wrapped in mastic tape, and that tape is the only thing holding these parts of the CTD together.

A relatively un-biofouled portion of the station shows where the previous CTD had been mounted, and where this CTD has been deployed in its place.  The station is build out of one long yellow fiberglass tube, although the portions of the station that are underwater were completely covered in black paint and no sign of yellow is normally visible.  Here is a view of cracks in the fiberglass resulting from the CTD's explosion.

This is a close-up of the damage to the station's structure.  The cracks in the fiberglass penetrate quite deeply and led to the team's decision to terminate operations atop the station.  It is believed that the structure is no longer safe for climbing.

Upcoming CREWS Swapout

In the week of May 26th - 30th a team from AOML will be visiting the CREWS station in St. Croix, USVI, that is located off the north coast of the island near Salt River Bay.  AOML's Ian Enochs and Mike Jankulak will be traveling to St. Croix, and contractor Jon Fajans will be joining them to receive training in station maintenance procedures.

The station has been offline since last December, and on this trip the station will be outfitted with a digital cellular modem to replace its GOES transmitter.  This will give AOML 24x7, instantaneous access to the station's complete data collection, as compared to the once-per-hour, 20-second, low-baud-rate GOES window used previously.  Data collection at this site by CREWS dates back to June of 2002.

This station is also a monitored site in NCRMP (the National Coral Reef Monitoring Program).  In service of this program, and as part of a collaboration with CRED (NOAA's Coral Reef Ecosystem Division in Hawai`i), the team will be installing ARMS (autonomous reef monitoring structures), CAUs (calcification accretion units) and BMUs (bioerosion monitoring units) to join the STRs (subsurface temperature recorders) that are already in place at the site.

(signed)
Mike Jankulak

Lightning Strike or Not? Photo Analysis

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is a slightly edited version of an email sent on April 16th, 2014, about our analysis of recent photos of the station and whether they supported the theory that a lightning strike had been the cause of the station's December communications failure.]

Thanks very much for sending photos! A picture really is worth a thousand words. I've looked over those six photos very carefully and my armchair-quarterbacking opinion is that there isn't any evidence of lightning damage, at least nothing observable from these photos.

Just to refresh everyone's memories, here is your Feb 26th report:

There must have been a large lightning strike, a couple of instruments at the top of the pylon showed blackened burn marks, and the lighting rod looked toasted, so we are assuming that a massive strike may have taken place.

Begging everyone's patience, here is my analysis of the state of the aerial instruments as shown in these photos. I am attaching a copy of your 008 photo that I've scribbled on to identify which instruments I'm talking about, which might help.

Marlon's photo as marked up by Mike Jankulak.  Click on the photo to see a larger version.

There are four bright white objects at the top of the station -- BIC (light sensor), WXT (integrated wind/weather transmitter), windbird (anemometer) and GOES antenna. The air temp in its radiation shield is also up there but somewhat lower down than the others.

I don't see any "blackened burn marks" on any of these, at least not visible in these photos. The closest to a black mark is on the BIC, on its S/SE side, visible from photos 007, 008, 009 and 017 (the side showing in photos 010 and 015 appears to be completely white). Here you can see the BIC's logo/label (see attached closeup photo of the BIC on my workbench) on the upper side. Also there appears to be some rust stains, orange/brown, below this, probably from the hose clamps that are holding the BIC to the aluminum mast. Some may recall that when we were there last August the BIC I'd brought with me was found to be dead on arrival, so I simply reattached the same BIC that had been there since 2011. So this same instrument has now been out there for almost 3 years.

A closeup of a Surface BIC on my workbench, showing its label/logo.  Click on this photo to see a larger version.

The windbird and the GOES antenna, as far as I can tell, are completely white.

The WXT too shows no signs of discoloration. At first I thought photo 007 might show that one of its three protruding acoustic wind sensors is bent or broken. Looking more closely though I think I'm just seeing one of the think metal bird spikes, slightly bent, and a little blurriness is making it look thicker than it really is.

Finally the lightning diffuser brush looks pretty good to me for its age, about the same as I remember (see attached photo from last August). Note that this is not a lightnight "rod", i.e. its purpose is not to attract lightning. Rather it is a diffuser, and when properly grounded it serves to "hide" the whole station from lightning activity and make it "appear" to the clouds as though it's just a stretch of featureless ocean. This lightning diffuser actually looks a lot healthier than the one at our Puerto Rico station, which has been under constant mistreatment by boobies.

Mike Jankulak in August of 2013, including a view of the station's lightning diffuser brush.  Click on this photo to see a larger version.

Anyhow, please speak up if you feel like there is something in these photos that I'm missing, or if you believe that there are further signs of lightning damage at the station that aren't well represented in these photos. But for now I am going to tentatively (and happily) conclude that a direct strike by lightning did not occur.

Naturally we will plan this next visit, as much as possible, to arrive carrying everything we would need to repair the station following an actual lightning strike. Better to be as prepared as possible.

(signed)
Mike Jankulak

Station Offline, NCRMP Sensors

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is based in part on an email composed on December 5th, 2013, and also describes the station's complete failure of communications that had occurred a few days earlier.]

Marlon Hibbert and José Sanchez visited the CREWS station on December 3rd, 2013.  In the wake of that visit Marlon wrote me with questions about two instruments that were found in the vicinity of the station, instruments that he had not previously noticed.  One was said to be attached to the station's support chains and labeled #1254 and the other was close to a coral head to the west of the station, labeled #1258.

These standalone or "hobo" temperature sensors were not directly related to the CREWS (Coral Reef Early Warning System) program, but were part of AOML's larger CHAMP (Coral Health and Monitoring Program) activities.  Another of our operations is under the auspices of NCRMP (National Coral Reef Monitoring Program) which monitors a large number of sites at coral reef ecosystems throughout the Gulf of Mexico, Coastal Florida, the Caribbean and the world.  In this case the NCRMP team had deployed some new sensors at the station site without keeping the CREWS team in the loop, and so our local colleagues could not be alerted to these developments beforehand.  I thanked Marlon for his report and assured him that the sensors were part of NOAA operations.

In more serious news, the station has ceased communications completely as of December 1st, 2013.  There is no obvious explanation for this failure, so we are left with the usual possibilities, which mainly involve various types of power failure or a GOES transmitter failure.  It does not seem likely that the problem stems from a flooded instrument because usually we see at least a few hours of sharply dropping voltage reports when that happens.  But given the power problems we've seen to one degree or another dating back to November of 2012, this failure of communications does not come as a complete surprise.

As resolved last August, on our next visit we will attempt to fix the power problem by installing new batteries.  This next trip is expected to take place in early-to-mid 2014.

(signed)
Mike Jankulak

Cleaning and calibration

On December 3rd, 2013 Park Ranger Jose Sanchez and Marlon Hibbert visited the station to perform needed calibration and cleaning. It was the second visit since the team from AOML visited in early August.

For the months of October and November no visits were accomplished due to either the weather or other unavoidable delays.

Conditions good

 
That said , the station appeared in fair condition. Due to the weather, we had the opportunity to clean the topside of the pylon,  normally it is near impossible to work at the top due to surge. The screens on the lower CTD were also changed.


The calibration begun @ approximately 1000 hrs and terminated @ 1400hrs.

Our next trip is planned for a few weeks time, before years end and contingent on weather conditions.




Jose at work on the chains

 

 

Before cleaning
 

After cleaning

 

A maintenance trip was planned and conducted on June 13, 2013. This was the first site visit since mid March due to logistics of getting to the site and poor weather conditions.

Park Interpretative Ranger ( Jose Sanchez) and Coral fellow ( Jenn Travis) assisted topside by Marlon Hibbert carried out routine cleaning of the station. Conditions were not conducive to attaching the CTD for calibration as wave conditions made it unsafe for the boat to pull alongside the stick.

Instead the team decided to that they would concentrate  efforts on the bottom portions of the pylon.  The top half of pylon has proven difficult to maintain and and can only be done in the calmest of weather as the surge and risk of injury is highest with increased wave action.

That said, Jose and Jenn did a great job in the time they spent, but reported coming back feeling as if they had been dragged along behind a bus , such were the conditions.