Station is offline until further notice

The SRVI2 station as seen in 2011.

The CREWS station located near Salt River Bay in St. Croix, USVI, will be offline until further notice.  The station suffered severe damage to its structure and integrity on May 28th, 2014, when a newly-installed CTD exploded.  This incident is described in detail this blog post, with a summary of what is known about its causes in another blog post.  The station additionally (and perhaps unrelatedly) suffered a loss of communications on May 30th, whose possible causes are discussed in yet another blog post.

The first CREWS station at this site went live on June 15th, 2002.  This station was recovered to shore on June 20th, 2006 and redeployed (reusing the original underlying fiberglass structure but in a new design) on September 23rd, 2006.  This means that a CREWS station has collected data at this site for nearly twelve years, and for almost eight uninterrupted years in its current form.  This far exceeds the station's expected lifetime, which was estimated by some to be approximately five years.

The station is currently considered unsafe for climbing, and therefore irreparable.  Plans are being made for the removal of this station, and a second inoperative CREWS station in Puerto Rico (LPPR1), to take place possibly as early as late 2014 or early 2015.  It is possible that the station may eventually be replaced by a CREWS buoy similar to those that have recently been deployed in Belize, Tobago (BUTO1 and ARTO1), Little Cayman (CCMI2) and Barbados (DRBB1).

Further information on this station's removal and possible replacement will be posted on this blog as it becomes available.

Conclusions about the CTD Explosion

[Posted in September/2014 but back-dated to match the date of RDI/Teledyne's report following their analysis of the remnants of the CTD that exploded at our St. Croix station on May 28th, 2014.]

Following our return to Miami from St. Croix and the arrival of our return shipment of tools and instruments, including the remnants of the CTD that exploded, we decided to tell RDI/Teledyne about what had occurred.  This was not our first priority since for reasons largely unconnected to this incident the NXIC CTD product line (originated by Falmouth Scientific and later acquired by RDI/Teledyne) no longer figures very largely in our CREWS/ICON operations or inventory.  Which means that we had less-than-urgent interest in preventing a recurrence of this sort of failure but still a more-than-merely-academic curiosity about what may have caused the explosion.

This, then, was the email I sent to RDI on June 5th:

It's been suggested to me that I should send you a report of recent events.  I'm not asking for any kind of response on the part of RDI-Teledyne, this message is simply offered for your information.

Last week we installed two NXIC CTD-ADC instruments on our coral monitoring station in St. Croix, on Tuesday.  We returned to the station Wednesday morning to continue our work, and shortly after our arrival the shallow CTD (at a depth of about 2m) exploded.  We think this was caused by a buildup of gasses from the battery pack, which may have been corroding from a saltwater leak, possibly due to an ill-seated o-ring in the lid of the CTD.  This is all speculation and supposition, of course.

What we do know for sure is that the explosion was quite forceful.  The plastic tube of the CTD was shattered into small fragments its entire length, and one of the two "hammer-head" ends was blown off.  [The CTD was wrapped in a flexible mastic tape before deployment, for ease of cleaning when it is recovered, so many of the fragments were more or less held in one place by the tape.]

The CTD was hose-clamped to the body of our station, which is a 40-foot fiberglass stick deployed (since 2006) in about 20 feet of water in the near vicinity to a coral reef on the north coast of St. Croix, USVI.  When the CTD exploded, it left noticeable cracks in the station's fiberglass, and we have judged that the station is no longer safe to climb.  We are making plans to remove and possibly replace the station, which in any case has already exceeded its expected lifetime.

I should say that there were no injuries of any kind suffered as a result of this instrument failure.  No divers were in the water at the time of the explosion.

I thought you might appreciate seeing some photos relating to this event, so I am attaching three (I hope they arrive alright -- if not, please let me know and I will post them on our website for your retrieval).  The first photo shows the exploded CTD about ten minutes following the incident.  The second shows the damage to the station, where layers of yellow fiberglass can be seen where previously only black paint and marine growth was visible.  The third is a photo taken in my office in Miami, showing the parts of the recovered CTD.

We were already aware of the importance of careful o-ring placement and the risk of damage to the instrument itself if the o-rings were not correctly placed.  We weren't aware, however, that the failure mode of this instrument could be so... dramatic.  I have browsed through the manual of the instrument since returning from the field and I'm not sure I see any warning that this sort of explosion is a risk.  If such a warning is not present, I think you might want to insert one.  I know that I would have treated the o-ring and the CTD itself with a great deal more respect if I had known that this sort of violent explosion was a possibility.

The exploded CTD was s/n 1607.  That serial number dates back to a 2005 purchase from Falmouth but, following an initial 2005 deployment in the Bahamas we arranged to have it upgraded with ADC plugs and the longer case.  In its current (ADC) form it was delivered to us by Falmouth in mid-2006.  After that it was deployed 2006-7 in the Bahamas, recalibrated by FSI in late 2007, deployed 2008-9 in St. Croix, and recalibrated most recently by RDI/Teledyne on 12/22/2011.  Since 2011 until its deployment it has been kept in its box on a shelf in my climate-controlled office.

Two of the photos I included with this email message have already been posted in this blog entry,  and the third is reproduced below, along with a few others taken on my office workbench:

Click on this photo to see a larger version.

Click on this photo to see a larger version.

Click on this photo to see a larger version.

Click on this photo to see a larger version.

This message led to some further questions and answers and ultimately RDI asked if we would be willing to return the CTD remnants to them for analysis, with shipping costs covered by RDI.  We did so, and later we additionally shipped an unused battery pack that we had built for us by a third party and were in the practice of using in these CTDs (instead of buying battery packs directly from RDI).

We had ourselves been inclined to place most of the blame on a potentially mis-seated o-ring at the top of the CTD.  I had installed the CTD's battery pack Tuesday morning on the boat, moments before its deployment, and after the CTD's explosion on Wednesday morning one of our team (Jon Fajans) removed the CTD's cover and stated that, in his opinion, the o-ring appeared to have been twisted in a way that could have caused a slow leak of seawater to enter the body of the CTD.  This of course did not explain why the CTD reacted by exploding, rather than experiencing a slow flooding, short-circuiting, and quiet loss of communications as has occasionally been seen with other CTDs at other CREWS stations.

RDI/Teledyne had nothing to say about the o-ring, however.  Their focus, once their analysis had been completed, was entirely on the battery pack that we had commissioned a third party to make for us according to specifications we provided.  This is an excerpt from RDI's report:

Our engineering team has completed their analysis and while they were not able to determine the exact cause of the explosion they did found opportunities for improvement. One improvement will be clarification in documentation regarding the use of batteries and the other is adding a pressure relief valve. The documentation revision will occur shortly. The implementation of a relief valve will continue through engineering evaluation for feasibility, design, etc.

The investigation did raise a concern that warrants your attention. The TRDI battery packs for this system utilize diode protection to prevent charging of the alkaline batteries. Alkaline batteries are not meant to be charged and to do so could lead to failure of the batteries, fire and possibly explosion.  In analyzing the battery packs being used we identified that the battery packs are NOT diode protected and therefore should not be used. TRDI is in process of changing our documentation to provide a warning regarding use of batteries not provided by TRDI.

It is worth noting that our engineers developed the specs for these battery packs by deconstructing one of the battery packs provided to us by the CTD original manufacturers, Falmouth Scientific.  Assuming nothing was overlooked, it is reasonable to assume that FSI's original battery packs for these CTDs did not include any diode protection.  At some point after we started commissioning battery packs from a third party, it seems like FSI or RDI decided to change the design of those battery packs to include the aforementioned diode protection to prevent any charging of the alkaline batteries, but by this time we had a ready supply of diodeless battery packs to draw upon for each new CTD deployment.

So the final conclusions on this event are somewhat murky.  We have been using these same battery packs in multiple CTDs at multiple sites (including the second CTD, identical in design, that was deployed at the same time and under the same conditions as the CTD that exploded) for several years now.  So clearly the use of these non-diode-protected battery packs is not a sufficient condition for triggering an explosion like this one.  Additionally, other CTDs of this same design with the same third-party battery packs have before now suffered seawater leaks and have short-circuited, and they have done so without exploding.  So the exact combination of circumstances that led to this explosion are likely to remain unknown and unknowable.

As a kind of postscript, it is worth highlighting where RDI states that they plan to add a pressure relief valve to this model, which is an important safety enhancement.  The YSI "EXO Sonde" that has been used for conductivity/salinity and temperature measurements (not pressure/depth) at many of the newer buoy-style CREWS station also includes this pressure-valve safety feature.

(signed) Mike Jankulak

Details and Theories about the Station's Loss of Power

[Posted in September/2014 but back-dated to match the date of the email this is largely lifted from.]

Exploding CTDs have a way of distracting you from other issues, and I just wanted to clear up any remaining confusion about the station's status:  I believe it has lost not only communications but also power.  See the below graph of voltage levels, with the day-of-year on the x-axis, beginning at initial switch-on Tuesday afternoon and ending with a final drop in the early hours on Friday morning.  These data points are spaced thirty seconds apart.

Click on the graph to see a larger version.

The thing is, I don't see any sign that the batteries are being charged by the station solar panels.  None.  There is a gap of 1.5 hours on Wednesday morning when Jon had shut down the station, and after that point, voltages resume from a slightly higher point but continue to decline.  Normally I would expect to see *some* kind of up-and-down during periods of daylight and darkness, but here I see nothing.

Here are some possible explanations, none of which is entirely convincing to me:

1. There could be a short in station electronics that is draining the batteries faster than the solar panels can charge them.  I suppose this is possible.  But we removed the exploded CTD and dummy-plugged its cable.  We also removed both BICs and dummy-plugged those cables, since we don't have any more underwater BICs in our inventory.  The only two underwater sensors are the Deep CTD, which continued to report good data through the end of this dataset, and the "groundtruth" CT, which we left ziptied to the station and likewise continued to report data until the end.  If there is a short, I don't know where it is coming from, except possibly from the brain itself.
2. The solar panels might not be correctly connected to the brain.  But I have Jon's assurance that all five solar panel plugs were connected, and both battery plugs, and the grounding wire.  Furthermore, the solar panels are the only wires that use a two-prong plug on our stations, so it is impossible that they could have been plugged into the wrong plugs.
3. There could be something wrong with one or more of the solar panels on this station.  This too seems unlikely, because from the evidence of the recovered dataset we see that the station continued to operate (without communications) in the entire period leading up to this week.
4. There could be something wrong with the batteries.  But these are brand-new batteries that were purchased only a few months ago.  It's possible that in the process of installation the battery cables were damaged or otherwise disconnected, but Jon raised that very concern after installing the first battery and checked voltages on the battery plug after installation.
5. There could be something wrong with the brain hardware.  This is perhaps the most likely explanation, but there are still points of evidence against it.  The central problem was that brain construction was not completed by April 25th as expected, and in fact was still incomplete at the time of our May 16th shipping cutoff.  So the parts of the brain that were finished were included in our shipment, and the rest of the brain was completed on May 23rd, the final lab day before travel.  Those final parts were carried to St. Croix by myself in my personal luggage.  This meant that I had to do some significant "brain surgery" on the evening of Monday, May 26th, in my hotel room, and the final assembled brain was never tested (!) before installation.  [This is also one reason why the CTD batteries had not been connected the night before deployment as is usual, but were connected on the boat the next morning immediately before deployment.]  However, arguing against this theory is the fact that the partial brain was tested for a bit more than a week on AOML's roof, and during that time it was showing every indication of solar-panel charging.  So if there is now a brain-based charging problem, it must be a new problem introduced during brain surgery on the night of the 26th.  This is certainly possible but not entirely likely.

Naturally the failure of communications/power (whatever its cause), coming as it does on top of the CTD explosion and the resulting structural damage, is hugely disappointing.  At this point there is clearly no value to the station as deployed, and some risk of further structural failure.

(signed) Mike Jankulak

Swapout operations, NCRMP experiments, Damage to structure

[Note:  this is a back-dated blog entry, posted in September 2014, to document the events of the station swap-out that led to a moratorium on climbing this station.]

In the week of May 26th - 30th, AOMLers Ian Enochs and Mike Jankulak traveled to St. Croix, USVI, with C-ARMS contractor Jon Fajans for work at the CREWS station near Salt River Bay on the north coast of the island.  There were four goals: swap out all instruments on the station, install a new cellular modem to improve station communications, deploy a number of NCRMP (National Coral Reef Monitoring Program) experiments, and train Jon Fajans as a fully-redundant backup for Mike Jankulak's station-climbing activities.

These activities were interrupted by the unexpected and catastrophic failure of a RDI/Teledyne CTD within 24 hours of its deployment.  The CTD is believed to have exploded due to an internal buildup of gasses from its batteries, although the exact cause of the malfunction is not yet understood.  This explosion was witnessed by the team Wednesday morning and the resulting structural damage to the fiberglass pylon was documented in photos.  A decision was made to cut short all work on the station until the structure could be reevaluated for its fitness to climb, leaving the station with a mixture of instruments (four new, four old, three missing) and with key elements of the training exercise incomplete.  Although the station was left on Wednesday in a functional state, two days later it ran out of power because its solar panels were not correctly charging its batteries.

On the plus side, the NCRMP activities led by Ian Enochs resulted in the successfully deployment of three autonomous reef monitoring structures (ARMS), five calcification accretion units (CAUs) and five bioerosion monitoring units (BMUs).  On the CREWS side, the team was able to recover six months of station data that had not been expected to exist, following the station's apparent loss of power and cessation of communications in December of 2013, which considerably extends the data coverage at this site.

Three photos of the CTD and station, post-explosion, follow (photo credit: Ian Enochs).  Clicking on any of these photos will load the original, larger version of the image file.

(signed) Mike Jankulak

Here the exploded CTD is seen in two pieces connected by some previously-internal wiring.  Two hose clamps which had circled the CTD and the station are holding the CTD up by this internal wire.  The CTD's top cap blew off entirely and is hanging off the cable which connects to the rest of the station, while shards of the CTD's hard plastic shell have fallen to the ocean floor below (not shown).  The CTD was deployed after being wrapped in mastic tape, and that tape is the only thing holding these parts of the CTD together.

A relatively un-biofouled portion of the station shows where the previous CTD had been mounted, and where this CTD has been deployed in its place.  The station is build out of one long yellow fiberglass tube, although the portions of the station that are underwater were completely covered in black paint and no sign of yellow is normally visible.  Here is a view of cracks in the fiberglass resulting from the CTD's explosion.

This is a close-up of the damage to the station's structure.  The cracks in the fiberglass penetrate quite deeply and led to the team's decision to terminate operations atop the station.  It is believed that the structure is no longer safe for climbing.

Upcoming CREWS Swapout

In the week of May 26th - 30th a team from AOML will be visiting the CREWS station in St. Croix, USVI, that is located off the north coast of the island near Salt River Bay.  AOML's Ian Enochs and Mike Jankulak will be traveling to St. Croix, and contractor Jon Fajans will be joining them to receive training in station maintenance procedures.

The station has been offline since last December, and on this trip the station will be outfitted with a digital cellular modem to replace its GOES transmitter.  This will give AOML 24x7, instantaneous access to the station's complete data collection, as compared to the once-per-hour, 20-second, low-baud-rate GOES window used previously.  Data collection at this site by CREWS dates back to June of 2002.

This station is also a monitored site in NCRMP (the National Coral Reef Monitoring Program).  In service of this program, and as part of a collaboration with CRED (NOAA's Coral Reef Ecosystem Division in Hawai`i), the team will be installing ARMS (autonomous reef monitoring structures), CAUs (calcification accretion units) and BMUs (bioerosion monitoring units) to join the STRs (subsurface temperature recorders) that are already in place at the site.

(signed)
Mike Jankulak

Lightning Strike or Not? Photo Analysis

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is a slightly edited version of an email sent on April 16th, 2014, about our analysis of recent photos of the station and whether they supported the theory that a lightning strike had been the cause of the station's December communications failure.]

Thanks very much for sending photos! A picture really is worth a thousand words. I've looked over those six photos very carefully and my armchair-quarterbacking opinion is that there isn't any evidence of lightning damage, at least nothing observable from these photos.

Just to refresh everyone's memories, here is your Feb 26th report:

There must have been a large lightning strike, a couple of instruments at the top of the pylon showed blackened burn marks, and the lighting rod looked toasted, so we are assuming that a massive strike may have taken place.

Begging everyone's patience, here is my analysis of the state of the aerial instruments as shown in these photos. I am attaching a copy of your 008 photo that I've scribbled on to identify which instruments I'm talking about, which might help.

Marlon's photo as marked up by Mike Jankulak.  Click on the photo to see a larger version.

There are four bright white objects at the top of the station -- BIC (light sensor), WXT (integrated wind/weather transmitter), windbird (anemometer) and GOES antenna. The air temp in its radiation shield is also up there but somewhat lower down than the others.

I don't see any "blackened burn marks" on any of these, at least not visible in these photos. The closest to a black mark is on the BIC, on its S/SE side, visible from photos 007, 008, 009 and 017 (the side showing in photos 010 and 015 appears to be completely white). Here you can see the BIC's logo/label (see attached closeup photo of the BIC on my workbench) on the upper side. Also there appears to be some rust stains, orange/brown, below this, probably from the hose clamps that are holding the BIC to the aluminum mast. Some may recall that when we were there last August the BIC I'd brought with me was found to be dead on arrival, so I simply reattached the same BIC that had been there since 2011. So this same instrument has now been out there for almost 3 years.

A closeup of a Surface BIC on my workbench, showing its label/logo.  Click on this photo to see a larger version.

The windbird and the GOES antenna, as far as I can tell, are completely white.

The WXT too shows no signs of discoloration. At first I thought photo 007 might show that one of its three protruding acoustic wind sensors is bent or broken. Looking more closely though I think I'm just seeing one of the think metal bird spikes, slightly bent, and a little blurriness is making it look thicker than it really is.

Finally the lightning diffuser brush looks pretty good to me for its age, about the same as I remember (see attached photo from last August). Note that this is not a lightnight "rod", i.e. its purpose is not to attract lightning. Rather it is a diffuser, and when properly grounded it serves to "hide" the whole station from lightning activity and make it "appear" to the clouds as though it's just a stretch of featureless ocean. This lightning diffuser actually looks a lot healthier than the one at our Puerto Rico station, which has been under constant mistreatment by boobies.

Mike Jankulak in August of 2013, including a view of the station's lightning diffuser brush.  Click on this photo to see a larger version.

Anyhow, please speak up if you feel like there is something in these photos that I'm missing, or if you believe that there are further signs of lightning damage at the station that aren't well represented in these photos. But for now I am going to tentatively (and happily) conclude that a direct strike by lightning did not occur.

Naturally we will plan this next visit, as much as possible, to arrive carrying everything we would need to repair the station following an actual lightning strike. Better to be as prepared as possible.

(signed)
Mike Jankulak