Monday, July 28, 2014

Conclusions about the CTD Explosion

[Posted in September/2014 but back-dated to match the date of RDI/Teledyne's report following their analysis of the remnants of the CTD that exploded at our St. Croix station on May 28th, 2014.]

Following our return to Miami from St. Croix and the arrival of our return shipment of tools and instruments, including the remnants of the CTD that exploded, we decided to tell RDI/Teledyne about what had occurred.  This was not our first priority since for reasons largely unconnected to this incident the NXIC CTD product line (originated by Falmouth Scientific and later acquired by RDI/Teledyne) no longer figures very largely in our CREWS/ICON operations or inventory.  Which means that we had less-than-urgent interest in preventing a recurrence of this sort of failure but still a more-than-merely-academic curiosity about what may have caused the explosion.

This, then, was the email I sent to RDI on June 5th:
It's been suggested to me that I should send you a report of recent events.  I'm not asking for any kind of response on the part of RDI-Teledyne, this message is simply offered for your information.

Last week we installed two NXIC CTD-ADC instruments on our coral monitoring station in St. Croix, on Tuesday.  We returned to the station Wednesday morning to continue our work, and shortly after our arrival the shallow CTD (at a depth of about 2m) exploded.  We think this was caused by a buildup of gasses from the battery pack, which may have been corroding from a saltwater leak, possibly due to an ill-seated o-ring in the lid of the CTD.  This is all speculation and supposition, of course.

What we do know for sure is that the explosion was quite forceful.  The plastic tube of the CTD was shattered into small fragments its entire length, and one of the two "hammer-head" ends was blown off.  [The CTD was wrapped in a flexible mastic tape before deployment, for ease of cleaning when it is recovered, so many of the fragments were more or less held in one place by the tape.]

The CTD was hose-clamped to the body of our station, which is a 40-foot fiberglass stick deployed (since 2006) in about 20 feet of water in the near vicinity to a coral reef on the north coast of St. Croix, USVI.  When the CTD exploded, it left noticeable cracks in the station's fiberglass, and we have judged that the station is no longer safe to climb.  We are making plans to remove and possibly replace the station, which in any case has already exceeded its expected lifetime.

I should say that there were no injuries of any kind suffered as a result of this instrument failure.  No divers were in the water at the time of the explosion.

I thought you might appreciate seeing some photos relating to this event, so I am attaching three (I hope they arrive alright -- if not, please let me know and I will post them on our website for your retrieval).  The first photo shows the exploded CTD about ten minutes following the incident.  The second shows the damage to the station, where layers of yellow fiberglass can be seen where previously only black paint and marine growth was visible.  The third is a photo taken in my office in Miami, showing the parts of the recovered CTD.

We were already aware of the importance of careful o-ring placement and the risk of damage to the instrument itself if the o-rings were not correctly placed.  We weren't aware, however, that the failure mode of this instrument could be so... dramatic.  I have browsed through the manual of the instrument since returning from the field and I'm not sure I see any warning that this sort of explosion is a risk.  If such a warning is not present, I think you might want to insert one.  I know that I would have treated the o-ring and the CTD itself with a great deal more respect if I had known that this sort of violent explosion was a possibility.

The exploded CTD was s/n 1607.  That serial number dates back to a 2005 purchase from Falmouth but, following an initial 2005 deployment in the Bahamas we arranged to have it upgraded with ADC plugs and the longer case.  In its current (ADC) form it was delivered to us by Falmouth in mid-2006.  After that it was deployed 2006-7 in the Bahamas, recalibrated by FSI in late 2007, deployed 2008-9 in St. Croix, and recalibrated most recently by RDI/Teledyne on 12/22/2011.  Since 2011 until its deployment it has been kept in its box on a shelf in my climate-controlled office.
Two of the photos I included with this email message have already been posted in this blog entry,  and the third is reproduced below, along with a few others taken on my office workbench:

Click on this photo to see a larger version.

Click on this photo to see a larger version.

Click on this photo to see a larger version.

Click on this photo to see a larger version.


This message led to some further questions and answers and ultimately RDI asked if we would be willing to return the CTD remnants to them for analysis, with shipping costs covered by RDI.  We did so, and later we additionally shipped an unused battery pack that we had built for us by a third party and were in the practice of using in these CTDs (instead of buying battery packs directly from RDI).

We had ourselves been inclined to place most of the blame on a potentially mis-seated o-ring at the top of the CTD.  I had installed the CTD's battery pack Tuesday morning on the boat, moments before its deployment, and after the CTD's explosion on Wednesday morning one of our team (Jon Fajans) removed the CTD's cover and stated that, in his opinion, the o-ring appeared to have been twisted in a way that could have caused a slow leak of seawater to enter the body of the CTD.  This of course did not explain why the CTD reacted by exploding, rather than experiencing a slow flooding, short-circuiting, and quiet loss of communications as has occasionally been seen with other CTDs at other CREWS stations.

RDI/Teledyne had nothing to say about the o-ring, however.  Their focus, once their analysis had been completed, was entirely on the battery pack that we had commissioned a third party to make for us according to specifications we provided.  This is an excerpt from RDI's report:
Our engineering team has completed their analysis and while they were not able to determine the exact cause of the explosion they did found opportunities for improvement. One improvement will be clarification in documentation regarding the use of batteries and the other is adding a pressure relief valve. The documentation revision will occur shortly. The implementation of a relief valve will continue through engineering evaluation for feasibility, design, etc.

The investigation did raise a concern that warrants your attention. The TRDI battery packs for this system utilize diode protection to prevent charging of the alkaline batteries. Alkaline batteries are not meant to be charged and to do so could lead to failure of the batteries, fire and possibly explosion.  In analyzing the battery packs being used we identified that the battery packs are NOT diode protected and therefore should not be used. TRDI is in process of changing our documentation to provide a warning regarding use of batteries not provided by TRDI.
It is worth noting that our engineers developed the specs for these battery packs by deconstructing one of the battery packs provided to us by the CTD original manufacturers, Falmouth Scientific.  Assuming nothing was overlooked, it is reasonable to assume that FSI's original battery packs for these CTDs did not include any diode protection.  At some point after we started commissioning battery packs from a third party, it seems like FSI or RDI decided to change the design of those battery packs to include the aforementioned diode protection to prevent any charging of the alkaline batteries, but by this time we had a ready supply of diodeless battery packs to draw upon for each new CTD deployment.

So the final conclusions on this event are somewhat murky.  We have been using these same battery packs in multiple CTDs at multiple sites (including the second CTD, identical in design, that was deployed at the same time and under the same conditions as the CTD that exploded) for several years now.  So clearly the use of these non-diode-protected battery packs is not a sufficient condition for triggering an explosion like this one.  Additionally, other CTDs of this same design with the same third-party battery packs have before now suffered seawater leaks and have short-circuited, and they have done so without exploding.  So the exact combination of circumstances that led to this explosion are likely to remain unknown and unknowable.

As a kind of postscript, it is worth highlighting where RDI states that they plan to add a pressure relief valve to this model, which is an important safety enhancement.  The YSI "EXO Sonde" that has been used for conductivity/salinity and temperature measurements (not pressure/depth) at many of the newer buoy-style CREWS station also includes this pressure-valve safety feature.

(signed) Mike Jankulak