2013 Swapout + Aftermath

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post summarizes what was known about the state of the station as of August 21, 2013, which was a few weeks after our 2013 annual swapout operations.]

In the week of July 29th - August 2nd, 2013, a team from AOML traveled to the USVI to do our annual swapout of electronics on our St. Croix CREWS station near Salt River Bay.  Traveling from AOML were Rachel Kotkowski (NOAA Corps) and Mike Jankulak (UMiami).  José Sanchez of the Department of Planning and Natural Resources (DPNR) provided a boat and acted as boat operator for this work.

The backstory of this operation is that there had been some uncertainty about the station's power infrastructure.  Its four rechargeable batteries were first deployed in September of 2006 and could now be reaching the end of their useful lifetimes.  Since November of 2012, the station had been dropping transmissions mostly in nighttime hours, which could indicate insufficient power levels when running only on battery power, i.e. without the aid of current from the solar panels during daylight hours.

We arrived knowing only that we wanted to evaluate the battery situation, and potentially replace the batteries with off-brand rechargeable batteries procurable on the island, since the Odyssey PC1200 rechargeable batteries favored by the CREWS engineers could not be obtained locally.  During equipment removal we realized that one of the metal eyelets connecting the batteries to the station had snapped, effectively isolating the station from two of its four batteries.  It is not known whether this had occurred long ago or whether it happened during our operations, but we replaced the broken eyelets and cleaned the battery terminals for reinstallation.

There are three aluminum masts on this station that hold our aerial instruments (anemometer/compass, integrated "WXT" weather transmitter, and "BIC" light sensor) and which must be lifted out of their mounts and lowered to the boat for recovery/replacement.  These masts are 2 - 3 ft of aluminum that slide into very close-fitting fiberglass tubes and then are secured with locking bolts.  Removing these masts is always the most (physically) difficult part of this operation, as the space between the aluminum and fiberglass tends to accumulate dirt and bird guano and rain, all of which tends to act as a kind of concrete.  Also, lifting the masts involves grabbing them at a point above one's head and using force to push them even higher.  In the case of the BIC, there is a fairly heavy instrument to contend with as well.  See the photo of the empty WXT hole, below.

The empty hole where the WXT's aluminum mast would go, as seen in 2010.  Click this photo to see a larger version.

At this station the anemometer mast is generally the hardest to remove, but it usually comes free with prolonged effort and the use of lines and zip ties to allow the climber to use their entire weight to pull on the mast.  The WXT is generally easiest, and the BIC a little more difficult because of the sensor's weight.  However, at this 2013 swapout the BIC mast would not budge even a little.  It would not winch side-to-side, it would not shift up or down, it was solidly locked in place beyond all attempts to move it.

When faced with this unexpected situation I (for the first time) climbed high enough on the station to stand atop the solar panels and secure my lines to the satellite antenna.  From this point I was able to loosen the hose clamps that held the BIC to its mast and remove the sensor that way.  Unfortunately, on-land testing the next day revealed our replacement BIC to have mysteriously failed, so in the end I was forced to re-deploy the same light sensor I had just removed.  Deploying the sensor from this very high position was a little trickier than removing it had been, and on the whole I would not consider this activity to be within reasonable bounds of safety.  In the future we must either find another way of loosening the Surface BIC mast and extracting it as before, or resign outselves to leaving this one instrument in place permanently until the station itself can be recovered to land and its masts/mounts repaired or redesigned.

Mike J+ atop the station during the 2013 swapout.  At the far right is the BIC mast, which could not be removed, with its sensor already removed for (intended) replacement.  Click on this photo to see a larger version.

Another instrument problem befell the Shallow CTD.  Once it was deployed on the station its communications appeared to be functional and its depth/pressure values were reasonable.  However its Conductivity, Temperature and Salinity measurements were all off the scales.  We are now planning to ship a replacement CTD to St. Croix for later swapout by Marlon Hibbert (NOAA's Coral Reef Conservation Program) and José (DPNR).

Another piece of the power-infrastructure puzzle was the Deep BIC.  In July of 2012 it started dropping some of its reports (i.e. reporting less than its expected 120 measurements hourly at 30-second intervals), and in September the problem grew worse.  By November, when the station's power levels were showing signs of trouble, the fear was that the sensor might have flooded and could short-circuit the entire station.  A dummy plug was shipped to USVI in January and by the end of the month Marlon had removed this sensor and plugged its cable.  In a piece of relatively good news, we found on this visit that a connector pin on this cable had broken which explains the sensor problems and likely means that the recovered BIC was not flooded/destroyed.  We replaced this cable, and then went on to replace the underwater cable for the "groundtruth" CT when we found that it, too, was not working properly.  It made for a very long Thursday afternoon but we left the station with all communications functional.  As it happens, it was pin #2 (+12V power input) that was broken (BIC) or loose (CT) on both damaged cables.

Note that this means four of the station's original five underwater cables have now been replaced:  Shallow CTD (August 2010), Deep CTD (May 2011), Deep BIC (August 2013) and Groundtruth CT (August 2013).  Only the Shallow BIC cable is still original to the station's redeployment in 2006.

Rachel seen during replacement of the underwater sensors.  Click on this photo to see a larger version.

One final note: as of August 21st, and with the benefit of three weeks' worth of collected data following these operations, it seems the news on the status of the batteries has turned sour.  We were initially pleased to observe that station power levels returned to 100% normal patterns for the first 11 days following our visit.  Then on August 13th we dropped one transmission during the dark hours.  For four days, we dropped one or two transmissions per night.  Then three transmissions.  Then for the following four days, 5 - 7 transmissions were dropped per night.  We are now back to the same level of performance that we had pre-visit.  See chart of 2013 station voltages, below:

Battery voltages plotted against Julian Day, 2013, through August 21st.  Click on this picture to see a larger version.

The best conclusion, given the available evidence, is that the station's batteries must be replaced during the next swapout operation.

(signed)
Mike Jankulak