Wednesday, May 21, 2014

Upcoming CREWS Swapout

In the week of May 26th - 30th a team from AOML will be visiting the CREWS station in St. Croix, USVI, that is located off the north coast of the island near Salt River Bay.  AOML's Ian Enochs and Mike Jankulak will be traveling to St. Croix, and contractor Jon Fajans will be joining them to receive training in station maintenance procedures.

The station has been offline since last December, and on this trip the station will be outfitted with a digital cellular modem to replace its GOES transmitter.  This will give AOML 24x7, instantaneous access to the station's complete data collection, as compared to the once-per-hour, 20-second, low-baud-rate GOES window used previously.  Data collection at this site by CREWS dates back to June of 2002.

This station is also a monitored site in NCRMP (the National Coral Reef Monitoring Program).  In service of this program, and as part of a collaboration with CRED (NOAA's Coral Reef Ecosystem Division in Hawai`i), the team will be installing ARMS (autonomous reef monitoring structures), CAUs (calcification accretion units) and BMUs (bioerosion monitoring units) to join the STRs (subsurface temperature recorders) that are already in place at the site.

Mike Jankulak

Wednesday, April 16, 2014

Lightning Strike or Not? Photo Analysis

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is a slightly edited version of an email sent on April 16th, 2014, about our analysis of recent photos of the station and whether they supported the theory that a lightning strike had been the cause of the station's December communications failure.]

Thanks very much for sending photos! A picture really is worth a thousand words. I've looked over those six photos very carefully and my armchair-quarterbacking opinion is that there isn't any evidence of lightning damage, at least nothing observable from these photos.

Just to refresh everyone's memories, here is your Feb 26th report:

There must have been a large lightning strike, a couple of instruments at the top of the pylon showed blackened burn marks, and the lighting rod looked toasted, so we are assuming that a massive strike may have taken place.

Begging everyone's patience, here is my analysis of the state of the aerial instruments as shown in these photos. I am attaching a copy of your 008 photo that I've scribbled on to identify which instruments I'm talking about, which might help.

Marlon's photo as marked up by Mike Jankulak.  Click on the photo to see a larger version.

There are four bright white objects at the top of the station -- BIC (light sensor), WXT (integrated wind/weather transmitter), windbird (anemometer) and GOES antenna. The air temp in its radiation shield is also up there but somewhat lower down than the others.

I don't see any "blackened burn marks" on any of these, at least not visible in these photos. The closest to a black mark is on the BIC, on its S/SE side, visible from photos 007, 008, 009 and 017 (the side showing in photos 010 and 015 appears to be completely white). Here you can see the BIC's logo/label (see attached closeup photo of the BIC on my workbench) on the upper side. Also there appears to be some rust stains, orange/brown, below this, probably from the hose clamps that are holding the BIC to the aluminum mast. Some may recall that when we were there last August the BIC I'd brought with me was found to be dead on arrival, so I simply reattached the same BIC that had been there since 2011. So this same instrument has now been out there for almost 3 years.

A closeup of a Surface BIC on my workbench, showing its label/logo.  Click on this photo to see a larger version.

The windbird and the GOES antenna, as far as I can tell, are completely white.

The WXT too shows no signs of discoloration. At first I thought photo 007 might show that one of its three protruding acoustic wind sensors is bent or broken. Looking more closely though I think I'm just seeing one of the think metal bird spikes, slightly bent, and a little blurriness is making it look thicker than it really is.

Finally the lightning diffuser brush looks pretty good to me for its age, about the same as I remember (see attached photo from last August). Note that this is not a lightnight "rod", i.e. its purpose is not to attract lightning. Rather it is a diffuser, and when properly grounded it serves to "hide" the whole station from lightning activity and make it "appear" to the clouds as though it's just a stretch of featureless ocean. This lightning diffuser actually looks a lot healthier than the one at our Puerto Rico station, which has been under constant mistreatment by boobies.

Mike Jankulak in August of 2013, including a view of the station's lightning diffuser brush.  Click on this photo to see a larger version.

Anyhow, please speak up if you feel like there is something in these photos that I'm missing, or if you believe that there are further signs of lightning damage at the station that aren't well represented in these photos. But for now I am going to tentatively (and happily) conclude that a direct strike by lightning did not occur.

Naturally we will plan this next visit, as much as possible, to arrive carrying everything we would need to repair the station following an actual lightning strike. Better to be as prepared as possible.

Mike Jankulak

Thursday, December 05, 2013

Station Offline, NCRMP Sensors

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is based in part on an email composed on December 5th, 2013, and also describes the station's complete failure of communications that had occurred a few days earlier.]

Marlon Hibbert and José Sanchez visited the CREWS station on December 3rd, 2013.  In the wake of that visit Marlon wrote me with questions about two instruments that were found in the vicinity of the station, instruments that he had not previously noticed.  One was said to be attached to the station's support chains and labeled #1254 and the other was close to a coral head to the west of the station, labeled #1258.

These standalone or "hobo" temperature sensors were not directly related to the CREWS (Coral Reef Early Warning System) program, but were part of AOML's larger CHAMP (Coral Health and Monitoring Program) activities.  Another of our operations is under the auspices of NCRMP (National Coral Reef Monitoring Program) which monitors a large number of sites at coral reef ecosystems throughout the Gulf of Mexico, Coastal Florida, the Caribbean and the world.  In this case the NCRMP team had deployed some new sensors at the station site without keeping the CREWS team in the loop, and so our local colleagues could not be alerted to these developments beforehand.  I thanked Marlon for his report and assured him that the sensors were part of NOAA operations.

In more serious news, the station has ceased communications completely as of December 1st, 2013.  There is no obvious explanation for this failure, so we are left with the usual possibilities, which mainly involve various types of power failure or a GOES transmitter failure.  It does not seem likely that the problem stems from a flooded instrument because usually we see at least a few hours of sharply dropping voltage reports when that happens.  But given the power problems we've seen to one degree or another dating back to November of 2012, this failure of communications does not come as a complete surprise.

As resolved last August, on our next visit we will attempt to fix the power problem by installing new batteries.  This next trip is expected to take place in early-to-mid 2014.

Mike Jankulak

Wednesday, December 04, 2013

Cleaning and calibration

On December 3rd, 2013 Park Ranger Jose Sanchez and Marlon Hibbert visited the station to perform needed calibration and cleaning. It was the second visit since the team from AOML visited in early August.

For the months of October and November no visits were accomplished due to either the weather or other unavoidable delays.

Conditions good


That said , the station appeared in fair condition. Due to the weather, we had the opportunity to clean the topside of the pylon,  normally it is near impossible to work at the top due to surge. The screens on the lower CTD were also changed.

The calibration begun @ approximately 1000 hrs and terminated @ 1400hrs.

Our next trip is planned for a few weeks time, before years end and contingent on weather conditions.

Jose at work on the chains
Before cleaning
After cleaning


A maintenance trip was planned and conducted on June 13, 2013. This was the first site visit since mid March due to logistics of getting to the site and poor weather conditions.

Park Interpretative Ranger ( Jose Sanchez) and Coral fellow ( Jenn Travis) assisted topside by Marlon Hibbert carried out routine cleaning of the station. Conditions were not conducive to attaching the CTD for calibration as wave conditions made it unsafe for the boat to pull alongside the stick.

Instead the team decided to that they would concentrate  efforts on the bottom portions of the pylon.  The top half of pylon has proven difficult to maintain and and can only be done in the calmest of weather as the surge and risk of injury is highest with increased wave action.

That said, Jose and Jenn did a great job in the time they spent, but reported coming back feeling as if they had been dragged along behind a bus , such were the conditions.

Monday, October 21, 2013

Missing CTD located in Utah

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is based on email updates I wrote when FedEx located the CTD that we'd attempted to ship to St. Croix in August.] 

Following the resumption of normal activities after the US Government shutdown, I was put into touch with a NOAA employee whose role it was to act as liaison (and arbiter, in case of problems) between NOAA and FedEx.  She was able to accomplish within a few days what I'd been unable to do in two months:  get FedEx to look for the missing package.  And they found it!  It was found in some sort of waystation for lost items in Utah.

Initially I told them to just send it on its way to St. Croix as originally planned, but I thought the better of it and asked them to ship it back to me in Miami.  I figured I should reevaluate the CTD and make sure it hadn't suffered any damage during its two-month journey from Memphis to Utah.

As of this writing (late October) the plan is to evaluate the CTD and then send it (or another) on to St. Croix so that the Shallow CTD can be replaced as originally planned last August.

Mike Jankulak

Wednesday, September 18, 2013

Power Levels update, CTD non-update

[n.b. This is a backdated post, part of a May/2014 blog catchup.  Marlon Hibbert (NOAA's Coral Reef Conservation Program) had reported on September 5th that the station appeared to be offline, with concerns about a particularly bad lightning storm seen in the area the night before.  I'd been away from the office for a few weeks but penned this (slightly edited) reply on September 18th.]

I'm aware that there was some kind of brief interruption in the St. Croix data feed when you wrote this but I think it's been mostly online since then. It continues to drop transmissions overnight (which may be indicative of battery failure) -- but only sometimes. So I'm not sure if we know the whole story of what's going on with the power systems at this station. I'm attaching a more recent graph of station power levels, where you can see that power levels are falling abnormally low, below 12V, except when they, well, aren't.

Battery voltages plotted against Julian Day, 2013, through September 13th.  Click on this picture to see a larger version.

Anyhow I would encourage you to go ahead and do the September maintenance visit whenever you can.

About the CTD -- FedEx has dropped the ball so completely on this lost package that I'm almost speechless. They promise updates whenever I call but the last time they initiated any kind of contact with me was August 20th. But I kept them on the phone a good long time this morning and was given repeated promises that they would update me once more within 24 hours, so we'll see. I also sent them a series of photos I've taken of a similar CTD so hopefully those will help their lost+found department in their search.

I will let you know if I hear more about the missing CTD, and please do continue to send me your comments about the station's data feed if you notice something of interest. I do appreciate your attention and enthusiasm for the health and operation of the station.

Mike Jankulak

Wednesday, August 21, 2013

Replacement CTD shipped, lost

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post's content is a slightly edited version of emails sent on August  15th and 20th, 2013, about our attempt to ship a replacement instrument to St. Croix via FedEx.]

During our last annual swapout of equipment at the St. Croix CREWS station, the Shallow CTD failed on deployment.  Its depth/pressure readings were reasonable but its conductivities, temperatures and salinities were entirely outside of normal ranges.  A decision was made to ship out a replacement sensor so that our local colleagues (Marlon Hibbert and José Sanchez) could swap out the Shallow CTD during their next maintenance visit.

Unfortunately, FedEx appears to have lost this package and isn't being terribly responsive about looking for it.  I will provide updates as soon as we know more.  FedEx's link for this shipment is:

Mike Jankulak

2013 Swapout + Aftermath

[n.b. This is a backdated post, part of a May/2014 blog catchup.  This post summarizes what was known about the state of the station as of August 21, 2013, which was a few weeks after our 2013 annual swapout operations.]

In the week of July 29th - August 2nd, 2013, a team from AOML traveled to the USVI to do our annual swapout of electronics on our St. Croix CREWS station near Salt River Bay.  Traveling from AOML were Rachel Kotkowski (NOAA Corps) and Mike Jankulak (UMiami).  José Sanchez of the Department of Planning and Natural Resources (DPNR) provided a boat and acted as boat operator for this work.

The backstory of this operation is that there had been some uncertainty about the station's power infrastructure.  Its four rechargeable batteries were first deployed in September of 2006 and could now be reaching the end of their useful lifetimes.  Since November of 2012, the station had been dropping transmissions mostly in nighttime hours, which could indicate insufficient power levels when running only on battery power, i.e. without the aid of current from the solar panels during daylight hours.

We arrived knowing only that we wanted to evaluate the battery situation, and potentially replace the batteries with off-brand rechargeable batteries procurable on the island, since the Odyssey PC1200 rechargeable batteries favored by the CREWS engineers could not be obtained locally.  During equipment removal we realized that one of the metal eyelets connecting the batteries to the station had snapped, effectively isolating the station from two of its four batteries.  It is not known whether this had occurred long ago or whether it happened during our operations, but we replaced the broken eyelets and cleaned the battery terminals for reinstallation.

There are three aluminum masts on this station that hold our aerial instruments (anemometer/compass, integrated "WXT" weather transmitter, and "BIC" light sensor) and which must be lifted out of their mounts and lowered to the boat for recovery/replacement.  These masts are 2 - 3 ft of aluminum that slide into very close-fitting fiberglass tubes and then are secured with locking bolts.  Removing these masts is always the most (physically) difficult part of this operation, as the space between the aluminum and fiberglass tends to accumulate dirt and bird guano and rain, all of which tends to act as a kind of concrete.  Also, lifting the masts involves grabbing them at a point above one's head and using force to push them even higher.  In the case of the BIC, there is a fairly heavy instrument to contend with as well.  See the photo of the empty WXT hole, below.

The empty hole where the WXT's aluminum mast would go, as seen in 2010.  Click this photo to see a larger version.
At this station the anemometer mast is generally the hardest to remove, but it usually comes free with prolonged effort and the use of lines and zip ties to allow the climber to use their entire weight to pull on the mast.  The WXT is generally easiest, and the BIC a little more difficult because of the sensor's weight.  However, at this 2013 swapout the BIC mast would not budge even a little.  It would not winch side-to-side, it would not shift up or down, it was solidly locked in place beyond all attempts to move it.

When faced with this unexpected situation I (for the first time) climbed high enough on the station to stand atop the solar panels and secure my lines to the satellite antenna.  From this point I was able to loosen the hose clamps that held the BIC to its mast and remove the sensor that way.  Unfortunately, on-land testing the next day revealed our replacement BIC to have mysteriously failed, so in the end I was forced to re-deploy the same light sensor I had just removed.  Deploying the sensor from this very high position was a little trickier than removing it had been, and on the whole I would not consider this activity to be within reasonable bounds of safety.  In the future we must either find another way of loosening the Surface BIC mast and extracting it as before, or resign outselves to leaving this one instrument in place permanently until the station itself can be recovered to land and its masts/mounts repaired or redesigned.

Mike J+ atop the station during the 2013 swapout.  At the far right is the BIC mast, which could not be removed, with its sensor already removed for (intended) replacement.  Click on this photo to see a larger version.
Another instrument problem befell the Shallow CTD.  Once it was deployed on the station its communications appeared to be functional and its depth/pressure values were reasonable.  However its Conductivity, Temperature and Salinity measurements were all off the scales.  We are now planning to ship a replacement CTD to St. Croix for later swapout by Marlon Hibbert (NOAA's Coral Reef Conservation Program) and José (DPNR).

Another piece of the power-infrastructure puzzle was the Deep BIC.  In July of 2012 it started dropping some of its reports (i.e. reporting less than its expected 120 measurements hourly at 30-second intervals), and in September the problem grew worse.  By November, when the station's power levels were showing signs of trouble, the fear was that the sensor might have flooded and could short-circuit the entire station.  A dummy plug was shipped to USVI in January and by the end of the month Marlon had removed this sensor and plugged its cable.  In a piece of relatively good news, we found on this visit that a connector pin on this cable had broken which explains the sensor problems and likely means that the recovered BIC was not flooded/destroyed.  We replaced this cable, and then went on to replace the underwater cable for the "groundtruth" CT when we found that it, too, was not working properly.  It made for a very long Thursday afternoon but we left the station with all communications functional.  As it happens, it was pin #2 (+12V power input) that was broken (BIC) or loose (CT) on both damaged cables.

Note that this means four of the station's original five underwater cables have now been replaced:  Shallow CTD (August 2010), Deep CTD (May 2011), Deep BIC (August 2013) and Groundtruth CT (August 2013).  Only the Shallow BIC cable is still original to the station's redeployment in 2006.

Rachel seen during replacement of the underwater sensors.  Click on this photo to see a larger version.
One final note: as of August 21st, and with the benefit of three weeks' worth of collected data following these operations, it seems the news on the status of the batteries has turned sour.  We were initially pleased to observe that station power levels returned to 100% normal patterns for the first 11 days following our visit.  Then on August 13th we dropped one transmission during the dark hours.  For four days, we dropped one or two transmissions per night.  Then three transmissions.  Then for the following four days, 5 - 7 transmissions were dropped per night.  We are now back to the same level of performance that we had pre-visit.  See chart of 2013 station voltages, below:

Battery voltages plotted against Julian Day, 2013, through August 21st.  Click on this picture to see a larger version.
The best conclusion, given the available evidence, is that the station's batteries must be replaced during the next swapout operation.

Mike Jankulak