[n.b. This is a backdated post, part of a May/2014 blog catchup.
This post's content is based in part on an email composed on December 5th, 2013, and also describes the station's complete failure of communications that had occurred a few days earlier.]
Marlon Hibbert and José Sanchez visited the CREWS station on December 3rd, 2013. In the wake of that visit Marlon wrote me with questions about two instruments that were found in the vicinity of the station, instruments that he had not previously noticed. One was said to be attached to the station's support chains and labeled #1254 and the other was close to a coral head to the west of the station, labeled #1258.
These standalone or "hobo" temperature sensors were not directly related to the CREWS (Coral Reef Early Warning System) program, but were part of AOML's larger CHAMP (Coral Health and Monitoring Program) activities. Another of our operations is under the auspices of NCRMP (National Coral Reef Monitoring Program) which monitors a large number of sites at coral reef ecosystems throughout the Gulf of Mexico, Coastal Florida, the Caribbean and the world. In this case the NCRMP team had deployed some new sensors at the station site without keeping the CREWS team in the loop, and so our local colleagues could not be alerted to these developments beforehand. I thanked Marlon for his report and assured him that the sensors were part of NOAA operations.
In more serious news, the station has ceased communications completely as of December 1st, 2013. There is no obvious explanation for this failure, so we are left with the usual possibilities, which mainly involve various types of power failure or a GOES transmitter failure. It does not seem likely that the problem stems from a flooded instrument because usually we see at least a few hours of sharply dropping voltage reports when that happens. But given the power problems we've seen to one degree or another dating back to November of 2012, this failure of communications does not come as a complete surprise.
As resolved last August, on our next visit we will attempt to fix the power problem by installing new batteries. This next trip is expected to take place in early-to-mid 2014.
(signed)
Mike Jankulak
Thursday, December 05, 2013
Wednesday, December 04, 2013
Cleaning and calibration
On December 3rd, 2013 Park Ranger Jose Sanchez and Marlon Hibbert visited the station to perform needed calibration and cleaning. It was the second visit since the team from AOML visited in early August.
For the months of October and November no visits were accomplished due to either the weather or other unavoidable delays.
That said , the station appeared in fair condition. Due to the weather, we had the opportunity to clean the topside of the pylon, normally it is near impossible to work at the top due to surge. The screens on the lower CTD were also changed.
The calibration begun @ approximately 1000 hrs and terminated @ 1400hrs.
Our next trip is planned for a few weeks time, before years end and contingent on weather conditions.
For the months of October and November no visits were accomplished due to either the weather or other unavoidable delays.
 |
| Conditions good |
That said , the station appeared in fair condition. Due to the weather, we had the opportunity to clean the topside of the pylon, normally it is near impossible to work at the top due to surge. The screens on the lower CTD were also changed.
The calibration begun @ approximately 1000 hrs and terminated @ 1400hrs.
Our next trip is planned for a few weeks time, before years end and contingent on weather conditions.
 |
| Jose at work on the chains |
 |
| Before cleaning |
 |
| After cleaning |
A maintenance trip was planned and conducted on June 13, 2013. This was the first site visit since mid March due to logistics of getting to the site and poor weather conditions.
Park Interpretative Ranger ( Jose Sanchez) and Coral fellow ( Jenn Travis) assisted topside by Marlon Hibbert carried out routine cleaning of the station. Conditions were not conducive to attaching the CTD for calibration as wave conditions made it unsafe for the boat to pull alongside the stick.
Instead the team decided to that they would concentrate efforts on the bottom portions of the pylon. The top half of pylon has proven difficult to maintain and and can only be done in the calmest of weather as the surge and risk of injury is highest with increased wave action.
That said, Jose and Jenn did a great job in the time they spent, but reported coming back feeling as if they had been dragged along behind a bus , such were the conditions.
Park Interpretative Ranger ( Jose Sanchez) and Coral fellow ( Jenn Travis) assisted topside by Marlon Hibbert carried out routine cleaning of the station. Conditions were not conducive to attaching the CTD for calibration as wave conditions made it unsafe for the boat to pull alongside the stick.
Instead the team decided to that they would concentrate efforts on the bottom portions of the pylon. The top half of pylon has proven difficult to maintain and and can only be done in the calmest of weather as the surge and risk of injury is highest with increased wave action.
That said, Jose and Jenn did a great job in the time they spent, but reported coming back feeling as if they had been dragged along behind a bus , such were the conditions.
Monday, October 21, 2013
Missing CTD located in Utah
[n.b. This is a backdated post, part of a May/2014 blog catchup.
This post's content is based on email updates I wrote when FedEx located the CTD that we'd attempted to ship to St. Croix in August.]
Following the resumption of normal activities after the US Government shutdown, I was put into touch with a NOAA employee whose role it was to act as liaison (and arbiter, in case of problems) between NOAA and FedEx. She was able to accomplish within a few days what I'd been unable to do in two months: get FedEx to look for the missing package. And they found it! It was found in some sort of waystation for lost items in Utah.
Initially I told them to just send it on its way to St. Croix as originally planned, but I thought the better of it and asked them to ship it back to me in Miami. I figured I should reevaluate the CTD and make sure it hadn't suffered any damage during its two-month journey from Memphis to Utah.
As of this writing (late October) the plan is to evaluate the CTD and then send it (or another) on to St. Croix so that the Shallow CTD can be replaced as originally planned last August.
(signed)
Mike Jankulak
Following the resumption of normal activities after the US Government shutdown, I was put into touch with a NOAA employee whose role it was to act as liaison (and arbiter, in case of problems) between NOAA and FedEx. She was able to accomplish within a few days what I'd been unable to do in two months: get FedEx to look for the missing package. And they found it! It was found in some sort of waystation for lost items in Utah.
Initially I told them to just send it on its way to St. Croix as originally planned, but I thought the better of it and asked them to ship it back to me in Miami. I figured I should reevaluate the CTD and make sure it hadn't suffered any damage during its two-month journey from Memphis to Utah.
As of this writing (late October) the plan is to evaluate the CTD and then send it (or another) on to St. Croix so that the Shallow CTD can be replaced as originally planned last August.
(signed)
Mike Jankulak
Wednesday, September 18, 2013
Power Levels update, CTD non-update
[n.b. This is a backdated post, part of a May/2014 blog catchup. Marlon Hibbert (NOAA's Coral Reef Conservation Program) had reported on
September 5th that the station appeared to be offline, with concerns
about a particularly bad lightning storm seen in the area the night before.
I'd been away from the office for a few weeks but penned this (slightly edited) reply on September 18th.]
I'm aware that there was some kind of brief interruption in the St. Croix data feed when you wrote this but I think it's been mostly online since then. It continues to drop transmissions overnight (which may be indicative of battery failure) -- but only sometimes. So I'm not sure if we know the whole story of what's going on with the power systems at this station. I'm attaching a more recent graph of station power levels, where you can see that power levels are falling abnormally low, below 12V, except when they, well, aren't.
Anyhow I would encourage you to go ahead and do the September maintenance visit whenever you can.
About the CTD -- FedEx has dropped the ball so completely on this lost package that I'm almost speechless. They promise updates whenever I call but the last time they initiated any kind of contact with me was August 20th. But I kept them on the phone a good long time this morning and was given repeated promises that they would update me once more within 24 hours, so we'll see. I also sent them a series of photos I've taken of a similar CTD so hopefully those will help their lost+found department in their search.
I will let you know if I hear more about the missing CTD, and please do continue to send me your comments about the station's data feed if you notice something of interest. I do appreciate your attention and enthusiasm for the health and operation of the station.
(signed)
Mike Jankulak
I'm aware that there was some kind of brief interruption in the St. Croix data feed when you wrote this but I think it's been mostly online since then. It continues to drop transmissions overnight (which may be indicative of battery failure) -- but only sometimes. So I'm not sure if we know the whole story of what's going on with the power systems at this station. I'm attaching a more recent graph of station power levels, where you can see that power levels are falling abnormally low, below 12V, except when they, well, aren't.
 |
| Battery voltages plotted against Julian Day, 2013, through September 13th. Click on this picture to see a larger version. |
Anyhow I would encourage you to go ahead and do the September maintenance visit whenever you can.
About the CTD -- FedEx has dropped the ball so completely on this lost package that I'm almost speechless. They promise updates whenever I call but the last time they initiated any kind of contact with me was August 20th. But I kept them on the phone a good long time this morning and was given repeated promises that they would update me once more within 24 hours, so we'll see. I also sent them a series of photos I've taken of a similar CTD so hopefully those will help their lost+found department in their search.
I will let you know if I hear more about the missing CTD, and please do continue to send me your comments about the station's data feed if you notice something of interest. I do appreciate your attention and enthusiasm for the health and operation of the station.
(signed)
Mike Jankulak
Wednesday, August 21, 2013
Replacement CTD shipped, lost
[n.b. This is a backdated post, part of a May/2014 blog catchup.
This post's content is a slightly edited version of emails sent on August 15th and 20th, 2013, about our attempt to ship a replacement instrument to St. Croix via FedEx.]
During our last annual swapout of equipment at the St. Croix CREWS station, the Shallow CTD failed on deployment. Its depth/pressure readings were reasonable but its conductivities, temperatures and salinities were entirely outside of normal ranges. A decision was made to ship out a replacement sensor so that our local colleagues (Marlon Hibbert and José Sanchez) could swap out the Shallow CTD during their next maintenance visit.
Unfortunately, FedEx appears to have lost this package and isn't being terribly responsive about looking for it. I will provide updates as soon as we know more. FedEx's link for this shipment is:
https://www.fedex.com/fedextrack/?tracknumbers=796471727921&cntry_code=us
(signed)
Mike Jankulak
During our last annual swapout of equipment at the St. Croix CREWS station, the Shallow CTD failed on deployment. Its depth/pressure readings were reasonable but its conductivities, temperatures and salinities were entirely outside of normal ranges. A decision was made to ship out a replacement sensor so that our local colleagues (Marlon Hibbert and José Sanchez) could swap out the Shallow CTD during their next maintenance visit.
Unfortunately, FedEx appears to have lost this package and isn't being terribly responsive about looking for it. I will provide updates as soon as we know more. FedEx's link for this shipment is:
https://www.fedex.com/fedextrack/?tracknumbers=796471727921&cntry_code=us
(signed)
Mike Jankulak
2013 Swapout + Aftermath
[n.b. This is a backdated post, part of a May/2014 blog catchup. This post summarizes what was known about the state of the station as of August 21, 2013, which was a few weeks after our 2013 annual swapout operations.]
In the week of July 29th - August 2nd, 2013, a team from AOML traveled to the USVI to do our annual swapout of electronics on our St. Croix CREWS station near Salt River Bay. Traveling from AOML were Rachel Kotkowski (NOAA Corps) and Mike Jankulak (UMiami). José Sanchez of the Department of Planning and Natural Resources (DPNR) provided a boat and acted as boat operator for this work.
The backstory of this operation is that there had been some uncertainty about the station's power infrastructure. Its four rechargeable batteries were first deployed in September of 2006 and could now be reaching the end of their useful lifetimes. Since November of 2012, the station had been dropping transmissions mostly in nighttime hours, which could indicate insufficient power levels when running only on battery power, i.e. without the aid of current from the solar panels during daylight hours.
We arrived knowing only that we wanted to evaluate the battery situation, and potentially replace the batteries with off-brand rechargeable batteries procurable on the island, since the Odyssey PC1200 rechargeable batteries favored by the CREWS engineers could not be obtained locally. During equipment removal we realized that one of the metal eyelets connecting the batteries to the station had snapped, effectively isolating the station from two of its four batteries. It is not known whether this had occurred long ago or whether it happened during our operations, but we replaced the broken eyelets and cleaned the battery terminals for reinstallation.
There are three aluminum masts on this station that hold our aerial instruments (anemometer/compass, integrated "WXT" weather transmitter, and "BIC" light sensor) and which must be lifted out of their mounts and lowered to the boat for recovery/replacement. These masts are 2 - 3 ft of aluminum that slide into very close-fitting fiberglass tubes and then are secured with locking bolts. Removing these masts is always the most (physically) difficult part of this operation, as the space between the aluminum and fiberglass tends to accumulate dirt and bird guano and rain, all of which tends to act as a kind of concrete. Also, lifting the masts involves grabbing them at a point above one's head and using force to push them even higher. In the case of the BIC, there is a fairly heavy instrument to contend with as well. See the photo of the empty WXT hole, below.
At this station the anemometer mast is generally the hardest to remove, but it usually comes free with prolonged effort and the use of lines and zip ties to allow the climber to use their entire weight to pull on the mast. The WXT is generally easiest, and the BIC a little more difficult because of the sensor's weight. However, at this 2013 swapout the BIC mast would not budge even a little. It would not winch side-to-side, it would not shift up or down, it was solidly locked in place beyond all attempts to move it.
When faced with this unexpected situation I (for the first time) climbed high enough on the station to stand atop the solar panels and secure my lines to the satellite antenna. From this point I was able to loosen the hose clamps that held the BIC to its mast and remove the sensor that way. Unfortunately, on-land testing the next day revealed our replacement BIC to have mysteriously failed, so in the end I was forced to re-deploy the same light sensor I had just removed. Deploying the sensor from this very high position was a little trickier than removing it had been, and on the whole I would not consider this activity to be within reasonable bounds of safety. In the future we must either find another way of loosening the Surface BIC mast and extracting it as before, or resign outselves to leaving this one instrument in place permanently until the station itself can be recovered to land and its masts/mounts repaired or redesigned.
Another instrument problem befell the Shallow CTD. Once it was deployed on the station its communications appeared to be functional and its depth/pressure values were reasonable. However its Conductivity, Temperature and Salinity measurements were all off the scales. We are now planning to ship a replacement CTD to St. Croix for later swapout by Marlon Hibbert (NOAA's Coral Reef Conservation Program) and José (DPNR).
Another piece of the power-infrastructure puzzle was the Deep BIC. In July of 2012 it started dropping some of its reports (i.e. reporting less than its expected 120 measurements hourly at 30-second intervals), and in September the problem grew worse. By November, when the station's power levels were showing signs of trouble, the fear was that the sensor might have flooded and could short-circuit the entire station. A dummy plug was shipped to USVI in January and by the end of the month Marlon had removed this sensor and plugged its cable. In a piece of relatively good news, we found on this visit that a connector pin on this cable had broken which explains the sensor problems and likely means that the recovered BIC was not flooded/destroyed. We replaced this cable, and then went on to replace the underwater cable for the "groundtruth" CT when we found that it, too, was not working properly. It made for a very long Thursday afternoon but we left the station with all communications functional. As it happens, it was pin #2 (+12V power input) that was broken (BIC) or loose (CT) on both damaged cables.
Note that this means four of the station's original five underwater cables have now been replaced: Shallow CTD (August 2010), Deep CTD (May 2011), Deep BIC (August 2013) and Groundtruth CT (August 2013). Only the Shallow BIC cable is still original to the station's redeployment in 2006.
One final note: as of August 21st, and with the benefit of three weeks' worth of collected data following these operations, it seems the news on the status of the batteries has turned sour. We were initially pleased to observe that station power levels returned to 100% normal patterns for the first 11 days following our visit. Then on August 13th we dropped one transmission during the dark hours. For four days, we dropped one or two transmissions per night. Then three transmissions. Then for the following four days, 5 - 7 transmissions were dropped per night. We are now back to the same level of performance that we had pre-visit. See chart of 2013 station voltages, below:
The best conclusion, given the available evidence, is that the station's batteries must be replaced during the next swapout operation.
(signed)
Mike Jankulak
In the week of July 29th - August 2nd, 2013, a team from AOML traveled to the USVI to do our annual swapout of electronics on our St. Croix CREWS station near Salt River Bay. Traveling from AOML were Rachel Kotkowski (NOAA Corps) and Mike Jankulak (UMiami). José Sanchez of the Department of Planning and Natural Resources (DPNR) provided a boat and acted as boat operator for this work.
The backstory of this operation is that there had been some uncertainty about the station's power infrastructure. Its four rechargeable batteries were first deployed in September of 2006 and could now be reaching the end of their useful lifetimes. Since November of 2012, the station had been dropping transmissions mostly in nighttime hours, which could indicate insufficient power levels when running only on battery power, i.e. without the aid of current from the solar panels during daylight hours.
We arrived knowing only that we wanted to evaluate the battery situation, and potentially replace the batteries with off-brand rechargeable batteries procurable on the island, since the Odyssey PC1200 rechargeable batteries favored by the CREWS engineers could not be obtained locally. During equipment removal we realized that one of the metal eyelets connecting the batteries to the station had snapped, effectively isolating the station from two of its four batteries. It is not known whether this had occurred long ago or whether it happened during our operations, but we replaced the broken eyelets and cleaned the battery terminals for reinstallation.
There are three aluminum masts on this station that hold our aerial instruments (anemometer/compass, integrated "WXT" weather transmitter, and "BIC" light sensor) and which must be lifted out of their mounts and lowered to the boat for recovery/replacement. These masts are 2 - 3 ft of aluminum that slide into very close-fitting fiberglass tubes and then are secured with locking bolts. Removing these masts is always the most (physically) difficult part of this operation, as the space between the aluminum and fiberglass tends to accumulate dirt and bird guano and rain, all of which tends to act as a kind of concrete. Also, lifting the masts involves grabbing them at a point above one's head and using force to push them even higher. In the case of the BIC, there is a fairly heavy instrument to contend with as well. See the photo of the empty WXT hole, below.
 |
| The empty hole where the WXT's aluminum mast would go, as seen in 2010. Click this photo to see a larger version. |
When faced with this unexpected situation I (for the first time) climbed high enough on the station to stand atop the solar panels and secure my lines to the satellite antenna. From this point I was able to loosen the hose clamps that held the BIC to its mast and remove the sensor that way. Unfortunately, on-land testing the next day revealed our replacement BIC to have mysteriously failed, so in the end I was forced to re-deploy the same light sensor I had just removed. Deploying the sensor from this very high position was a little trickier than removing it had been, and on the whole I would not consider this activity to be within reasonable bounds of safety. In the future we must either find another way of loosening the Surface BIC mast and extracting it as before, or resign outselves to leaving this one instrument in place permanently until the station itself can be recovered to land and its masts/mounts repaired or redesigned.
 |
| Mike J+ atop the station during the 2013 swapout. At the far right is the BIC mast, which could not be removed, with its sensor already removed for (intended) replacement. Click on this photo to see a larger version. |
Another piece of the power-infrastructure puzzle was the Deep BIC. In July of 2012 it started dropping some of its reports (i.e. reporting less than its expected 120 measurements hourly at 30-second intervals), and in September the problem grew worse. By November, when the station's power levels were showing signs of trouble, the fear was that the sensor might have flooded and could short-circuit the entire station. A dummy plug was shipped to USVI in January and by the end of the month Marlon had removed this sensor and plugged its cable. In a piece of relatively good news, we found on this visit that a connector pin on this cable had broken which explains the sensor problems and likely means that the recovered BIC was not flooded/destroyed. We replaced this cable, and then went on to replace the underwater cable for the "groundtruth" CT when we found that it, too, was not working properly. It made for a very long Thursday afternoon but we left the station with all communications functional. As it happens, it was pin #2 (+12V power input) that was broken (BIC) or loose (CT) on both damaged cables.
Note that this means four of the station's original five underwater cables have now been replaced: Shallow CTD (August 2010), Deep CTD (May 2011), Deep BIC (August 2013) and Groundtruth CT (August 2013). Only the Shallow BIC cable is still original to the station's redeployment in 2006.
 |
| Rachel seen during replacement of the underwater sensors. Click on this photo to see a larger version. |
 |
| Battery voltages plotted against Julian Day, 2013, through August 21st. Click on this picture to see a larger version. |
(signed)
Mike Jankulak
Subscribe to:
Posts (Atom)