Monday, June 02, 2014

Details and Theories about the Station's Loss of Power

[Posted in September/2014 but back-dated to match the date of the email this is largely lifted from.]

Exploding CTDs have a way of distracting you from other issues, and I just wanted to clear up any remaining confusion about the station's status:  I believe it has lost not only communications but also power.  See the below graph of voltage levels, with the day-of-year on the x-axis, beginning at initial switch-on Tuesday afternoon and ending with a final drop in the early hours on Friday morning.  These data points are spaced thirty seconds apart.

Click on the graph to see a larger version.


The thing is, I don't see any sign that the batteries are being charged by the station solar panels.  None.  There is a gap of 1.5 hours on Wednesday morning when Jon had shut down the station, and after that point, voltages resume from a slightly higher point but continue to decline.  Normally I would expect to see *some* kind of up-and-down during periods of daylight and darkness, but here I see nothing.

Here are some possible explanations, none of which is entirely convincing to me:
  1. There could be a short in station electronics that is draining the batteries faster than the solar panels can charge them.  I suppose this is possible.  But we removed the exploded CTD and dummy-plugged its cable.  We also removed both BICs and dummy-plugged those cables, since we don't have any more underwater BICs in our inventory.  The only two underwater sensors are the Deep CTD, which continued to report good data through the end of this dataset, and the "groundtruth" CT, which we left ziptied to the station and likewise continued to report data until the end.  If there is a short, I don't know where it is coming from, except possibly from the brain itself.
  2. The solar panels might not be correctly connected to the brain.  But I have Jon's assurance that all five solar panel plugs were connected, and both battery plugs, and the grounding wire.  Furthermore, the solar panels are the only wires that use a two-prong plug on our stations, so it is impossible that they could have been plugged into the wrong plugs.
  3. There could be something wrong with one or more of the solar panels on this station.  This too seems unlikely, because from the evidence of the recovered dataset we see that the station continued to operate (without communications) in the entire period leading up to this week.
  4. There could be something wrong with the batteries.  But these are brand-new batteries that were purchased only a few months ago.  It's possible that in the process of installation the battery cables were damaged or otherwise disconnected, but Jon raised that very concern after installing the first battery and checked voltages on the battery plug after installation.
  5. There could be something wrong with the brain hardware.  This is perhaps the most likely explanation, but there are still points of evidence against it.  The central problem was that brain construction was not completed by April 25th as expected, and in fact was still incomplete at the time of our May 16th shipping cutoff.  So the parts of the brain that were finished were included in our shipment, and the rest of the brain was completed on May 23rd, the final lab day before travel.  Those final parts were carried to St. Croix by myself in my personal luggage.  This meant that I had to do some significant "brain surgery" on the evening of Monday, May 26th, in my hotel room, and the final assembled brain was never tested (!) before installation.  [This is also one reason why the CTD batteries had not been connected the night before deployment as is usual, but were connected on the boat the next morning immediately before deployment.]  However, arguing against this theory is the fact that the partial brain was tested for a bit more than a week on AOML's roof, and during that time it was showing every indication of solar-panel charging.  So if there is now a brain-based charging problem, it must be a new problem introduced during brain surgery on the night of the 26th.  This is certainly possible but not entirely likely.
Naturally the failure of communications/power (whatever its cause), coming as it does on top of the CTD explosion and the resulting structural damage, is hugely disappointing.  At this point there is clearly no value to the station as deployed, and some risk of further structural failure.

(signed) Mike Jankulak