Lessons learned: Do not overestimate Dynamic Positioning Systems’ capabilities

Dynamic positioning (DP) systems meeting equipment class 2 (DP2) and equipment class 3 (DP3) have control features and redundancies that are often regarded by the crews of vessels using these systems as infallible.

A recently investigated DP incident revealed that a DP system may still have weaknesses, even when the design and testing of the system aims to remove or mitigate those weaknesses through engineering controls.

The incident

During the recent incident, a DP3 drillship suffered a complete blackout and subsequent loss of position while conducting critical Outer Continental Shelf (OCS) activities.

The drillship was equipped with highly-complex power management, generator monitoring, control, and safety systems designed and tested to provide blackout protection and rapid blackout recovery.

This particular ship had equipment arrangements that exceeded minimum DP equipment class 3 requirements and was approved for, and operating with, the 11kV electrical system connected in a “closed-bus” configuration.

The drillship was performing critical well activities where a loss of position could result in personal injury, environmental pollution, or catastrophic damage.

Fortunately, these serious consequences were avoided due to the successful post-blackout actions taken by the crew, which included performing an emergency disconnect from the well using established procedures.

Probable cause

The subsequent investigation determined that the Main diesel generator (MDG) No. 6 experienced neutral over-voltage alarms that indicated a ground fault after the generator was disconnected from the bus. However, all electrical parameters appeared normal while MDG 6 was connected to the bus.

This same condition had been experienced on three separate occasions prior to the day of the incident, yet the ship continued to operate in the closed-bus configuration with an intermittent ground fault of unknown origin.

As a result, the affected generator was not inhibited from connecting to the bus. Additionally, the ship’s crew did not determine the cause of the neutral over-voltage alarms before disconnecting and reconnecting the affected generator numerous times in an effort to recreate the problem for troubleshooting purposes.

The MDG 6 intermittent ground fault subsequently developed into a high-level ground fault that propagated through the 11kV distribution system and damaged sensitive automated power management system components.

Potential transformer circuit breakers supplying switchboard under-voltage protection were tripped as a result and caused a complete blackout of the vessel, loss of position, and prevented the power management system from executing a blackout recovery.

Since key components of the automated power management system were damaged, the crew had to manually restore power to the 11kV switchboards.

The investigation determined that a defective vacuum interrupter (VI) caused a cascading electrical failure that ultimately caused the blackout.

A VI is a sealed component with extremely high reliability and this particular failure was not previously envisioned as a possibility. A microscopic pinhole was found in a seam weld during forensic analysis of the VI.

This allowed the VI to gradually lose vacuum and prevented the internal extinguishing agent from flooding the chamber to extinguish arcing when the breaker was opened and damaged the VI.

This damage allowed voltage spikes that caused the failure of two MDG 6 potential transformers and the associated high-level faults. The defective VI was not identified and degraded slowly enough to avoid initiating alarms that would have indicated a VI issue.

The degrading condition of the VI initiated the progressive failures that led to the complete blackout of this drillship.

Lessons learned

The USCG strongly recommends that owners and operators of mobile offshore drilling units (MODUs) and vessels equipped with DP2 and DP3 systems:

• Use an open-bus configuration as the preferred mode of DP operation while performing critical activities. This eliminates the possibility that an unforeseen failure could propagate through a closed-bus and limits the blackout to the affected bus while the unaffected bus(es) would retain some, although diminished, position keeping capabilities.
• Avoid conducting troubleshooting that may have unknown effects and cause a loss of power or position while DP MODUs or vessels are connected to the well (i.e., “latched up”) or conducting other critical activities. If power system components show signs of malfunction or intermittent faults, an open-bus configuration should be used to reduce the risk of blackout. In the event a critical generator alarm or an automated protection function is triggered, the affected generator should be isolated until the problem is identified through proper troubleshooting methods, to include review of the cause and effects associated with the alarm or safety function, review of available data logs, and consultation with the manufacturer(s) of the affected component(s).
• Follow DP guidance provided in the “DP Operations Guidance” prepared through the Dynamic Positioning Committee of the Marine Technology Society (MTS), Part 2, Appendix 1 (DP MODUs), Appendix 2 (DP Project/Construction Vessels or Appendix 3 (DP Logistics Vessels), as applicable to the type of vessel. See our notice published in the Federal Register (77 FR 26562 and 77 FR 62247) and https://www.dco.uscg.mil/OCSNCOE/DP/Regulatory-Guidance/#mts-guidance for additional details.
• Ensure proficiency in responding to DP system faults that require dynamic positioning operator or vessel engineer intervention, as discussed in Chapter 5 of MTS DP Operations Guidance, Part 2, Appendixes 1, 2 or 3, as applicable, to prevent escalation of the failure effect, loss of position, or actions that may circumvent the automatic protection initiated by the power management system.