Assessment of Utility Components Using Airborne Remote Sensing

The present invention generally relates to utility systems, and more particularly to monitoring and inspecting utility system components.

The North American power grid has been characterized by the Smithsonian Institution as the largest machine ever built by mankind. The size, geographic diversity, environmental diversity, and the many components comprising the power grid present unique challenges in the rapid and efficient system upgrading with diverse new technologies that realize America's objective of improved power grid reliability and hardening. Accordingly, utility systems are an integral part of modern day life. Unfortunately, components of these systems may become inoperable. For example, consider an electrical power substation that is part of a power grid. Substations perform various functions, such as transforming voltage, connecting two or more transmission lines, transferring power, and protecting the grid from short circuits and overload currents. In many instances, substation equipment is susceptible to damage, which may result in power outages throughout the grid. Power outages decrease customer satisfaction, and damaged substation equipment increases costs incurred by the utility provider.

Reducing time to action enables emergency responders to provide lifesaving support in major disaster events such as hurricanes. In order to most effectively allocate resources, those in charge of restoration and response efforts need to understand where the most highly impacted areas are, and for electric utilities, the focus is on downed poles and wires.

Damage to electric utility infrastructure may be caused by a variety of scenarios such as tropical cyclones, tornadoes, lightning storms, hailstorms, blizzards, wildfires, earthquakes, tsunamis, avalanches, landslides, volcanic eruptions, or manmade disasters such as dam failures or bombings. No matter the cause of asset damage at scale, utility resource planners need a way to effectively know which areas have considerable damage, such as downed poles and wires.

Reducing time to action enables emergency responders to provide lifesaving support in major disaster events such as hurricanes. In-order to most effectively allocate resources those in charge of restoration and response efforts need to understand where the most highly impacted areas are—and for electric utilities the focus is on downed poles and wires.

Damage to electric utility infrastructure may be caused from a variety of scenarios such as tropical cyclones, tornadoes, lightning storms, hailstorms, blizzards, wildfires, earthquakes, tsunamis, avalanche, landslides, volcanic eruptions, or manmade disasters such as dam failures or bombings. No matter the cause of asset damage at scale, utility resource planners need a way to effectively know which areas have considerable damage such as downed poles and wires.

Disclosed is a system and method to reduce the time needed to identify infrastructure that has been damaged due to a storm, earthquake, or other event. At a high level, the presently claimed invention includes the following steps. Step 1: Assigned airborne response equipped with high-powered lidar sensors to fly over impacted areas to collect a 3D point cloud. This data focuses on the 3D geometry of the built environment and may be processed in a highly automated fashion to derive the locations of downed poles and wires. Step 2: run automated processes to identify highly impacted areas—providing an output of precise XY locations of downed poles and wires. And step 3: develop unique resource allocation response given the areas of known major damage.

More specifically, disclosed is a system and method for detecting an abnormality on a utility infrastructure. The method begins with establishing a historical basis of scan data of utility infrastructure. This historical basis is created by selecting a predefined flight path from a plurality of predefined flight paths based a location of utility infrastructure to be inspected. An aerial vehicle is instructed to traverse the at least one predefined flight path that has been selected. Remote sensing during flight is accomplished using Light Detection and Ranging (LiDAR) remote sensing during the aerial vehicle traversing the predefined flight path, to create a first set of scan data in a 3D coordinate system of the utility infrastructure along with corresponding location coordinates and yaw, pitch, and roll of the aerial vehicle. The data captured by LiDAR may be converted from a native LiDAR sensor data format into LAS or LAZ format.

Next, the method establishes a current basis of scan data of utility infrastructure for comparison. The method begins with selecting a predefined flight corresponding to the location of utility infrastructure to be inspected. Next, an aerial vehicle is instructed to traverse the at least one predefined flight path that has been selected. The second aerial vehicle traverses the predefined path using Lidar remote sensing to create a second set of scan data in a 3D coordinate system of the utility infrastructure along with corresponding location coordinates and yaw, pitch, and roll of the aerial vehicle. Some or all of this data may be processed in flight. The data captured by LiDAR may be converted from a native LiDAR sensor data format into LAS or LAZ format.

Next, the first set of scan data is compared with the second set of scan data by using georeferencing to prioritize identifying utility infrastructure based on settable metrics (e.g., the highest customer count, the highest voltage, the highest volume (for gas or water or waste), critical theatres (military installations, health care facilities, first responders, schools).

Next, the utility infrastructure with a highest priority using vectorization and elevational data is identified. The utility infrastructure with the highest priority is that has been identified using vectorization and elevational data is compared with the first set of scan data using georeferencing. In one example the utility infrastructure with the highest priority that has been identified is compared again the first sect of scan data to identify missing utility structure poles and utility structure poles that exceed a settable tilt angle.

Based on the comparing, presenting on a screen, a discrepancy between the first set of scan data and the second set of scan data corresponding to the utility infrastructure with the highest priority that has been identified, which exceeds a settable threshold.

As required, detailed embodiments are disclosed herein; however, it is to be understood that the disclosed embodiments are merely examples and that the systems and methods described below can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the disclosed subject matter in virtually any appropriately detailed structure and function. Further, the terms and phrases used herein are not intended to be limiting, but rather, to provide an understandable description.

The term “3-D measurements” are measurements, typically non-contact measurements, taken of an object to create a 3-D point cloud of an object that is dimensionally accurate and a photorealistic model of the object, such as through photogrammetry.

The term “aerial vehicle” refers to both manned and unmanned aerial systems (UAS including fixed-winged aircraft and lighter-than-air aircraft e.g., airships, dirigibles, and rotary-wing aircraft.

The terms “a” or “an”, as used herein, are defined as one or more than one. The term plurality, as used herein, is defined as two or more than two.

The term “adapted to” describes the hardware, software, or a combination of hardware and software that is capable of, able to accommodate, to make, or that is suitable to carry out a given function.

The term “another”, as used herein, is defined as at least a second or more.

The term “class” or “classifier” or “label” is a class label applied to data input in a machine learning algorithm.

The term “configured to” describes hardware, software, or a combination of hardware and software that is adapted to, set up, arranged, built, composed, constructed, designed, or that has any combination of these characteristics to carry out a given function.

The term “coupled,” as used herein, is defined as “connected,” although not necessarily directly and not necessarily mechanically.

The term “inspection parameters” means any type of data to capture, including angles, field-of-view, resolution, and position at which to capture images.

The term “image editing software” means software for editing and manipulating images, such as Blender.org or Photoshop from Adobe.

The terms “including” and “having,” as used herein, are defined as comprising (i.e., open language).

The term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

The term “optimization” means selecting a flight path segment that best meets the requirement for that specific segment. For example, suppose there is a flight path segment from point A to point B and back to point A. The flight path from point A to point B may closely follow the components, including powerlines of a power grid to monitor the equipment within a certain distance during the inspection itself. This path from point A to point B may not be in a straight line. However, when the flight path segment from point B to point A is generated, there is no need to inspect the components, and the flight path back may be at a higher altitude and along more of a straight path to preserve the battery life of the UAV.

The term “photogrammetry” is a technique to extract three-dimensional measurements of an object for obtaining reliable information, such as three-dimensional measurements, through processing and interpreting a series of photographic images. Photogrammetry may be complemented by techniques like LiDAR, laser scanners (using time of flight, triangulation or interferometry), white-light digitizers and any other technique that scans an area and returns x, y, z coordinates for multiple discrete points, commonly called “point clouds”.

The term “real-world” means existing in reality, as opposed to one that is virtual, imaginary, simulated, or theoretical.

The term “simultaneous” means computations are carried out at the same time, which for larger data sets with various constraints is not possible to be carried out completed by a group of humans and must be performed by a computer. For example, one human could not compute one simulation with all the constraints for ten crews across fifty jobs. It is infeasible for a human to calculate one simulation loop with one constraint, let alone perform it in parallel to a sort of global optimum.

The term “synthetic” means creating a computer-generated composite scene including equipment and background in which each of the equipment and the background scene were previously captured independently of each other.

The term “uniform data format” means data in a given format, whether date format, time format, currency format, scientific format, text format, or fractional format, so that all values of data are presented in a single consistent format for a given category or criteria.

The term “unmanned aerial systems” (UAS) and “unmanned aerial vehicle” (UAV) refers to piloted, autonomous, and semi-autonomous aircraft.

It should be understood that the steps of the methods set forth herein are not necessarily required to be performed in the order described, and the order of the steps of such methods should be understood to be merely exemplary. Likewise, additional steps may be included in such methods, and certain steps may be omitted or combined in methods consistent with various embodiments of the present device.

The below-described systems and methods provide for safe and efficient aerial vehicle inspection of system components within areas of interest (AOIs) experiencing natural events such as severe wind, rain, fire, etc. utilizing large scale aerial vehicle aerial vehicles. Embodiments of the present invention further facilitate the aerial vehicle generation of work orders for rapid deployment of repair crews. In some embodiments, AOIs are geographical areas comprising utility system components. However, embodiments of the present invention are not limited to utility systems. Components of a utility system may wear down, become damaged, or become inoperable. Depending on the geographical location of these components; current weather conditions; types of damage or operational issues; and/or the like it may be difficult to detect, locate, and remedy the issues within an acceptable amount of time. This may result in increased downtime of the system component(s), which decreases customer satisfaction and increases costs incurred by the utility provider.

Conventional utility system inspection/monitoring mechanisms generally involve dispatching work crews to inspect and identify any worn down or damaged component(s), the extent of damage, the cause of damage, etc. These conventional mechanisms are problematic because they increase the downtime of the system component, increase outages experienced by the customer, increase expenses incurred by the utility provider, etc. For example, it takes time for a crew to reach a site to assess damage, identify inoperable components, and receive repair components. In addition, the work crew may need to operate in dangerous environmental conditions to identify and repair the problematic components. Even further, the environmental conditions (e.g., wind speed) may be such that work crews may be prevented by the conditions, various laws, company policies, and/or the like from traveling to and/or operating in the AOIs. Also, conventional work orders usually do not provide very detailed information or require users to access multiple menus/pages to drill down to information of interest. This can be problematic when viewing work orders on portable electronic devices such as mobile phones, tablets, etc.

Embodiments of the present invention allow for system components, such as utility systems components, to be aerial vehicle monitored and inspected for real-time or near real-time during environmental conditions that may prevent human personnel and/or convention aerial vehicles from operating therein. Therefore, embodiments of the present invention enable the detection and identification of problems experienced by the components during dangerous operating conditions that would normally prevent work crews and aerial vehicles from operating. In addition, the aerial vehicle system is able to process large amounts of data of different types captured by large-scale unmanned aerial vehicles, which allows for more efficient and accurate detection of damaged system components than conventional systems. Work orders may be aerial vehicle generated before (or shortly after) the environmental conditions have subsided, and the required parts, equipment, and work crews identified within the work order may be aerial vehicle provisioned. This provides an advantageous improvement in response time when compared to conventional systems. The above allows for system/component downtime, customer dissatisfaction, and utility expenses to be greatly decreased since work crews do not need to be dispatched to diagnose the problem. In addition, embodiments of the present invention generate an interactive map allowing work crew members to see important work orders, system components, and inspection data information on displays of, for example, mobile phones and tablets without having to parse through multiple windows, menus etc.

shows one example of an operating environmentfor remote aerial vehicle inspection of utility system components. In one embodiment, the operating environmentcomprises one or more geographical areas,,. At least one geographical areamay comprise one or more AOIs. The AOI may be a defined area(s) within the geographical areacomprising geographical features, components of a utility systemsituated at various locations within the AOI, and/or the like.

Examples of geographical features includes rivers, streams, hills, cliffs, mountains, trees, boulders, and/or the like. Examples of utility systems include power grid systems (e.g., fossil fuel-based, solar-based, wind-based, nuclear-based generation, transmission and/or distribution subsystems), telephone systems (landline and wireless), water systems, gas systems, and oil systems. Each of these different types of utility systems may have multiple types of subsystems. For example, an electric power delivery system generally comprises a generation subsystem, a transmission subsystem, and a distribution subsystem. Each of these subsystems performs one or more specific functions and comprises multiple components. For example, the distribution subsystem of an electric power system comprises substations where each substation performs various functions for a power grid, such as transforming voltage, connecting transmission lines, transferring power, and protecting the grid from short circuits and overload currents, and/or the like. Components of a substation include but are not limited to, incoming and outgoing power lines, transformers, disconnect switches, circuit breakers, arresters, etc. Other non-limiting examples of utility system components include utility poles, transmission lines, solar panels, cooling towers, pipelines, and/or the like.

The operating environmentmay further comprise one or more information processing systemsdisposed within one or more of the geographical areasto. As will be discussed in greater detail below, the information processing system(s)may manage the aerial vehicle inspection of utility system components, generation of work orders, and provisioning of resources. The information processing system(s)may be located within the same geographical area as the AOIbeing inspected or be located within a geographical area that is remote from the AOIbeing inspected.

The information processing system(s)may be communicatively coupled to other components of the operating environment(and components outside the environment) by one or more networks. The networkmay comprise wired and/or wireless networking mechanisms and may further comprise wireless communication networks, non-cellular networks such as Wireless Fidelity (WiFi) networks, public networks such as the Internet, private networks, and/or the like. The wireless communication networks support any wireless communication standard such as, but not limited to, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), General Packet Radio Service (GPRS), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiplexing (OFDM), or the like. The wireless communication networks may include one or more networks based on such standards. For example, in one embodiment, a wireless communication network may comprise one or more of a Long Term Evolution (LTE) network, LTE Advanced (LTE-A) network, an Evolution Data Only (EV-DO) network, a General Packet Radio Service (GPRS) network, a Universal Mobile Telecommunications System (UMTS) network, and the like.

further shows that the operating environmentmay comprise one or more monitoring/inspection devices,disposed at a location(s) within one or more of the geographical areasto. As will be discussed in greater detail below, the aerial vehicles,may be configured to remotely and aerial vehicle inspect utility system components. The aerial vehicles,may be associated with a base station(s),located within one or more of the geographical areasto. The base station(s),may comprise charging/fueling areas for the devices,; shelter for the devices,; and/or the like.

In some embodiments, the aerial vehicle monitoring/inspection devices,may be unmanned mobile aerial vehicles such as (but are not limited to) unmanned aerial vehicles (UAVs), drones, rovers, climbing robots, and/or the like having monitoring systems such as optical cameras, infrared sensors, LIDAR, RADAR, acoustic systems, and/or the like. The aerial vehicles,may be manually and/or aerial vehicle operated. At least one of the monitoring/inspection devices,is a large-scale aerial vehicle such as a large-scale UAV, rover, and/or the like.

A large-scale aerial vehicle may be an aerial vehicle that is capable of traversing through environmental conditions that are deemed unsafe for human personnel. Large-scale mobile aerial vehicles may have an increased size over conventional-scale mobile aerial vehicles; increased flying/roving ranges over conventional-scale aerial vehicles; increased payload capacities over conventional-scale mobile aerial vehicles; increased environmental capabilities over conventional-scale mobile aerial vehicles such that these devices may traverse in/through events having environmental conditions that conventional-scale mobile aerial vehicles are unable to traverse or are prohibited from traversing; and/or the like. In one or more embodiments, a large-scale aerial vehicle may be an aerial vehicle that exceeds one or more specifications provided in the U.S. Code of Federal Regulations for small unmanned aircraft.

In many instances, a geographical areamay experience natural (or man-made) events,such as heavy winds, rain, tornados, hurricanes, fires, earthquakes, flooding, and/or the like that make it unsafe and difficult for human personnel (e.g., work crews) and/or conventional sized unmanned mobile aerial vehicles to operate within. Therefore, as will be discussed in greater detail below, embodiments may utilize one or more aerial vehicles,such as a large-scale unmanned mobile aerial vehicle that are capable of operating in during events,and associated environmental conditions to inspect the AOIsand their utility system(s). The large-scale unmanned mobile aerial vehicle may be deployed from the geographic areaexperiencing the event,and/or may be deployed from a remote geographical area,that may or may not be experiencing the event (or another event).

shows a more detailed example of an AOIlocated within a geographical area. In the example shown in, the AOIincludes an electrical power “grid” that is used to provide electrical power to consumer premises. AOImay contain a multitude of individual or overlapping AOIs. The example shown indepicts a number power generation componentsfor the utility system. Illustrated are a combined cycle gas generator, a solar array farm, and a wind farmAOIs. In further examples, operational contexts are able to include one power generation component, multiple collocated power generation components, power generation components that are physically separated and supply a common electrical power transmission or distribution system, any one or more power generation components, or combinations of these. These power generation components are able to be of any suitable type or design.

In this example, electrical power generated by one or more power generation components is provided to a power transmission system. The illustrated example depicts a transmission connectionthat couples one or more sources within power generation componentsto the power transmission system. In an example, the transmission connectionand power transmission systemAOIs include suitable step-up transformers and long-distance transmission lines to convey the generated electrical power to remote power distribution networks, other electrical power consumers, or both.

The illustrated power transmission systemprovides electrical power to one or more distribution systems, including a substation, distribution lines, and premises. The substationAOI may include transformers, protection devices, and other components to provide electrical power to power distribution lines. The power distribution linesdeliver power produced by the generating componentsto customer premises, such as the illustrated home. In general, customer premises are coupled to the power distribution systemand can include any combination of residential, commercial, or industrial buildings.further shows one or more monitoring/inspection devicestobeing placed at and/or traversing one or more locations within the AOIs.

shows one non-limiting example of a large-scale aerial vehiclecorresponding to the aerial vehicles,of. In this example, the aerial vehiclecomprises one or more processors, a monitoring unit, mobility controls, one or more storage units, one or more power systems, one or more guidance systems, one or more wireless communication systems, and a monitoring system. The processor(s)may perform various computing functions for the aerial vehicle. The monitoring unitmay control automated mobility (e.g., flying, roving, climbing, etc.) operations of the device; receive data from the information processing systemsuch inspection path data and instructions indicating that the aerial vehicleis to initiate mobility operations; manages monitoring/inspection operations to be performed by the devicefor one or more system components of the AOI; and/or the like.

In one embodiment, the monitoring unitutilizes the monitoring systemand computer/machine learning mechanisms to aerial vehicle identify system components; determine a current operational state of the system components; determine any problems with and/or damage to the components; monitor current weather conditions; and/or the like. The monitoring unitmay also control automated mobility operations of the device. For example, if the deviceis a UAV the monitoring unit(and/or processor) may aerial vehicle control the various systems and mobility controls/components that enable the aerial vehicleto traverse an inspection path. The monitoring unitmay be part of the processor, is the processor, or is a separate processor. The monitoring unitis discussed in greater detail below.

The mobility controlscomprise various mechanisms and components such as propellers, tracks, motors, gyroscopes, accelerometers, and/or the like that enable the aerial vehicleto take flight, rove, climb, and/or the like. The mobility controlsare aerial vehicles managed and controlled by the monitoring unitand/or processor. The storage unit(s)includes random-access memory, cache, solid-state drives, hard drives, and/or the like. In one embodiment, the storage unit(s)may comprise inspection path data, inspection data, weather data, and/or the like. The inspection path data, in some embodiments, may be received by the monitoring unitfrom the information processing systemand/or is an aerial vehicle generated by the monitoring unit. The inspection path datamay include, for example, predefined and/or aerial vehicle-generated coordinates that form a path to be traversed by the aerial vehiclefor inspecting/monitoring one or more system components within an AOI. The inspection path datamay also include altitude data and speed data that indicate the altitude and speed at which the aerial vehicleis to traverse one or more portions of the inspection path. The inspection path datamay further include data indicating specific angles at which the aerial vehicleis to position itself relative to a given system component for capturing inspection data.