Multi-Receiver Distributed Pen Mapping

This application is a continuation and claims priority of International Application No. PCT/US2024/016914, filed Feb. 22, 2024, which claims the benefit of U.S. Provisional Application 63/486,540, filed on Feb. 23, 2023, the entire contents of which are incorporated herein by reference.

The present specification relates to modeling rigid or semi-rigid structures using sensor data.

People can use nets for many reasons in marine environments. For example, fish nets are used in open-ocean fish farms, to contain the fish and the equipment used to care for the fish.

Fish nets can be flexible or semi-rigid structures, and can move, flex, and change shape while being used. For example, during turbulent weather when underwater currents are strong, nets can move or change shape or position.

A net with sensors attached at known locations can monitor the relative positions of the sensors with respect to each other, so as to more accurately model the shape of the net, e.g., so that in-water equipment can be moved appropriately to avoid collisions with or damage to the net. For example, a net with embedded hydrophones at fixed intervals or at predefined stations can coordinate with an emitter to map the shape and movement of a net. The net can incorporate any number of sensors and emitters throughout the net such that a system could model the shape and movement of the net.

This system can utilize sensors throughout the net to map the shape and position of different portions of the net in order to inform operators and automated controllers for other pieces of equipment in order to avoid collision with the net. A generated model of the net can provide important information to the farmers and researchers involved with the fishery. For example, a farmer could view a model of the net and see whether tangles or twists exist within the net that may injure or harm the fish.

The ability to monitor the movement of a net through a system such as this can prevent the entanglement of equipment and other nets by alerting operators or controlling equipment associated with the net. For example, a camera that can move throughout the net to monitor fish can coordinate with the described monitoring system in order to move the camera out of the way of the net as the net moves in the ocean.

This can provide the benefit of being able to monitor the fish and equipment within and around net. The monitoring can lead to preventing the entanglement of the net with either the equipment or the net folding over itself and becoming tangled. These benefits can lead to a safer and more productive fishery.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Like reference numbers and designations in the various drawings indicate like elements.

As shown in, marine environments utilizing nets can be particularly difficult for managing equipment that is used to support these nets. The ability to map the position of different portions of a semi-rigid or non-rigid dynamic structure through a fluid is beneficial to solving many types of problems.

Generally speaking, the system of, can map a likely 3D position of a semi rigid or non-rigid structure in fluid or gas. Through sensors distributed at predefined and fixed locations throughout the non-rigid structure, the system can determine the likely shape of the non-rigid structure. The system can use a transmission to the sensors from a predefined location and calculate based on the time of arrival of the transmission at the sensors, the position of each sensor throughout the structure. With the predefined position of the sensors embedded in fixed and predefined locations within the structure the system can utilize algorithms to map and predict the material of the structure between the sensor locations in order to develop a 3D model of the structure. The rate at which the sensors are activated by the signal and the number of sensors within the structure can determine the accuracy or resolution of the model.

In more detail,is an example environmentthat can include a semi-rigid structure, specifically a fishery net, with a number of sensorsincorporated into fixed locations through the net. The netcan include one or more sensorsincorporated into the netat predefined locations. The environmentcan include an emitterthat can be used to send a signal to the sensors, a cameraused to monitor the fish or the net, anchor points to stabilize portions of the net, and fish.

The netcan be of various sizes and shapes (e.g., circular, rectangular, cylindrical, or more). The netcan incorporate the sensorsinto the netin several ways for example, the sensors can stitch into the net a fixed intervals to create rings through-throughout the length of the net. Other attachments include, but are not limited to crimped cabling, tied in, chained, woven, or linked. In other examples, the netcan have the sensorsin the eight corners of a cube shaped net. In still other examples, the sensorscan be distributed in a spiral down through the net. The netcan be made of various materials and have various configurations for attaching the sensors. For instance, the netcan have additional sensorsat key parts of the netsuch as the bottom, connection points, predefined areas in which the cameraoperates, where other equipment such as the feeder is located. The additional sensorsin the netcan aid the system in mapping areas of the netthat can require additional resolution.

The sensorsin the netcan exist in various numbers and types. The sensorscan be all of one type such as hydrophones or of various types such as cameras, salinity sensors, depth sensors, temperature sensors, pressure sensors, and more. The sensorscan attach to the netin various ways. The sensorscan all be connected through wiring that is intertwined with the net. The sensorscan communicate wirelessly, or through hydrophones. The sensorscan network together, such that the data transmitted from the furthest sensorfrom the emitter travels through the other sensors. The number and location of the sensorscan vary. For instance, a netwhich ends at a single point (as in), can have one sensorat the bottom of the net and rings of sensors from the bottom of the nettowards the top. The sensorsreceive signals from the emitter. More sensors can possibly increase the resolution of the mapping. Fewer sensors can require the system to interpret additional data and account for more variables, thereby increasing the time to create the model. An appropriate balance of sensors and the correctly locations of sensors can increase the speed and efficiency of the mapping.

The emittercan emit a signal to the sensorsin order to map the sensorlocations. The emittercan be in or outside of the net. The one or more emitterscan coordinate to determine the shape of the net. The emitterscan be located vertically separated such that one emitter corresponds to one ring of sensors. Two emitterscan be located near each other, but can emit different frequencies and thus help distinguish the received signal at the sensors. Emitterscan be located in several locations around the exterior of one or more nets. The emitterposition can depend on the shape of the net, the location of the sensors, and more. In some instances, the emittercan move throughout the netor be removed from the nettemporarily. The emittercan be removed from the net for service, or when the emitteris not needed by the system. The emittercan be a hydrophone, light, or more. The emittercan coexist as part of the cameraor be on the same mounting as the camera. The emittercan have two or more emitters incorporated at fixed and predefined distances from each other. The emitterand sensorpositions and design can facilitate the calculation of both range and bearing of the sensors from the emitter using constraints within the system or transducer design.

The cameracan be mounted into the netin such a way as to monitor the fish or the net. The cameracan move throughout the net and observe portions of the netfor damage, monitor the health of the fish, monitor the sensorsand emitterfor damage and more. The cameracan verify the 3D model of the net created by the system and can contribute visual data to aid in the 3D modelling of the net. The cameracan move vertically up and down the net. The cameracan be removed from the netin response to detecting a near collision or when the camerais not needed. The cameracan also monitor other equipment in the netsuch as the fish feeder.

Additional examples include the use of multiple sensor types to determine the shape of the net. For instance, depth sensors can provide the z-axis values of depth of each sensorin the net, while the system can use the transmission time of the signal between the emitterand sensorsto determine the x and y-axis components of the net.

In other examples unique shapes of emitters or sensors can aid in the system analysis of the netshape. For instance, one continuous string of hydrophones from the top of the netto the bottom can provide a very accurate representation of the edge of the net. The spacing of the hydrophones in the continuous sensorcould provide data for points every inch along the sensor. In this example, the system could minimize the use of computational power to statistically analyze the space between sensors points that could be spread throughout the net.

In some implementations the system can contain values representing the constraints of the semi-rigid structure. The constraints can aid in the modeling of the semi-rigid structure by providing limitations to the semi-rigid structure. When the constraints and predefined positions of the sensors are combined, the system can more readily approximate the model of the semi-rigid structure. For instance, the if two sensorshave the same arrival time for the emitted energy, and the system knows that the two sensorsare in fixed positions within the semi-rigid structure, then the probabilistic scenario is that the portion of the semi-rigid structure has folded over itself such that the sensorsoverlap. Several factors can contribute to the constraints including, but not limited to the shape of the semi-rigid structure, the number of sensors, the placement of sensors, the number of emitters, and the location of the one or more emitters.

In some implementations the system can use the time that the emitterproduced the emitted energy with the arrival time of the emitted energy at the sensorsto produce the model of the semi-rigid structure. For example, the system can use the emitted energy time and the arrival time of the emitted energy to know the distance and calculate the location of the sensors throughout the semi-rigid structure. The system can operate with the emitted energy originating from within our without the semi-rigid structure, and can use the time of emission to calculate and create the model of the semi-rigid structure.

In some implementations the system can use the arrival time of the emitted energy at the sensorscombined with physical constraints of the semi-rigid structure in order to calculate a best fit solution for the model. For example, the system can received arrival time data for the emitted energy for three sensors. The three sensorscan all exist in line with each other along a single side of the semi-rigid structure. Each arrival time limits the modelling of successive portions of the semi-rigid structure. In this example, the first sensors is bound by a physical constraint in that where it is attached to the semi-rigid structure is physically limited to be within a set distance of both the emitter and the other sensors. Therefore, as the system receives each arrival time of the emitted energy at the sensors, the solution is constrained by how the sensorsare physically located throughout the semi-rigid structure.

In some implementations the emittercan have a spherical array of transducers such that emissions can travel down particular bearings. Additionally, the use of transducers can allow for the position of emitterand sensorsto be reversed. For example, the spherical transducer array can receive emissions from emitters in the net.

In some implementations both the sensorsand the emitterare transducers and can act to either receive a signal or transmit a signal. For instance, with each sensorsthroughout the net having the ability to emit a signal, the system can roll call each sensorthroughout the net and receive all the individual transmissions at the emitter.

In some implementations the system can utilize calibration routines to determine the location of portions of the semi-rigid structure, sensors, and emitters. For example, a system can have a predefined number of sensorsin the semi-rigid structure at fixed locations such as a floating portions of the semi-rigid structure, anchor points, or more. These predefined sensor locations can coordinate with one or more emittersto calibrate the system periodically, on a schedule, or as needed. Calibration routines can perform manually by a user or automatically by the system.

is an example of the components used for modeling a semi-rigid dynamic structure in fluid. The sensorscan exist throughout the net in predefined and fixed locations. The emitterlocation is either predefined in relation to key points of the netor a second emittercan be utilized. The emitteremits a first signal that is received by the sensorsthroughout the net at different times. The system can account for salinity and temperature differences through the fluid. When all signals are received at the various sensors, the data is transmitted back to the system. The systemcan utilize various computer methods to compute the arrival times of the emittersignal at the sensorsto calculate the shape of the net.

The emittercan emit a first signal to the sensors. For example, an emitter, a hydrophone, located at the center of the rigid ring at the top of the netcan emit a single pulse or signal. The signal can travel and arrive at each sensorsat a different time. The emittercan emit two pulses, a pulse that is phase shifted, or a pattern of pulses to either increase reception resolution at the sensorsor distinguish the received pulse at the sensorsin order to identify one netfrom another net. The emittercan also operate at different frequencies depending on the environment and the need to de-conflict pulses with other nets.

The sensorscan receive the pulse or signal from the emitter. The sensorscan timestamp the received pulse and record data such as frequency, pressure, signal strength, temperature, salinity, and more. The sensorscan transmit the data back to the systemin several ways. The sensorscan network together to transmit data, or connected through wires, or each independently connected through a dedicated wire, or any combination thereof. The packaging of the data can vary based on systemneeds. For instance, the sensorscan parse the data such that only the time stamp from each sensorsis included in the data stream separated by commas. When received at the system, the systemcan determine the received order of timestamps to be the physical layout of the sensors, thus distinguishing one sensorfrom another. In other instances, the sensorscan transmit data that includes information that identifies the sensorsfrom other sensors such that the systemcan determine which sensors the data is from. The systemcan receive data from the sensors.

The systemreceives the signals and processes the data from the sensors. The systemcan employ several methods to interpret the data. For example, a systemcan employ artificial intelligence (AI), neural networks, modelling algorithms, graphing algorithms, or more in order to present a representation of the semi-rigid structure. In some examples, the output of systemcan be interpreted by other machines in order to move equipment in order to avoid collision. In other instances, the output is transmitted to users with reports on the netstatus.

The process can include a time synchronization between components. The components can require very precise timing and therefore may require a calibration or initialization of the clocks between components. For example, the emittercan emit energy that is received by the sensors. The system can measure the time-of-flight of the emitted energy. The emitterand sensorscan establish timestamps for both the emitted and received energies. Some implementations can include clock synchronization methods similar to Network Time Protocol (NTP). Various other methods can be used to synchronize clocks. The synchronization may occur at install, startup, periodically, on a schedule, or any combination thereof.

The process can occur on a schedule, on demand, in response to detecting an event within or around the net. In some instances, the sampling frequency increases base inclement weather in the area. In other instances, users can set the sampling frequency in response to the needs of the user.

is an example process of determining the shape of a semi rigid structure and displaying it to a user. The process involves the emission of energy from either the emitteror from another source. The sensorsthen receive the emitted energy and record and transmit the data to the system. The systemthen receives the data from the sensorsand processes it to develop a model representing the semi-rigid structure. The model of the semi rigid structure is then displayed to a user.

The process involves the emission of energy from either the emitteror from another source. In some instances, the emission of energy can occur from within the semi-rigid structure or from outside the semi-rigid structure. The emitted energy can have variable power, frequencies, repeated or non-repeated signals, or other waveform shaping characteristics. The system can utilize various strategies to allow multiple emissions to be distinguished from one another.

Sensorsthen receive the emitted energy and record and transmit the data to the system. The sensorscan have directional features that can allow for the sensorto determine the direction of arrival of the energy. In some examples, the sensorshave enough length to measure the curvature of the arriving energy wave. In these examples, the sensor can determine the arrival direction based on the difference in arrival times from one end of the sensor to the other. In other examples, the sensors can have an X shape that can allow for further wave form curvature analysis and directionality determination.

The systemthen receives the data from the sensorsand processes it to develop a model representing the semi-rigid structure. The systemcan utilize multiple methods to determine the representation of the semi-rigid structure. For example, using the predefined location of the sensors and the time of arrival, the system can use an existing and predefined model of the semi-rigid structure that includes accurate dimensions and shapes. With the existing model and the data from the sensors, the systemcan run a statistical algorithm to determine the shape of the semi-rigid structure. In other examples, the systemcan receive additional data from the sensors including depth, salinity, or more. The additional data provided to the systemcan aid in developing the model of the semi-rigid structure using the combined data. The systemcan use AI, Neural networks, remote processing through cloud computing, multiple processors, or more to aid in determining the model of the semi-rigid structure.

The model of the semi rigid structure is then displayed to a user. The model of the semi-rigid structure can be a 3D model, a plot of points, a mathematical representation, or more. The system can provide the model in a visual format for users to see and understand the shape of the semi-rigid structure. For example, the model can display in virtual or augmented reality, through mobile device screens, laptop or desktop computers, televisions, or any other manner in which to display a visual representation to a user. In other examples, the system can use the model to take action such as alert a user, move equipment out of the way of the net, or trigger other events to occur.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. The semi-rigid and rigid structures are used as an example and be any form of structure including rigid structures.

Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computers suitable for the execution of a computer program include, by way of example, general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a smart phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display), OLED (organic light emitting diode) or other monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.

Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., an Hypertext Markup Language (HTML) page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received from the user device at the server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

In each instance where an HTML file is mentioned, other file types or formats may be substituted. For instance, an HTML file may be replaced by an XML, JSON, plain text, or other types of files. Moreover, where a table or hash table is mentioned, other data structures (such as spreadsheets, relational databases, or structured files) may be used.

In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of transmitting, by an emitter, a signal; receiving, by each of multiple sensors located at different, predetermined positions on a semi-rigid structure, the signal; determining, by each of the multiple sensors, timing information for the signal; determining, using the timing information for each sensor, a current shape of the semi-rigid structure; and controlling an item of equipment based on the current shape of the semi-rigid structure.