Mapping and segmentation of oral maxillofacial complex and extracting clinical data

The present implementations relate generally to biomedical imaging, and more particularly to transforming a dentition structure.

Biomedical imaging requires increasing complex biomedical data input and computational processing to achieve successful medical outcomes. Conventional systems may not effectively process biomedical imaging information with sufficient speed and at sufficient granularity to support individualized patient care.

It may be advantageous to process biomedical data associated with a patient and generate a reconstructed model indicating a two-dimensional projection of a model of a dentition mode from an predetermined viewing plane. Thus, a technological solution for transforming a dentition structure can be provided.

A technical solution can include a method of transforming a dentition structure, by generating a surface geometrically connecting one or more landmarks, the landmarks being associated with one or more physical features of a physical object, generating a model including the edges and the landmarks, identifying at least one boundary of the model based on one or more of the edges and the landmarks, selecting at least one region associated with at least one corresponding physical feature of the physical object, projecting the selected region onto a predetermined projection plane to form a projection, and transforming a shape of the physical features of the physical object based on the projection.

The method can include smoothing the model based on the surface and at least one target landmark among the landmarks connected to the edge.

The method can include generating one or more planes corresponding to one or more surfaces of the physical object.

The method can include planes that correspond to one or more of the landmarks.

The method can include generating one or more planes tangential to one or more surfaces of the physical object.

The method can include a boundary that includes a plurality of boundary edges among the edges.

The method can include a physical object that is a dentition structure and the physical features include one or more teeth and soft tissue adjacent to the teeth.

The method can include a predetermined projection plane corresponds to a frontal view of the physical object, and is substantially aligned with an anatomical frontal plane of the physical object.

The method can include selecting a first region corresponding to a first physical feature of the physical object, and selecting a second region corresponding to a second physical feature of the physical object.

The method can include a physical object including a maxillofacial structure, and the first physical feature includes an upper dental arch region.

The method can include a second physical feature includes a lower dental arch region.

The method can include projecting the selected region onto a predetermined projection plane to form a projection by projecting at least one error indication onto the predetermined projection plane.

A technical solution can include a system for transforming a dentition structure. The system can include a topography engine configured to generate a surface geometrically connecting one or more landmarks, the landmarks being associated with physical features of a physical object, and to generate a model including the surface and the landmarks, and a construction engine operatively coupled to the topography engine and configured to identify at least one boundary of the model based on the surface and the landmarks, to select at least one region associated with at least one corresponding physical feature of the physical object, to project the selected region onto a predetermined projection plane to form a projection and to transform a shape of the physical features of the physical object based on the projection.

The system can include an image processing engine configured to obtain one or more images of the physical object, and to generate a three-dimensional model of the physical structure including the physical features, and a landmark processing engine operatively coupled to the image processing engine and the topography engine, and configured to associate the landmarks with one or more corresponding geometric locations with the three-dimensional model, the geometric location corresponding to one or more of the physical features.

The system can include a topography engine configured to smooth the model based on the surface and at least one target landmark among the landmarks connected to the edge.

The system can include a topography engine configured to generate one or more planes corresponding to one or more surfaces of the physical object.

The system can include planes correspond to one or more of the landmarks.

The system can include generating one or more planes tangential to one or more surfaces of the physical object.

A technical solution can include a computer readable medium including one or more instructions stored thereon and executable by a processor to generate, by a processor, a surface geometrically connecting one or more landmarks, the landmarks being associated with one or more physical features of a physical object, generate, by the processor, a model including the edges and the landmarks, identifying at least one boundary of the model based on one or more of the edges and the landmarks, select, by the processor, at least one region associated with at least one corresponding physical feature of the physical object, project, by the processor, the selected region onto a predetermined projection plane to form a projection, and transform, by the processor, a shape of the physical features of the physical object based on the projection.

The computer readable medium can include instructions executable by the processor to project, by the processor, at least one error indication onto the predetermined projection plane.

The present implementations will now be described in detail with reference to the drawings, which are provided as illustrative examples of the implementations so as to enable those skilled in the art to practice the implementations and alternatives apparent to those skilled in the art. Notably, the figures and examples below are not meant to limit the scope of the present implementations to a single implementation, but other implementations are possible by way of interchange of some or all of the described or illustrated elements. Moreover, where certain elements of the present implementations can be partially or fully implemented using known components, only those portions of such known components that are necessary for an understanding of the present implementations will be described, and detailed descriptions of other portions of such known components will be omitted so as not to obscure the present implementations. Implementations described as being implemented in software should not be limited thereto, but can include implementations implemented in hardware, or combinations of software and hardware, and vice-versa, as will be apparent to those skilled in the art, unless otherwise specified herein. In the present specification, an implementation showing a singular component should not be considered limiting; rather, the present disclosure is intended to encompass other implementations including a plurality of the same component, and vice-versa, unless explicitly stated otherwise herein. Moreover, applicants do not intend for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such. Further, the present implementations encompass present and future known equivalents to the known components referred to herein by way of illustration.

illustrates a model corresponding to a physical structure, in accordance with present implementations. As illustrated by way of example in, an example object modelmay include an object model source. The object model sourcemay include one or more 2D images (e.g., grayscale, RGB, etc.) associated with a patient and combined into 3D model for that patient. The object model sourcemay be formed from multiple Digital Imagining and Communications in Medicine (DICOM) images, the object model sourcemay be presented as a 2D projection on predetermined plane through the 3D model associated with the patient. As on example, a predetermined plane can be a transverse plane, a coronal plane, or a sagittal plane dividing or bisecting the 3D model.

The object modelmay include a plurality of landmarks. The plurality of landmarks are positioned on the object model source. The plurality of landmarksinclude one or more coordinates based on the determination that features of the 3D model or 2D images associated with the 3D model correspond to a particular structure (e.g., maxillofacial structure, etc.) of the 2D model. A structural feature may include a shape, or the like detected in a 2D image or a 3D model. The landmarksmay include landmark identifiers indicating a particular feature of a 3D model associated with a patient. As one example, a landmark identifier can include a label or the like indicating that a corresponding landmark is located at or proximate to a particular feature or structure of the 3D model, such as a bone joint, curvature, protrusion, boundary, or the like.

Each of the plurality of landmarksinclude at least one vectorextending in a direction from the landmark. Each of the vectorson the plurality of landmarks may be randomly oriented. The vectorsmay be oriented to indicate a particular feature of a 3D model associated with a patient. The vectorsare used to define at least one plane which define a 3D model associated with the patient, as described herein. The plane defined intersects with the plurality of landmarks. The plane defined may be perpendicular to the plurality of vectors.

illustrates an object modelcorresponding to a physical structure and including a first object model, in accordance with present implementations. As illustrated by way of example in, the object modelmay include an object model source. The object model sourcecan be substantially similar to the object model source. The object modelmay include a plurality of landmarks. The plurality of landmarksare substantially similar to the plurality of landmarks. Each of the plurality of landmarksinclude a vector. The vectorcan be substantially similar to the vector. The vectorsof the plurality of landmarksextend so as to cover a greater surface area proportionate to the number of landmarksand vectors.

The object modelmay include a plurality of planes. The plurality of planesmay be a mesh to define the structure in 2D of the object model source, as described herein. The plurality of planesmay include a first plane, a second plane, and a third plane. The plurality of planesmay include more than three planes. The first planecan be a rectangular plane which intersects with a portion of the plurality of landmarks. The first planemay intersect with the portion of the plurality of landmarksand is perpendicular to the vectorsof the portion of the plurality of landmarksto approximate the left side of the object model sourcein 2D. The second planecan be a rectangular plane and intersects with the first plane. The second planeintersects with a portion of the plurality of landmarksand may be perpendicular to the vectorsof the portion of the plurality of landmarks to approximate a portion of the center of the object model sourcein 2D. The second planemay approximate the center of the object model source. The third planeintersects with the second planeand can be a rectangular plane. The third planeintersects with a portion of the plurality of landmarksand may be perpendicular to the vectorsof the portion of the plurality of landmarksto approximate a second portion of the center of the object model sourcein 2D. The third planemay approximate the left side of the object model sourcein 2D. The plurality of planes may include a fourth plane which intersects with the third plane and approximates the right side of the object model sourcein 2D. The first plane, the second plane, the third plane, and the fourth plane form a rough mesh by connecting the plurality of landmarksto approximate the object model source. The rough mesh may approximate a maxillofacial structure of a patient.

illustrates the object modelcorresponding to a physical structure and including a second object model, further to the model of. As illustrated by way of example in, the object modelcan be a refined model of object model. The object modelmay include an object model source. The object model sourcesubstantially similar to the object model source. The object modelmay include a plurality of landmarks. The plurality of landmarks substantially similar to the plurality of landmarks. Each of the plurality of landmarksinclude a vector. The vectorcan be substantially similar to the vector.

The object modelmay include a plurality of planes. The plurality of planesmay include a first plane, a second plane, a third plane, a fourth plane, and a fifth plane. The plurality of planes may include more than five planes. The first plane, the second plane, the third plane, the fourth plane, and the fifth planecan be smoothed to result in a mesh including curved surfaces to match curvature, shape, or the like, of physical features corresponding to the object model, as described herein. As illustrated by way of example in, the first planeofcan be refined to be the first plane. The first planemay extend through a portion of the plurality of landmarkspositioned on the left side of the object model source. The first planemay be perpendicular to the vectorsof the portion of the plurality of landmarks. The first planemay refine the approximation of the left side of the object model sourceofin 2D. The first planeofmay be altered (e.g., smoothed, etc.) from a rectangular plane to curvilinear plane as shown insuch that first planehas a shape, curvature, or the like to match the physical features of the object model source. The number of planes depicted by way of example herein can be greater than or less than the particular number of planes depicted.

A first portion of the second planeofcan be refined to form the second plane. The second planemay extend through a portion of the plurality of landmarksto approximate the center and a portion of the left side of the object model sourcein 2D. The second planemay be perpendicular to the vectorsof the portion of the plurality of landmarks. The first portion of the second planemay be altered from a rectangular plane to a curvilinear plane as shown by the second planesuch that second planehas a shape, curvature, or the like to match the physical features of the object model source.

A second portion of the second planeofcan be refined to be the third plane. The third planemay extend through a portion of the plurality of landmarksat the center of the object model source. The third planemay be perpendicular to the vectorsof portion of the plurality of landmarksat the center of the object model source. The second portion of the second planemay be altered from a rectangular plane to form a curvilinear plane as shown by the third planesuch that third planehas a shape, curvature, or the like to match the physical features of the object model source. The third planemay approximate a portion of the center of the object model sourcein 2D.

A third portion of the second planeofand a first portion of the third planecan be refined to be the fourth plane. The fourth planemay extend through a portion of the plurality of landmarksat the center of the object model sourceand a portion of the plurality of landmarks at the right side of the object model source. The fourth planemay be perpendicular to the vectorsof portion of the plurality of landmarksat the center and at the right side of the object model source. The second portion of the second planeand the first portion of the third planemay be altered from a rectangular plane to form a curvilinear plane as shown by the fourth planesuch that fourth planehas a shape, curvature, or the like to match the physical features of the object model source. The fourth planeapproximates a portion of the center and a portion of the right side of the object model sourcein 2D. The second plane, the third plane, and the fourth planemay approximate the center of the object model sourcein 2D.

A second portion of the third planeand the fourth plane (not shown) ofcan be refined to be the fifth plane. The fifth planemay extend through a portion of the plurality of landmarksat the right side of the object model source. The fifth planemay be perpendicular to the vectorsof portion of the plurality of landmarksat the right side of the object model source. The second portion of the third planeand the fourth plane (not shown in) may be altered from a rectangular plane to form a curvilinear plane such that fifth planehas a shape, curvature, or the like to match the physical features of the object model source. The fifth planemay approximate a portion of the right side of the object model sourcein 2D. The fifth planemay approximate the entire right side of the object model sourcein 2D.

The first plane, the second plane, the third plane, the fourth plane, and the fifth planemay intersect with one another to form a single plane on which the entire maxillofacial structure of the physical structure of the object model could be projected onto the plane, as described herein.

illustrates a first projection of a physical structure in accordance with present implementations. As illustrated by way of example in, an example view of an example maxillofacial modelcan be generated by projecting the object model sourceonto the plurality of planesof. The maxillofacial modelmay include a first region, a second region, and a third region. The maxillofacial modelmay include more than three regions. The first regionmay illustrate a first portion (e.g., left, right, center, etc.) of the maxillofacial structure. By way of example, the first regionmay illustrate the left portion of the object model sourcewhich can be projected on to the first planeofto depict the left portion of the maxillofacial model. The second regionmay illustrate a second portion (e.g., left, right, center, etc.) of the maxillofacial model. By way of example the second regionmay illustrate the projection of the center portion of the object model sourceon to the second planeand the third planeofto depict the center portion of the maxillofacial model. The third regionmay illustrate a third portion (e.g., left, right, center, etc.) of the maxillofacial model. By way of example, the third regionmay illustrate the projection of the right portion of the object model sourceonto the fourth plane (not shown in) to depict the right portion of the maxillofacial model. The maxillofacial modelgenerated by the plurality of planesofmay be distorted so as to not accurately represent the maxillofacial structure of a patient. As illustrated by way of example in, the maxillofacial modelmay have a distortion regionwithin the first region. The distortion regionmay depict the distortion to the teeth and jaw line of a portion of the maxillofacial model. The distortion regionmay depict the distortion to the teeth and the jaw of the entire maxillofacial model. The distortion to the maxillofacial model can be generated by projecting the object model source (e.g., object model source, object model source, object model source, etc.) on a plurality of planes which are rectangular and do not match the shape, curvature, or the like of the object model source.

illustrates a second projection of a physical structure further to the first projection of. As illustrated by way of example in, an example view of an example maxillofacial modelcan be generated by projecting the object model sourceonto the plurality of planesof. The maxillofacial modelmay include a first region, a second region, and a third region. The first regionillustrates a first portion (e.g., left, right, center, etc.) of the maxillofacial model. By way of example, the first regiondepicts a projection of the left portion of object model sourceonto the first planeofto depict the left portion of the maxillofacial model. The second regionillustrates a second portion (e.g., left, right, center, etc.) of the maxillofacial model. By way of example, the second region illustrates the center portion of the object model sourceprojected onto the second plane, third plane, and a portion of the fourth planeofto depict the center portion of the maxillofacial model. The third region illustrates a third portion (e.g., left, right, center, etc.) of the maxillofacial model. By way of example, the third region illustrates the left portion of the object model sourceofprojected onto the portion of the fourth planeand the fifth plane (not shown in). The maxillofacial modelgenerated by the projection of the object model sourceonto the plurality of planesdoes not include distortion. As illustrated by way of example in, the maxillofacial model has a non-distorted region. The non-distorted regiondepicts the teeth and jaw of the maxillofacial modelhaving little to no distortion as compared to teeth and jaw in the distortion regionas shown in. The distortion may be drastically reduced due to the projecting the object model sourceon to the plurality of planeswhich are curvilinear to match the shape, curvature, or the like of the object model source.

illustrates a user interface including a second projection further to the second projection of. As illustrated by way of example in, an example view of an example maxillofacial modeland an error system. The maxillofacial modelcan be substantially similar to the maxillofacial modelas illustrated in. The error systemcan be configured to depict the errors of the maxillofacial model. The error systemmay depict the errors in the location of portions of the patient's dental structure or maxillofacial structure (e.g., teeth, jaw, etc.) between the maxillofacial modeland the location of the portions in the patient's dental structure or maxillofacial structure. The error systemmay include edge portions. The edge portionsdepict the empty space where the maxillofacial modelmay not be mapped. The edge portionassist in locating the maxillofacial modelsuch that a mesh gridformed by the error systemcan be overlaid on the maxillofacial model. The user interface may be presented on a display. The display may be associated with a computer, a smart phone, a tablet, a television, a monitor or any other device capable of displaying an image. In some implementations, a user may interact with the user interface via touch on the display.

The mesh gridmay include a plurality of true lines. The plurality of true linesform a grid pattern over the maxillofacial modeland depict the location of the portion of the patient's dental structure or maxillofacial structure generated by the maxillofacial model. The mesh gridmay include a plurality of error lines. The plurality of error linesform a grid pattern over the maxillofacial modeland the plurality of true linesand depict the actual location of the portion of the patient's dental structure or maxillofacial structure. The distance, d, between at least one of the error linesand the at least one true linedefines the error between the maxillofacial modeland the actual patient's dental structure or maxillofacial structure. In some embodiments, the error linesoverlap with the true linessuch that no error exists between the maxillofacial modeland actual patient's dental structure or maxillofacial structure.

illustrates an example processing system, in accordance with present implementations. As illustrated by way of example in, an example processing systemmay include a system processor, a parallel processor, a transform processor, a system memory, and a communication interface. In some implementations, at least one of the example processing systemand the system processormay include a processor busand a system bus.

The system processormay be operable to execute one or more instructions. The instructions may be associated with at least one of the system memoryand the communication interface. The system processormay be an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The system processormay include but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), or the like. The system processormay include a memory operable to store or storing one or more instructions for operating components of the system processorand operating components operably coupled to the system processor. The one or more instructions may include at least one of firmware, software, hardware, operating systems, embedded operating systems, and the like.

The processor buscan be operable to communicate one or more instructions, signals, conditions, states, or the like between one or more of the system processor, the parallel processor, and the transform processor. The processor busmay include one or more digital, analog, or like communication channels, lines, traces, or the like. It can be to be understood that any electrical, electronic, or like devices, or components associated with the system buscan also be associated with, integrated with, integrable with, supplemented by, complemented by, or the like, the system processoror any component thereof.

The system busmay be operable to communicate one or more instructions, signals, conditions, states, or the like between one or more of the system processor, the system memory, and the communication interface. The system busmay include one or more digital, analog, or like communication channels, lines, traces, or the like. It can be to be understood that any electrical, electronic, or like devices, or components associated with the system buscan also be associated with, integrated with, integrable with, supplemented by, complemented by, or the like, the system processoror any component thereof.

The parallel processormay be operable to execute one or more instructions concurrently, simultaneously, or the like. The parallel processormay be operable to execute one or more instructions in a parallelized order in accordance with one or more parallelized instruction parameters. In some implementations, parallelized instruction parameters include one or more sets, groups, ranges, types, or the like, associated with various instructions. The parallel processormay include one or more execution cores variously associated with various instructions. The parallel processormay include one or more execution cores variously associated with various instruction types or the like. The parallel processormay be an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, communication buses, volatile memory, nonvolatile memory, and the like. The parallel processormay include but is not limited to, at least one graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), gate array, programmable gate array (PGA), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or the like. It can be to be understood that any electrical, electronic, or like devices, or components associated with the parallel processorcan also be associated with, integrated with, integrable with, supplemented by, complemented by, or the like, the system processoror any component thereof.

In some implementations, various cores of the parallel processorare associated with one or more parallelizable operations in accordance with one or more metrics, engines, models, and the like, of the example computing system of. As one example, parallelizable operations include processing portions of an image, video, waveform, audio waveform, processor thread, one or more layers of a learning model, one or more metrics of a learning model, one or more models of a learning system, and the like. In some implementations, a predetermined number or predetermined set of one or more particular cores of the parallel processorare associated exclusively with one or more distinct sets of corresponding metrics, engines, models, and the like, of the example computing system of. As one example, a first core of the parallel processorcan be assigned to, associated with, configured to, fabricated to, or the like, execute one engine of the example computing system of. In this example, a second core of the parallel processorcan also be assigned to, associated with, configured to, fabricated to, or the like, execute another engine of the example computing system of. Thus, the parallel processormay be configured to parallelize execution across one or more metrics, engines, models, and the like, of the example computing system of. Similarly, in some implementations, a predetermined number or predetermined set of one or more particular cores of the parallel processorare associated collectively with corresponding metrics, engines, models, and the like, of the example computing system of. As one example, a first plurality of cores of the parallel processor can be assigned to, associated with, configured to, fabricated to, or the like, execute one engine of the example computing system of. In this example, a second plurality of cores of the parallel processor can also be assigned to, associated with, configured to, fabricated to, or the like, execute another engine of the example computing system of. Thus, the parallel processormay be configured to parallelize execution within one or more metrics, engines, models, and the like, of the example computing system of.

The transform processormay be operable to execute one or more instructions associated with one or more predetermined transformation processes. As one example, transformation processes include Fourier transforms, matrix operations, calculus operations, combinatoric operations, trigonometric operations, geometric operations, encoding operations, decoding operations, compression operations, decompression operations, image processing operations, audio processing operations, and the like. The transform processormay be operable to execute one or more transformation processes in accordance with one or more transformation instruction parameters. In some implementations, transformation instruction parameters include one or more instructions associating the transform processorwith one or more predetermined transformation processes. The transform processormay include one or more transformation processes. Alternatively, in some implementations, the transform processormay be a plurality of transform processorassociated with various predetermined transformation processes. Alternatively, the transform processormay include a plurality of transformation processing cores each associated with, configured to execute, fabricated to execute, or the like, a predetermined transformation process. The parallel processormay be an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, communication buses, volatile memory, nonvolatile memory, and the like. The parallel processormay include but is not limited to, at least one graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), gate array, programmable gate array (PGA), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or the like. It can be to be understood that any electrical, electronic, or like devices, or components associated with the transform processorcan also be associated with, integrated with, integrable with, supplemented by, complemented by, or the like, the system processoror any component thereof.

The transform processormay be associated with one or more predetermined transform processes in accordance with one or more metrics, engines, models, and the like, of the example computing system of. In some implementations, a predetermined transform process of the transform processoris associated with one or more corresponding metrics, engines, models, and the like, of the example computing system of. As one example, the transform processorcan be assigned to, associated with, configured to, fabricated to, or the like, execute one matrix operation associated with one or more engines, metrics, models, or the like, of the example computing system of. As another example, the transform processorcan alternatively be assigned to, associated with, configured to, fabricated to, or the like, execute another matrix operation associated with one or more engines, metrics, models, or the like, of the example computing system of. Thus, the transform processormay be configured to centralize, optimize, coordinate, or the like, execution of a transform process across one or more metrics, engines, models, and the like, of the example computing system of. The transform processor may be fabricated to, configured to, or the like, execute a particular transform process with at least one of a minimum physical logic footprint, logic complexity, heat expenditure, heat generation, power consumption, and the like, with respect to at least one metrics, engines, models, and the like, of the example computing system of.

The system memorymay be operable to store data associated with the example processing system. The system memorymay include ones or more hardware memory devices for storing binary data, digital data, or the like. The system memorymay include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units, or the like. The system memorymay include at least one of a nonvolatile memory device, a solid-state memory device, a flash memory device, and a NAND memory device. The system memorymay include one or more addressable memory regions disposed on one or more physical memory arrays. In some implementations, a physical memory array may include a NAND gate array disposed on a particular semiconductor device, integrated circuit device, printed circuit board device, and the like.

The communication interfacemay be operable to communicatively couple the system processorto an external device. In some implementations, an external device may include but is not limited to a smartphone, mobile device, wearable mobile device, tablet computer, desktop computer, laptop computer, cloud server, local server, and the like. The communication interfacemay be operable to communicate one or more instructions, signals, conditions, states, or the like between one or more of the system processorand the external device. The communication interfacemay include one or more digital, analog, or like communication channels, lines, traces, or the like. As one example, the communication interfaceis or may include at least one serial or parallel communication line among multiple communication lines of a communication interface. The communication interfacemay be or may include one or more wireless communication devices, systems, protocols, interfaces, or the like. The communication interfacemay include one or more logical or electronic devices including but not limited to integrated circuits, logic gates, flip-flops, gate arrays, programmable gate arrays, and the like. The communication interfacemay include ones or more telecommunication devices including but not limited to antennas, transceivers, packetizers, wired interface ports, and the like. It can be understood that any electrical, electronic, or like devices, or components associated with the communication interfacecan also be associated with, integrated with, integrable with, replaced by, supplemented by, complemented by, or the like, the system processoror any component thereof.