Data Integration Plug-In for Data Analysis Platform

This application claims priority to and the benefit of Indian Application No. 202411047564, entitled “DATA INTEGRATION PLUG-IN FOR DATA ANALYSIS PLATFORM,” filed Jun. 20, 2024, which is hereby incorporated by reference in its entirety for all purposes.

The present disclosure generally relates to systems and methods for providing a data integration plug-in for transferring data between a data analysis platform and a data aggregation platform.

This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

Petrotechnical data is collected from various domains of upstream business, spanning drilling simulation, seismic, well placement, reservoir characterization, reservoir simulation, fracture modeling, geological modeling, gridding and upscaling, well and completion design to production design and optimization, and so on. Automated data flows may be used to ingest, process, publish, and draw insights from this data.

A summary of certain embodiments described herein is set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of these certain embodiments and that these aspects are not intended to limit the scope of this disclosure.

In certain embodiments a method includes providing, by a data integration plug-in for a data analysis platform, a user interface comprising user input fields. Additionally, the method includes receiving, via the data integration plug-in, respective user inputs corresponding to the user input fields. Furthermore, the method includes generating, via the data integration plug-in, a query in a database query language based on the user inputs. Moreover, the method includes receiving, via the data integration plug-in, input data from a data aggregation platform in response to the query. The method further includes importing, via the data integration plug-in, the input data to the data analysis platform.

In certain embodiments, a computing system includes a data aggregation platform configured to store one or more databases. The computing system further includes a data analysis platform having a data flow user interface (UI) configured to provide an environment for a user to configure a data flow. Additionally, the data analysis platform includes a data integration plug-in comprising a function that, when executed, is configured to cause the data integration plug-in to receive a user inputs indicative of query parameters. Additionally, the function, when executed, is configured to cause the data integration plug-in to transform the user inputs into a query interpretable by the data aggregation platform. Furthermore, the function, when executed, is configured to execute the query to retrieve an input dataset from the data aggregation platform.

In certain embodiments, a method includes providing, via a data analysis platform, a data flow user interface (UI) for configuring a data flow in a computing system. The method further includes ingesting, by the data analysis platform, input data from a data aggregation platform via a data integration plug-in of the data analysis platform. Additionally, the method includes integrating, via the data analysis platform, the input data into the data flow. Furthermore, the method includes generating, via the data analysis platform, output data based on the input data via the data flow. Additionally, the method includes writing, via the data analysis platform, the output data to the data aggregation platform using an instruction generated by the data integration plug-in.

One or more specific embodiments of the present disclosure will be described below. These described embodiments are examples of the presently disclosed techniques. Additionally, in an effort to provide a concise description of these embodiments, all features of an actual implementation may not be described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” and “the” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to “one embodiment” or “an embodiment” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

As used herein, the terms “connect,” “connection,” “connected,” “in connection with,” and “connecting” are used to mean “in direct connection with” or “in connection with via one or more elements”; and the term “set” is used to mean “one element” or “more than one element.” Further, the terms “couple,” “coupling,” “coupled,” “coupled together,” and “coupled with” are used to mean “directly coupled together” or “coupled together via one or more elements.” As used herein, the terms “up” and “down,” “uphole” and “downhole”, “upper” and “lower,” “top” and “bottom,” and other like terms indicating relative positions to a given point or element are utilized to more clearly describe some elements. Commonly, these terms relate to a reference point as the surface from which drilling operations are initiated as being the top (e.g., uphole or upper) point and the total depth along the drilling axis being the lowest (e.g., downhole or lower) point, whether the well (e.g., wellbore, borehole) is vertical, horizontal or slanted relative to the surface.

In addition, as used herein, the terms “real time”, “real-time”, or “substantially real time” may be used interchangeably and are intended to described operations (e.g., computing operations) that are performed without any human-perceivable interruption between operations. For example, as used herein, data relating to the systems described herein may be collected, transmitted, and/or used in control computations in “substantially real time” such that data readings, data transfers, and/or data processing steps occur once every second, once every 0.1 second, once every 0.01 second, or even more frequent, during operations of the systems (e.g., while the systems are operating). In addition, as used herein, the terms “automatic” and “automated” are intended to describe operations that are performed or caused to be performed, for example, by a computing system (i.e., solely by the computing system, without human intervention). In addition, as used herein, the term “approximately equal to” may be used to mean values that are relatively close to each other (e.g., within 5%, within 2%, within 1%, within 0.5%, or even closer, of each other).

Oil and gas operations generate data across many domains including exploration, drilling, production, refining, and distribution. This data may be used to monitor operational processes, generate business insights, improve safety, drive operational efficiency, and enhance decision-making. Using data analytics technology (e.g., machine learning), valuable insights can be extracted from operational data automatically and at scale. These insights may encompass a wide range of domains, including reservoir characterization, well optimization, asset maintenance, supply chain management, and market intelligence, among many others.

A computing system for generating insights from data may include a data aggregation platform (e.g., Cognite Data Fusion, and so forth) configured to store and manage access to data across an enterprise. For example, the data aggregation platform may include servers (e.g., cloud servers, in certain embodiments) configured to receive and store data from sensors, applications, and other myriad data sources. The data stored in the data aggregation platform may be accessed using a query language, such as GraphQL or SQL. In certain embodiments, the data aggregation platform may implement a data framework (e.g., Flexible Data Model as but one non-limiting example) for managing and manipulating diverse data types and structures. As such, queries and manipulations of data on the data aggregation platform may be performed in accordance with the data framework. Such operations may require a level of technical expertise (e.g., programming knowledge) barring would-be users from easily interfacing with the data aggregation platform.

The computing system may further include a data analysis platform (e.g., Dataiku as but one non-limiting example) that analyzes the data to produce useful insights. For example, the data analysis platform may include tools to clean the data, generate statistics, train and run machine learning models, and/or visualize the data. Additionally, the data analysis platform may include a digital environment with a user interface for creating data flows. As referred to herein, a “data flow” is a sequence of operations that are performed to record, ingest, process, manipulate, draw insights from, and/or act upon one or more sets of data. In some cases, a data flow may be at least partially automated such that outputs (e.g., insights, visualizations, actions, and so forth) may be produced automatically as data flows into the data analysis platform.

The present disclosure relates to a data integration plug-in for a data analysis platform that facilitates user-friendly transfer (e.g., data ingestion, processing, and writing) of data between the data analysis platform and a data aggregation platform. The data integration plug-in may be integrated within the data analysis platform to add data ingestion, processing, and writing functionality to the data analysis platform. Specifically, the data integration plug-in may receive user inputs indicative of a desired action to be performed (i.e., a data action). Then, the data integration plug-in may generate a script, a query, and/or a command to interface with the data aggregation platform in a suitable format (e.g., a query language). The data action may be incorporated into a data flow running on the data analysis platform. That is, the data integration plug-in may perform the data action as part of the data flow within the data analysis platform. The data action may include ingesting raw input data from the data aggregation platform into the data flow, pre-processing (e.g., filtering, pivoting, selecting) the input data, and writing output data to the data aggregation platform. By making it easier for a user to interact with the data aggregation platform via the data analysis platform, barriers to the development of data flows may be lowered, enabling greater accessibility to data-based outcomes to a wider range of users (e.g., domain experts, managers, business-facing users, and so forth).

With the foregoing in mind,illustrates a computing systemthat includes a data analysis platformand a data aggregation platform. In certain embodiments, the data analysis platformis a software application that provides a development environment for creating, running, and managing one or more data flows. In certain embodiments, the data analysis platformmay be hosted on a server (e.g., cloud server, in some embodiments) that communicates with other devices (e.g., servers, clients, and so forth) over a network. The data aggregation platformmay receive aggregated data from various data sources, such as sensors, drilling equipment, and the internet. The data received from the various data sourcesmay include time series objects containing data points in time order. Examples of a time series are the temperature of a drill bit, an oil tank level, a flow rate through a valve over time, and so forth. The data aggregation platformmay record the data in one or more databases. The databasesmay be stored in one or more storage devices of a server (e.g., cloud server) that communicates with other devices (e.g., other servers, clients, data analysis platform, in some embodiments). Additionally, the data aggregation platformmay include data modelsthat organize data elements and standardize how they relate to one another and the properties of real-world entities (e.g., subsurface formations, drilling equipment, industrial systems, and so forth).

As discussed above, a data flowmay be defined as a sequence of operations that ingest, manipulate, analyze, or otherwise engage with data. Some data flowsmay include operations to ingest data from an external source (e.g., the data aggregation platform) and produce output data of various kinds, such as visualizations, actions, processed datasets, and so forth. For example, a data flowmay include an operation to interface with the data aggregation platform, such as ingesting a portion of a dataset from a certain databaseor modelstored on the data aggregation platform.

Presently recognized is a need to efficiently provide data from the data aggregation platformto the data analysis platformto be used in the data flows. Thus, the computing systemincludes a data integration plug-inconfigured to establish a data pipelinebetween the data analysis platformand the data aggregation platform. The data integration plug-inmay be a software component that adds functionality onto a pre-existing data analysis platform, such as Dataiku. For example, the data integration plug-inmay include a function to import a pre-processed dataset from the data aggregation platforminto a data flowso that the data analysis platformcan analyze the pre-processed dataset. Further, the data integration plug-inmay include a function to export (e.g., write) an output dataset generated by the data analysis platformto the data aggregation platform(e.g., database(s)).

As such, the data integration plug-inmay be configured to convert data flowsand associated datasets, for example, from organization-specific data types and structures (e.g., of an organization with which a particular useris associated) to industry-specific data types and structures (e.g., that are standardized based on industry standards in the data aggregation platform). Other examples of such data conversion that may be performed by the data integration plug-inmay be to convert role-specific data types and structures (e.g., based on specific roles of a particular userwith respect to their associated organization) to the industry-specific data types and structures (e.g., that are standardized based on industry standards in the data aggregation platform). In this manner, the data integration plug-inmay facilitate a particular userto interact with the data aggregation platformdespite the fact that the particular usermay not be particularly familiar with the particular data types and structures stored by the data aggregation platform, for example, if the particular userlacks particular type of knowledge, such as engineering-specific data types and structures if the particular useris a management level person.

A usermay engage with the data analysis platformvia a user interface (UI). In certain embodiments, the data analysis platformmay be hosted on a server, and the usermay access the data analysis platformfrom a user device(e.g., PC, laptop, mobile device) via which the UImay be displayed to the user. The UIof the data analysis platformmay include workspaces, menus, and tools to facilitate creation of a data flow. For example, the usermay drag-and-drop graphical elements (e.g., icons, arrows, and so forth) into a workspace to create a diagram (e.g., directed acyclic graph, data flow diagram, and so forth) representing the data flow. The data analysis platformmay interpret the arrangement of the graphical elements into computing operations (e.g., a script) and then execute a computational workflow corresponding to the data flow. In certain embodiments, the usermay be associated with a profilecontaining data regarding the user's identity, roles, and permissions. For example, the profilemay indicate that the useris permitted to view, modify, and/or execute a particular data flow.

One embodiment of the present disclosure is where the data analysis platformmay not, on its own, provide a UIfor interfacing with the data aggregation platform. Therefore, the data integration plug-inmay be provided as an add-on to the data analysis platform. In particular, the data integration plug-inmay provide its own UI specifically to receive user inputs related to operations involving the data aggregation platform. The user inputs may be indicative of query parameters defining a query to be executed on the data aggregation platform.

illustrates some example components of the data integration plug-in. For example, the data integration plug-inmay include a first component(e.g., function, operation, command, module, script, program, instruction) configured to read input data from the data aggregation platform, wherein the input data is automatically pre-processed (e.g., filtered) to provide an input dataset having certain properties, such as a particular format, scope, selection, and/or type. For example, the data integration plug-inmay provide user input fields and selectable options to the userto collect user inputs indicative of parameters to be used for pre-processing the input data. Then, the data integration plug-inmay generate and execute instructions (e.g., queries) in a query language (e.g., GraphQL) understood by the data aggregation platform. In this way, the usermay import the input data from the data aggregation platforminto the data flowin a readily usable form without writing much, or any, code. Thus, the data integration plug-inmay abstract the technology underlying the transfer of information between the data analysis platformand the data aggregation platform.

The data integration plug-inmay further include a second componentconfigured to read input data from the data aggregation platform, wherein the input data is manually pre-processed to provide an input dataset having certain properties, such as a particular format, scope, selection, and/or type. For example, the data integration plug-inmay receive instructions (e.g., queries) from the userdirectly in the query language, without the abstraction provided by the first component. In this way, the usermay interact with the data aggregation platformin a more customized way, which may be suitable for more advanced users and/or sophisticated use cases.

The data integration plug-inmay further include a third componentconfigured to read input data from the data aggregation platform, wherein the input data is raw data read directly from the databasesand/or the models. That is, the raw data may not be pre-processed or manipulated prior to ingestion to the data analysis platform. In such cases, further manipulation of the raw data may be performed in the data analysis platformto make the raw data usable.

The data integration plug-inmay further include a fourth componentconfigured to write output data from the data analysis platformto the data aggregation platform. For example, in certain embodiments, the data flowmay train a machine learning model on a training dataset received from the data aggregation platformor another data source. Then, the data flowmay receive additional dataset(s) and predict an output dataset using the additional dataset(s) as an input to the machine learning model. Then, the data flowmay write the output dataset to the data aggregation platformto be viewed, shared, and/or used in other data flows.

As such, the data integration plug-inmay provide various components,,,that provide varying functionalities to users, for example, based on the specific characteristics of the users(e.g., identities, roles, and permissions) such that, for example, usersof varying technical ability can interact with data aggregation platformsin a same or similar manner. Furthermore, as described in greater detail herein, the data integration plug-inmay provide plug-in UIs (e.g., that may generally correlate to functionality provided by the various components,,,of the data integration plug-in) that might not otherwise be available to users, thereby extending the functionalities of certain data aggregation platforms.

illustrates an example plug-in UIcorresponding to the first componentof the data integration plug-in. That is, the plug-in UIis configured to present user input fields for parameters associated with reading automatically pre-processed data from the data aggregation platform. In certain embodiments, the plug-in UImay be presented on a display of a user devicein response to selection of the first componentof the data integration plug-inby the user. It will be appreciated that other example plug-in UIsmay be provided in response to selection of other components,,of the data integration plug-inby the user.

The plug-in UIillustrated inincludes a project fieldin which the usermay enter or select a project of the data aggregation platformto which a query will be directed. Further, the usermay enter or select a particular model from a model field, a version of the model from a version field, and a view (e.g., table) of the particular model from a view field. User inputs to these user input fields identify a location of exploration in the data aggregation platformfor some target data. Additionally, the usermay select certain properties (e.g., data features, columns, and so forth) to investigate from the identified data location.

Once the properties are selected via the plug-in UI, the usermay further specify the query by selecting attributes of a time series from an attribute field, such as a target value and a timestamp. Additionally, the usermay select a time range to explore from a time range field. Alternatively, the usermay select a latest value optionto retrieve the latest value from a time series. Furthermore, the usermay select an aggregate optionto find an aggregate value of the time series within the time range. In addition, other statistics optionsmay be selected to find statistics, such as a count, an average, a sum, a maximum value, or a minimum value of the time series. Further, the usermay select a pivot optionto pivot the target data by a selected column.

illustrates further elements of the plug-in UIcontaining filters for the target data. For example, the usermay filter the target data based on a data type, a string type of property, an integer range, a Boolean value, and a time range for date/time fields(e.g., created date of asset). The user inputs into these user input fields specify (e.g., characterize, indicate, and so forth) the target data to be explored and filters for relevant data within the target data. The plug-in UImay include a preview windowto show a preview of an input datasetthat would be retrieved by the data integration plug-inusing the user inputs. Based on the user inputs, the data integration plug-inmay generate a query or other instruction in the query language to capture the information provided by the userin the user input fields. Then, the query may be executed as part of the data flowto import the input datasetto the data analysis platform.

illustrates a data flow UIof the data analysis platform. The data flow UImay include various graphical elements arranged in a workspace. For example, the graphical elements may include taskscontaining operationsto be performed on datasets. The tasksmay be connected to one another via arrowsindicating the flow of data. The tasks, operations, and arrowsmay be dragged and dropped onto the workspaceby the user. In this way, the usermay develop and/or view the data flowin an intuitive way. The graphical elements may also include an icon representing the first componentof the data integration plug-inas an operationto ingest the input dataset. Additionally, the data flowmay include the fourth componentof the data integration plug-into write an output datasetto the data aggregation platform. As such, as discussed above, the data integration plug-inmay enable the data flow UIof the data analysis platformto facilitate the functionality provided by the various components,,,of the data integration plug-into be presented to usersof the data analysis platformwhere the functionality might otherwise not be available.

illustrates a methodof operation of the data integration plug-infor interfacing with a data aggregation platform. Although the following description of the methodis described in a particular order, it should be noted that the methodmay be performed in any suitable order.

At block, the data integration plug-inmay receive a user selection of a data action to be performed (e.g., via a data flow UIof the data analysis platform). The selected data action may correspond to a first component, a second component, a third component, or a fourth componentof the data integration plug-in. For example, the selected data action may be a request to read automatically pre-processed input data from the data aggregation platform(e.g., using the first componentof the data integration plug-in).

At block, the data integration plug-inmay populate a plug-in UIwith user input fields based on the selected data action. That is, the selected data action may determine what user input fields are shown. For example, if the selected data action is to read automatically pre-processed input data from the data aggregation platform, then the user input fields may include the project field, the model field, the version field, the view field, and/or the properties field, as described above with respect to. Additionally, the user input fields may include fields to select filters, such as the data type, the string type of property, the integer range, the Boolean value, and the time range for date/time fields, as described above with respect to. These filters may include or exclude certain categories of data from the query.

At block, the data integration plug-inmay receive user inputs to the user input fields described with reference to block. For example, at each user input field, the data integration plug-inmay receive a string input, numerical input, a selection from a list, or a toggle (e.g., Boolean) input. These user inputs specify a target data for exploration.

Performance of subsequent steps of the methodmay depend on the data action selected at block. For example, if the selected data action is to read automatically pre-processed input data from the data aggregation platform, then the methodmay proceed to block. At block, the data integration plug-inmay generate a query or instruction based on the user inputs received at block. The query or instruction may be generated in the form of a query language interpretable by the data aggregation platformor the databasesand modelstherein. For example, the data integration plug-inmay incorporate the user inputs into a pre-determined query template corresponding to the selected data action. The data integration plug-inmay execute the query to retrieve input data from the data aggregation platform. At block, the data integration plug-inmay receive the input data in response to the query. At block, the data integration plug-inmay import the input data to the data analysis platform. In particular, the input data may be imported to the data flowwhere various operations may be performed to manipulate the data and derive insights.

However, if the data action selected at blockis to write output data to the data aggregation platform, then the methodmay proceed from blockto block, where the plug-in UImay again provide user input fields. In this case, however, the user input fields may differ based on the different data action. For example, the user input fields may include a write location for the output data. Then, the methodmay proceed to block, where the data integration plug-inreceives user inputs to the user input fields. At block, the data integration plug-inmay write the output data from a data flow to the data aggregation platformbased on the user inputs. For example, the output data may include predictions of a machine learning model trained on input data retrieved from the data aggregation platformat a preceding point of the data flow.

illustrates a methodof operation of the data analysis platformfor interfacing with the data aggregation platformusing the data integration plug-in. Although the following description of the methodis described in a particular order, it should be noted that the methodmay be performed in any suitable order.

At block, the data analysis platformmay provide a data flow UIfor developing a data flow. In certain embodiments, the data flow UImay be provided by a server of the data analysis platformto a client device (e.g., user device) for display. The client device may include input devices, such as a keyboard, a mouse, and/or a touchscreen for the userto interact with the data flow UI.

At block, the data analysis platformmay ingest input data from the data aggregation platformusing the data integration plug-in. For example, the data analysis platformmay execute a data flowcontaining a call for the data integration plug-into perform the methoddescribed above. In this way, the input data may be imported to the from the data aggregation platformto the data analysis platform.

At block, the data analysis platformmay integrate the input data into the data flow. That is, the input data may be selectively operated upon in a user-defined sequence as defined by the arrangement of graphical elements in the data flow UI. As part of the data flow, the input data may be cleaned, processed, analyzed, visualized, or otherwise manipulated in a desired manner. In certain embodiments, the input data may be used to train a machine learning model. Alternatively, the input data may be used as an input to an existing machine learning model to predict output data. At block, the data analysis platformmay generate the output data via the data flow.

At block, the data analysis platformmay write the output data to the data aggregation platformusing the data integration plug-in. For example, the data flowmay include a call for the data integration plug-into perform blockof the methoddescribed with reference to. Writing the output data may include transforming user inputs to the plug-in UIinto instructions in the query language. In this way, a bi-directional data pipelinemay be established by the data integration plug-inbetween the data analysis platformand the data aggregation platform.

The specific embodiments described above have been illustrated by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function] . . . ” or “step for [perform]ing [a function] . . . ”, it is intended that such elements are to be interpreted under 35 U.S.C. 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. 112(f).