Systems and methods are described for graphical user interfaces that enable users to graphically design nodes and interconnections of pipelines for preprocessing data later ingested into an indexing system. The preprocessing can include receiving messages published to a first publish-subscribe messaging system, the messages containing raw machine data generated by one or more components in an information technology environment, performing one or more processing operations on at least some of the messages to generate preprocessed messages, republishing the preprocessed messages to a second publish-subscribe messaging system, and providing to the indexing system, a subset of the messages from the second publish-subscribe messaging system. Nodes in the pipeline can specify the preprocessing operations, and interconnections in the pipeline can represent data flow through the nodes of the pipeline.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A method comprising: providing a graphical interface to receive user input to program operation of an iterative publish-subscribe-based message processing system by specifying a series of modules and interconnections between individual modules within the series, wherein the individual modules designate a transformation of messages on the iterative publish-subscribe-based message processing system and wherein the interconnections designate a routing of messages on the iterative publish-subscribe-based message processing system; receiving specification of the series of modules and the interconnections; and configuring the iterative publish-subscribe-based message processing system according to the series of modules and the interconnections, wherein configuring the system according to an individual module causes the system to retrieve a message from a source publish-subscribe messaging system designated by a source interconnection for the individual module, process the message according to the transformation designated by the individual module to result in a transformed message, and publish the message to a destination publish-subscribe messaging system designated by a destination interconnection for the individual module.
2. The method of claim 1 , further comprising implementing the iterative publish-subscribe-based message processing system at an intake system to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
3. The method of claim 1 , further comprising configuring an abstract syntax tree (AST) representation of the iterative publish-subscribe-based message processing system and implementing the AST representation to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
4. The method of claim 1 , further comprising configuring the graphical interface with a visual representation of the series of modules and the interconnections.
5. The method of claim 1 , further comprising configuring the graphical interface with selectable features that enable a user to add modules to the iterative publish-subscribe-based message processing system.
6. The method of claim 1 , further comprising configuring the graphical interface with selectable features that enable a user to add interconnections between modules.
7. The method of claim 1 , further comprising receiving specification of at least one additional source publish-subscribe messaging system, wherein the series of modules and the interconnections combines messages from the source publish-subscribe messaging system and the at least one additional source publish-subscribe messaging system for writing to the destination publish-subscribe messaging system.
8. The method of claim 1 , further comprising receiving specification of at least one additional destination publish-subscribe messaging system, wherein the series of modules and the interconnections provides messages from the source publish-subscribe messaging system to the destination publish-subscribe messaging system and the at least one additional destination publish-subscribe messaging system.
9. The method of claim 1 , wherein the message is a live data stream from the source publish-subscribe messaging system.
10. The method of claim 1 , further comprising publishing the iterative publish-subscribe-based message processing system, wherein the publishing causes iterative publish-subscribe-based message processing system to perform stream processing on messages between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
11. The method of claim 1 , wherein an interconnection of the graphical interface includes a selectable option to add a module before or after an existing module in the series.
12. The method of claim 1 , wherein an interconnection of the graphical interface includes a selectable option to add a branch after an existing module in the series.
13. The method of claim 1 , wherein the graphical interface includes at least one selectable option to add a module or a branch connecting to an existing module in the series, and wherein the graphical interface is updated to display a listing of available modules responsive to user selection of the at least one selectable option.
14. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system responsive to addition of additional module by displaying the additional module in-line with another module interconnected to the additional module.
15. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system responsive to addition of additional module by displaying the additional module in-line with other modules along a same branch as the additional module and by evenly spacing the additional module and the other modules along the same branch.
16. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system responsive to addition of additional module by automatically resizing the graphical representation.
17. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system by displaying, on each module, a graphical representation of an amount of data processed by, received by, or transmitted by that module.
18. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system by displaying, on each module, a graphical representation of a number of events processed by, received by, or transmitted by that module.
19. The method of claim 1 , wherein the graphical interface is configured to update display of a graphical representation of the iterative publish-subscribe-based message processing system by displaying, on each module, a graphical representation of a latency introduced by that module into the iterative publish-subscribe-based message processing system.
20. The method of claim 1 , wherein the source publish-subscribe messaging system receives messages that are raw machine data generated in an information technology environment.
21. The method of claim 1 , wherein the destination publish-subscribe messaging system is a data intake and query system.
22. The method of claim 1 , wherein the graphical interface integrates with a back-end system by accessing a function registry of the back-end system to define executable instructions generated by configuring the iterative publish-subscribe-based message processing system according to its individual modules.
23. The method of claim 1 , wherein the graphical interface communicates with a back-end system by providing the back-end system with an abstract syntax tree that represents the iterative publish-subscribe-based message processing system.
24. The method of claim 1 , wherein said providing a graphical interface comprises providing a graphical interface to receive user input to program operation of an iterative publish-sub scribe-based message processing system by specifying a series of modules and interconnections between a first individual module representing a data source, and a second individual module representing a data intake and query system.
25. The method of claim 1 , wherein said providing a graphical interface comprises providing a graphical interface to receive user input to program operation of an iterative publish-subscribe-based message processing system by specifying a series of modules and interconnections between a first individual module of the series of modules representing a data source, and a second individual module of the series of modules representing a data intake and query system, and one or more individual modules interposed between the first individual module and the second individual module.
26. The method of claim 1 , wherein said configuring the iterative publish-subscribe-based message processing system according to the series of modules and the interconnections, wherein configuring the system according to an individual module causes the system to further: publish the message to a destination publish-subscribe messaging system designated by a destination interconnection for the individual module, wherein the destination publish-subscribe messaging system is subscribed to by a data intake and query system represented in the graphical interface by a module of the series of modules.
27. The method of claim 1 , further comprising activating a preview mode that adds a function to the iterative publish-subscribe-based message processing system that prevents publishing the message to the destination publish-subscribe messaging system, wherein the preview mode causes the iterative publish-subscribe-based message processing system with the function added to retrieve a message from the source publish-subscribe messaging system, process the message according to the transformation designated by the individual modules to result in a transformed message, and cause output of an update to the graphical interface that depicts data output by at least one individual module.
28. The method of claim 1 , further comprising: activating a preview mode that adds a function to the iterative publish-subscribe-based message processing system, wherein the function captures data output by an individual module when the iterative publish-subscribe-based message processing system retrieves a message from the source publish-subscribe messaging system and processes the message according to the transformation designated by the individual modules to result in a transformed message; and causing output of an update to the graphical interface that depicts the data captured by the function.
29. The method of claim 1 , further comprising activating a preview mode that causes output of a user interface that displays data flowing through the iterative publish-subscribe-based message processing system from the source publish-subscribe messaging system, wherein the preview mode operates independently of configuring the publish-subscribe-based message processing system according to the series of modules and the interconnections for streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
30. The method of claim 1 , further comprising activating a preview mode that causes output of a user interface that displays data flowing through the iterative publish-subscribe-based message processing system from the source publish-subscribe messaging system, wherein individual modules in the series are selectable in the graphical interface, further comprising displaying the preview of an individual module responsive to user selection of the individual module.
31. The method of claim 1 , further comprising causing output of information regarding options for modules to add to the series, wherein the options for the modules are identified based on a compatibility with existing specified modules in the series.
32. The method of claim 1 , further comprising causing output of information regarding options for modules to add to the series, wherein the options for the modules are identified based on historical data that flowed from the source publish-subscribe messaging system.
33. The method of claim 1 , further comprising causing output of information regarding options for modules to add to the series, wherein the options for the modules are identified based on historical data that flowed into the destination publish-subscribe messaging system.
34. The method of claim 1 , further comprising: receiving specification of the source publish-subscribe messaging system; causing output of a plurality of module series templates that are compatible with the source publish-subscribe messaging system; and configuring the series based on a selected one of the plurality of module series templates.
35. A system comprising: a data store including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to: provide a graphical interface to receive user input to program operation of an iterative publish-subscribe-based message processing system by specifying a series of modules and interconnections between individual modules within the series, wherein the individual modules designate a transformation of messages on the iterative publish-subscribe-based message processing system and wherein the interconnections designate a routing of messages on the iterative publish-subscribe-based message processing system; receive specification of the series of modules and the interconnections; and configure the iterative publish-subscribe-based message processing system according to the series of modules and the interconnections, wherein configuring the system according to an individual module causes the system to retrieve a message from a source publish-subscribe messaging system designated by a source interconnection for the individual module, process the message according to the transformation designated by the individual module to result in a transformed message, and publish the message to a destination publish-subscribe messaging system designated by a destination interconnection for the individual module.
36. The system of claim 35 , wherein execution of the computer-executable instructions causes the system to implement the iterative publish-subscribe-based message processing system at an intake system to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
37. The system of claim 35 , wherein execution of the computer-executable instructions causes the system to configure an abstract syntax tree (AST) representation of the iterative publish-subscribe-based message processing system and implementing the AST representation to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
38. Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform one or more processing operations comprising: providing a graphical interface to receive user input to program operation of an iterative publish-subscribe-based message processing system by specifying a series of modules and interconnections between individual modules within the series, wherein the individual modules designate a transformation of messages on the iterative publish-subscribe-based message processing system and wherein the interconnections designate a routing of messages on the iterative publish-subscribe-based message processing system; receiving specification of the series of modules and the interconnections; and configuring the iterative publish-subscribe-based message processing system according to the series of modules and the interconnections, wherein configuring the system according to an individual module causes the system to retrieve a message from a source publish-subscribe messaging system designated by a source interconnection for the individual module, process the message according to the transformation designated by the individual module to result in a transformed message, and publish the message to a destination publish-subscribe messaging system designated by a destination interconnection for the individual module.
39. The non-transitory computer-readable media of claim 38 , wherein the one or more processing operations comprise implementing the iterative publish-subscribe-based message processing system at an intake system to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
40. The non-transitory computer-readable media of claim 38 , wherein the one or more processing operations comprise configuring an abstract syntax tree (AST) representation of the iterative publish-subscribe-based message processing system and implementing the AST representation to perform streaming processing between the source publish-subscribe messaging system and the destination publish-subscribe messaging system.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 1, 2018
September 15, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.