System and Method for the Centralized Management of Distributed Intermittently Connected Runtime Platforms

This application claims the benefit of U.S. provisional application 63/714,890, which was filed on Nov. 1, 2025 the contents of which is hereby incorporated by reference in its entirety.

Runtime platforms may provide tenant computer programs with an execution environment and supporting services, such as orchestration and scheduling of computer program executions, network service discovery, identity, credential and account management (ICAM), network traffic management, data storage and persistence, audit and application log aggregation, monitoring and observability, and security and compliance scanning, alerting, and enforcement. These platforms may be deployed to a single computing device, or a collection of multiple devices providing distributed compute resources, and may be installed in data centers intended for global user availability (in-cloud), co-located with a userbase or tightly coupled system (on-premises), or mobile (edge).

The individual management of multiple platforms may pose a significant administrative burden, making centralized management a valuable and important solution, especially in disconnected, denied, intermittent, and limited-bandwidth (DDIL) environments. When managing platforms distributed across in-cloud, on-premises, and/or edge installations, existing centralized management systems may depend on reliable connectivity between the centralized systems and distributed platforms. If connectivity is lost, the operational stability of the runtime platform may be jeopardized, impacting tenant program availability and risking data corruption and/or loss.

In one example implementation, a computer-implemented method executed on a computing device may include creating an artifact store pre-loaded with a set of utilities, libraries, and artifacts necessary to perform initial infrastructure creation in a local environment, creating a bootstrapping instance infrastructure for a central management system, and creating two or more availability zones, where each availability zone includes a private subnet and a public subnet. The method may further include transferring state data necessary to perform infrastructure as code (IaC) automation from the local environment to the artifact store.

One or more of the following example features may be included. The method may further include, performing initial configuration of the central management system, and creating a secondary management system from the central management system via IaC and one or more automation pipelines. The method may also include transferring management of the central management system to the secondary management system, and creating and managing distributed tenant platforms through the central management system. If the local environment is a hosted-server, then creating the bootstrapping instance infrastructure may include creating the set of administrator identities and permission policies through a hosted user-interface, creating a bootstrapping network with at least one of internet access, or private cloud-provider endpoint connections to a set of application programming interfaces (API) endpoints used to create the central management system infrastructure, creating a bootstrap bastion and assigning the set of administrator identities and permission policies to the bootstrap bastion, and transferring the artifact store contents to the bootstrap bastion. If the local environment is a dedicated workstation, then creating the bootstrapping instance infrastructure may include preloading the dedicated workstation with a first set of utilities, libraries, and artifacts necessary to perform the initial infrastructure creation, performing initial infrastructure creation using infrastructure-as-code (IaC) based automation, and securely transferring a second set of infrastructure management API keys and/or credentials to the dedicated workstation. Performing initial configuration of the central management system may further include deploying, via the bootstrap bastion, a first set of application images including at least a network driver, a storage driver, an ingress gateway, and an internal artifact store directly to each cluster of network nodes, copying a second set of application images for later use by the central management system to the artifact store, and deploying, via the bootstrap bastion, any remaining system components including at least an identity provider, a source control repository, a continuous integration automation system, a secrets manager, an automated application deployment system, a service mesh, a policy agent, a vulnerability scanner, and time-series and log observability systems. The computer-implemented method may further include performing initial configuration of a first identity, credential, and account management (ICAM) system, performing initial configuration of a first centralized identity management system, performing initial configuration of external authentication for an infrastructure provider, and decommissioning the bootstrapping instance infrastructure. The ICAM system may be configured to initially require mutual transport layer security (mTLS) connections, including client certificates signed by a trusted certificate authority, before allowing access, and a built-in administrator account is initially provided to the ICAM by a secret management system. During initial configuration, all systems may be configured using the built-in administrator credentials provided via the secret management system, where, after the ICAM system is configured, the built-in administrator account may be configured to set up single sign-on access via the ICAM system, and then the built-in administrator accounts may be disabled. The ICAM system may be configured to provide authentication access to an infrastructure's API endpoints and user interface. The ICAM system may be configured to allow automation pipelines requesting temporary credentials for the infrastructure provider to enable secure automated management of infrastructure via the automation pipelines. Before decommissioning the bootstrap bastion all infrastructure as code (IaC), libraries, and tools may be transferred over to the artifact store before decommissioning the bootstrap bastion. Creating a secondary management system from the central management system may further include creating two or more secondary availability zones, wherein each secondary availability zone includes a private subnet, and a public subnet, for each private subnet, creating a secondary cluster of network nodes and a load balancer, for each private subnet, creating a secondary control plane configured to manage connections to the cluster of networked nodes, for each private subnet, creating a secondary temporary bootstrap bastion and assigning a second set of administrator identities and permission policies to administer the cluster of networked nodes, and transferring state data necessary to perform IaC automation to a secondary artifact store. The computer-implemented method may further include creating one or more internal service endpoints to operatively connect a first set of components from the central management system, including at least infrastructure API endpoints, artifact stores, and secret managers, to a corresponding set of components in the secondary management system, performing initial configuration of a secondary identity, credential, and account management (ICAM) system, and performing initial configuration of a secondary centralized identity management system. Application images for the secondary management system deployed may be via the automated application deployment system of the central management system, and where the images served by the artifact store of the central management system may be provided by the secret manager of the central management system. After the secondary ICAM system is configured, a built-in administrator account may be used to configure single sign-on via the secondary ICAM system, and then the built-in administrator accounts may be disabled. Transferring management of the central management system to the secondary management system may include using a secondary automated application deployment system from the secondary management system to manage each subsystem of the central management system, transferring IaC state data for the central management system to the secondary management system, and creating one or more secondary automation pipelines to automate future infrastructure. Creating and managing distributed tenant platforms through the central management system may include installing a firewall configured to create and manage virtual subnets at a tenant system location, establishing a virtual private network tunnel between the tenant system location and the central management system, and creating a management bastion configured to act as an agent of an integration automation system of the central management system, creating identities and permission policies for managing tenant-location infrastructure and storing them in the secrets manager of the central management system, creating one or more IaC repositories in the central management system and one or more integration pipelines to apply the IaC using credentials securely retrieved from the secrets manager of the central management system, creating a cluster of control plane nodes, via the integration pipelines, preloaded with application images from the control plane component and creating a cluster of worker nodes preloaded with application images from the secondary artifact store, pushing a plurality of application images from the control plane to the secondary artifact store, creating and transferring administrator credentials and permission policies to the secrets manager of the central management system, deploying a plurality of components from the application deployment system central management system's using credentials securely provided by the secrets manager, and adding tenant applications to the cluster of control plane nodes via the integration pipelines configured to push application images to the secondary artifact store of cluster of worker nodes and deploying the applications via the application deployment system of the central management system. The integration pipelines may be executed from an agent on the management bastion to enable the centralized management of edge infrastructure.

In another example implementation, a centralized management system for distributed intermittently connected runtime platforms may include an artifact store pre-loaded with a set of utilities, libraries, and artifacts necessary to perform initial infrastructure creation, a primary management system deployed on a hosted server, and configured to manage and control a plurality of runtime platforms, where the primary management system includes two or more availability zones, and each availability zone includes a private subnet and a public subnet. The centralized management system may also include a secondary management system operatively connected to the primary management system, and configured to manage and control the primary management system, where the secondary management system may be further configured to ensure continuity of command and control in the event of a loss or disruption of the primary management system, and where the secondary management system may be configured to recover and replace the primary management system via autonomous methods and automated routines.

One or more of the following example features may be included. The primary management system may be further configured to store and execute automated routines that create, initialize, and monitor platform infrastructure for one or more connected runtime platforms, and deploy, initialize, and monitor a combination of services, tenant applications and security systems hosted on the one or more runtime platforms.

Described herein may be systems, methods, and computer program products for the centralized management of distributed intermittently connected runtime platforms. A high-availability pair of management systems may provide centrally available autonomous and user-driven services for the creation and management of distributed runtime platforms and their associated services, tenant programs, user and service identities, and data storage and persistence. Administrators may establish connectivity between a remote site and the centralized environment, then may invoke automated routines in the centralized management system to create and configure the distributed runtime platform and its services. Systems and computer programs required for operational stability of tenant applications may be deployed to or co-located with the distributed platform and maintain best-effort connectivity with the centralized management systems to enable remote management, configuration, and monitoring. Tenant programs may be initially deployed from the centralized management system but ultimately orchestrated by the host runtime platform and supported by its internal or co-located systems, ensuring that tenant program dependencies remain available when disconnected from the centralized management system. Identity, credential, and account management systems (IdAM/ICAM) may be centrally managed but distributed to each runtime platform to enable authentication, security, and compliance scanning, alerting, and enforcement systems that may be deployed to or co-located with the runtime platform. Automated enforcement may be configured by and observable from the centralized management system, but may not rely on connectivity for autonomous operations.

There may be a need for methods and systems to enable centralized management of runtime platforms, their services, and their tenant computer programs, including the management and monitoring of each, that retains operational stability and data integrity of the managed runtime platforms, their services, and their tenant computer programs during periods of degraded or lost network connectivity. The components of the invention may include: a primary and secondary management system; an independent orchestration system for each runtime platform instance; a service discovery and network traffic management system for each runtime platform instance; a centrally managed identity, credential and account management system with identity subsystems distributed with each runtime platform instance; centrally managed independent data storage and persistence systems distributed with each runtime platform instance; centrally managed independent security, compliance and alerting systems distributed with each runtime platform instance; centrally managed observability systems with centrally cascading alerting, logging and metrics data; an infrastructure-specific bootstrapping and low-level management system.

Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The present disclosure may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the disclosure to those skilled in the art.

1 FIG. 10 12 14 12 12 10 Referring to, there is shown a management processthat may reside on and may be executed by server computer, which may be connected to network(e.g., the internet or a local area network). Examples of server computersmay include, but are not limited to: a personal computer, a server computer, a series of server computers, a minicomputer, and a mainframe computer. Server computermay be a web server (or a series of servers) running a network operating system, examples of which may include but are not limited to: Microsoft Windows XP Server™; Novell Netware™; or Redhat Linux™, for example. Additionally, and/or alternatively, management processmay reside on a client electronic device, such as a personal computer, notebook computer, personal digital assistant, or the like.

10 16 12 12 16 The instruction sets and subroutines of the management process, which may be stored on storage devicecoupled to server computer, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into server computer. Storage devicemay include, but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID array; a random access memory (RAM); and a read-only memory (ROM).

12 12 14 14 18 Server computermay execute a web server application, examples of which may include but are not limited to: Microsoft IIS™, Novell Webserver™, or Apache Webserver™, which allows for HTTP (i.e., HyperText Transfer Protocol) access to server computervia network. Networkmay be connected to one or more secondary networks (e.g., network), examples of which may include but are not limited to: a local area network; a wide area network; or an intranet, for example.

12 20 20 22 24 26 28 10 22 24 26 28 12 10 20 20 tm Server computermay execute one or more server applications (e.g., server application), examples of which may include but are not limited to, e.g., Microsoft ExchangeServer, etc. Server applicationmay interact with one or more client applications (e.g., client applications,,,) in order to execute management process. Examples of client applications,,,may include, but are not limited to, EDAs or design verification tools such as those available from the assignee of the present disclosure. These applications may also be executed by server computer. In some embodiments, management processmay be a stand-alone application that interfaces with server applicationor may be applets/applications that may be executed within server application.

20 16 12 12 The instruction sets and subroutines of server application, which may be stored on storage devicecoupled to server computer, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into server computer.

12 10 38 40 42 44 30 32 34 36 10 22 24 26 28 10 12 38 40 42 44 As mentioned above, in addition, or as an alternative to being server-based applications residing on server computer, management processmay be a client-side application residing on one or more client electronic devices,,,(e.g., stored on storage devices,,,, respectively). As such, management processmay be a stand-alone application that interfaces with a client application (e.g., client applications,,,), or may be applets/applications that may be executed within a client application. As such, management processmay be a client-side process, server-side process, or hybrid client-side/server-side process, which may be executed, in whole or in part, by server computer, or one or more of client electronic devices,,,.

22 24 26 28 30 32 34 36 38 40 42 44 38 40 42 44 30 32 34 36 38 40 42 44 38 40 42 44 22 24 26 28 46 48 50 52 The instruction sets and subroutines of client applications,,,, which may be stored on storage devices,,,(respectively) coupled to client electronic devices,,,(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices,,,(respectively). Storage devices,,,may include but are not limited to: hard disk drives; tape drives; optical drives; RAID arrays; random access memories (RAM); read-only memories (ROM), compact flash (CF) storage devices, secure digital (SD) storage devices, and memory stick storage devices. Examples of client electronic devices,,,may include, but are not limited to, personal computer, laptop computer, personal digital assistant, notebook computer, a data-enabled, cellular telephone (not shown), and a dedicated network device (not shown), for example. Using client applications,,,, users,,,may utilize the EDA to create an electronic design.

46 48 50 52 20 22 24 26 28 38 40 42 44 46 48 50 52 20 14 18 12 20 14 18 54 Users,,,may access server applicationdirectly through the device on which the client application (e.g., client applications,,,) is executed, namely client electronic devices,,,, for example. Users,,,may access server applicationdirectly through networkor through secondary network. Further, server computer(e.g., the computer that executes server application) may be connected to networkthrough secondary network, as illustrated with phantom link line.

10 14 18 38 14 44 18 40 14 56 40 58 14 58 56 40 58 42 14 60 42 62 14 In some embodiments, management processmay be a cloud-based process as any or all of the operations described herein may occur, in whole, or in part, in the cloud or as part of a cloud-based system. The various client electronic devices may be directly or indirectly coupled to network(or network). For example, personal computeris shown directly coupled to networkvia a hardwired network connection. Further, notebook computeris shown directly coupled to networkvia a hardwired network connection. Laptop computeris shown wirelessly coupled to networkvia wireless communication channelestablished between laptop computerand wireless access point (i.e., WAP), which is shown directly coupled to network. WAPmay be, for example, an IEEE 802.11a, 802.11b, 802.11g, Wi-Fi, and/or Bluetooth device that is capable of establishing wireless communication channelbetween laptop computerand WAP. Personal digital assistantis shown wirelessly coupled to networkvia wireless communication channelestablished between personal digital assistantand cellular network/bridge, which is shown directly coupled to network.

802 11 x As is known in the art, all of the IEEE 802.11x specifications may use Ethernet protocol and carrier sense multiple access with collision avoidance (CSMA/CA) for path sharing. The various.specifications may use phase-shift keying (PSK) modulation or complementary code keying (CCK) modulation, for example. As is known in the art, Bluetooth is a telecommunications industry specification that allows e.g., mobile phones, computers, and personal digital assistants to be interconnected using a short-range wireless connection.

38 40 42 44 Client electronic devices,,,may each execute an operating system, examples of which may include but are not limited to Microsoft Windows™, Microsoft Windows CE™, Redhat Linux™, Apple iOS, ANDROID, or a custom operating system.

10 In some embodiments, management processmay include any or all of the features and operations described herein.

A bootstrap system may be required to instantiate the central primary management system. This system may be either a dedicated workstation computer with preloaded artifacts and applications or a temporary system hosted within the same regional or on-premises infrastructure provider as the primary management system. In the case of bootstrapping operations from a dedicated workstation, the bootstrapping process may include preloading the workstation with the utilities, libraries, and artifacts necessary to perform the initial infrastructure creation via IaC-based automation, and then securely transferring the appropriate infrastructure management API keys and/or credentials to the workstation.

2 FIG. 10 10 202 Referring now to, a flowchart of management processillustrating how a bootstrap system may be used to instantiate a central primary management system consistent with embodiments of the present disclosure is provided. Management processmay start by creating () an artifact store pre-loaded with a set of utilities, libraries, and artifacts necessary to perform initial infrastructure creation in a local environment

204 206 10 208 210 212 10 214 216 218 220 10 222 224 226 10 228 230 232 Creating () a bootstrapping instance infrastructure for a central management system, and creating () two or more availability zones, where each availability zone includes a private subnet, and a public subnet. Management processmay further include creating (), for each private subnet, a cluster of network nodes, a load balancer, a control plane configured to manage connections to the cluster of networked nodes, a temporary bootstrap bastion and assigning a set of administrator identities and permission policies to administer the cluster of networked nodes, transferring () state data necessary to perform infrastructure as code (IaC) automation to the artifact store, and performing () initial configuration of the central management system. Management processmay also include performing () initial configuration of a first identity, credential, and account management (ICAM) system, performing () initial configuration of a first centralized identity management system, performing () initial configuration of external authentication for an infrastructure provider, and decommissioning () the bootstrapping instance infrastructure. Management processmay then proceed by creating () a secondary management system from the central management system via IaC and one or more automation pipelines, creating () one or more internal service endpoints to operatively connect a first set of components including at least infrastructure API endpoints, internal artifact stores, and secret managers from the central management system to a corresponding set of components from the secondary management system, and performing () initial configuration of a secondary identity, credential, and account management (ICAM) system. Then management processmay conclude by performing () initial configuration of a secondary centralized identity management system, transferring () management of the central management system to the secondary management system, and creating () and managing distributed tenant platforms through the central management system.

3 FIG. 300 300 302 302 304 306 308 308 310 308 312 310 314 310 316 300 318 Referring now to, exampleof a hosted infrastructure based bootstrap configuration consistent with embodiments of the present disclosure is provided. The hosted infrastructure based bootstrap configuration of examplemay include a bootstrap network (e.g., bootstrap) hosted in a cloud region. Bootstrapmay further include a network gateway (e.g., gateway) and an availability zone (e.g., zone), where zonemay include a public subnet (e.g., public subnet) and a private subnet (e.g., private subnet). Public subnetmay also include a network address translation (NAT) gateway (e.g., NAT gateway) to enable instances in a subnet to access the internet, while preventing the internet from initiating connections back to those instances. Private subnetmay include a bastion (e.g., bastion) or a temporary or initial bastion host that may be used during the setup (“bootstrapping”) phase of infrastructure to provide secure administrative access to new or uninitialized systems. Private subnetmay also include cloud provider endpoints (e.g., endpoint), which may be network interfaces used to enable private and secure access to cloud provider services without using the public internet. The hosted infrastructure based bootstrap configuration of examplemay also include an artifact store (e.g., artifact store), which may be a centralized storage location where build artifacts such as compiled code, packages, container images, and other output files from a build or continuous integration (CI) or continuous deployment (CD) pipeline may be stored, managed, and versioned.

4 4 FIGS.A andB 3 FIG. 400 400 204 10 400 400 400 300 400 400 402 404 406 400 408 show example flowchartsA andB, outlining the bootstrapping process or the “creating () a bootstrapping instance infrastructure for a central management system” previously mentioned in the discussion of management process. FlowchartsA andB, may outline the bootstrapping process in different contexts. FlowchartA may consider the case of bootstrapping operations from a hosted infrastructure, as previously mentioned in the discussion of examplein, while flowchartB may consider the case of bootstrapping operations from a dedicated workstation (not shown). According to flowchartA, the infrastructure provider's hosted user interface may be used to perform a variety of initialization tasks, such as, creating () the bootstrapping identities and permission policies, creating () a bootstrapping network with internet access and/or private cloud-provider endpoint connections to the API endpoints used to create the central management system infrastructure, and creating () a bootstrap bastion and assigning it the identity and permissions needed to create the central management system infrastructure. The host-based bootstrapping process of flowchartA may further include transferring () the contents of the artifact store to the bootstrap bastion.

400 204 10 412 414 416 In some embodiments, bootstrapping operations may be executed from a dedicated workstation as described in flowchartB, in which case the act of “creating () a bootstrapping instance” in management processmay instead include preloading () the dedicated workstation with a first set of utilities, libraries, and artifacts necessary to perform the initial infrastructure creation, performing () initial infrastructure creation using infrastructure-as-code (IaC) based automation, and securely transferring () a second set of infrastructure management API keys and/or credentials to the dedicated workstation. In this context, “infrastructure-as-code (IaC) based automation” may refer to the practice of managing and provisioning IT infrastructure, such as servers, networks, databases, etc., using code rather than manual processes.

5 FIG. 500 500 400 500 502 504 504 506 508 506 508 506 510 512 510 514 510 516 Referring now to, exampleof an initiated central management system consistent with embodiments of the present disclosure is provided. The central management system infrastructure of examplemay be the result of the bootstrapping process outlined in flowchartA. The central management system of examplemay be hosted on a cloud region and include a bootstrapping network (e.g., bootstrap) and a management network (management network). Management networkmay include two availability zones (e.g., zones,respectively), such that each of zones,may include public and private subnets. First zonemay include a public subnet (e.g., public subnet) and a private subnet (e.g., private subnet), where public subnetmay include a network load balancer (e.g., load balancer) to distribute incoming network traffic across multiple servers. Public subnetmay also include a NAT gateway (e.g., NAT gateway).

512 518 518 516 512 520 512 522 524 526 528 512 524 526 508 506 Private subnetmay include cloud provider endpoints (e.g., endpoint), such that endpoint, in conjunction with NAT gatewaymay be configured to facilitate egress. Private subnetmay further include a hosted Kubernetes control plane (e.g., control plane) configured to manage the overall state of a cluster of Kubernetes nodes, make global decisions, and ensure that the desired state of applications may be maintained. Moreover, in this context, Kubernetes may refer to an open-source container orchestration platform designed to automate the deployment, scaling, and management of containerized applications. Private subnetmay also include managed autoscaling services (e.g., autoscaler) that may be configured to create and manage Kubernetes nodes (e.g., nodes,respectively). Network load balancers may be created to manage ingress into the Kubernetes cluster. A temporary Kubernetes bastion (e.g., bastion) may also be created in private subnetand assigned the necessary identity and permissions to administer the hosted Kubernetes cluster (nodes,). Second zonemay be similarly designed, such that both the public subnet and the private subnet mirror first zoneincluding an identical set of components.

2 6 FIGS.and 600 212 10 600 602 604 606 530 530 512 530 512 530 524 526 500 Referring now to, where flowchartmay outline in greater detail the process of “performing () the initial configuration of the central management system” provided in management process. According to flowchart, initial configuration of the central management system may include deploying (), via the bootstrap bastion, a first set of application images including at least a network driver, a storage driver, an ingress gateway, and an internal artifact store directly to each cluster of network nodes, copying () a second set of application images for later use by the central management system to the artifact store, and deploying (), via the bootstrap bastion, any remaining system components including at least an identity provider, a source control repository, a continuous integration automation system, a secrets manager (e.g., secrets manager), an automated application deployment system, a service mesh, a policy agent, a vulnerability scanner, and time-series and log observability systems. In some embodiments, representations of secrets manager, repositories, and pipelines may be shown in private subnet. This placement may reflect the architectural design where these core management and automation components reside within the secure private subnet of the central management system. Secrets manager, source repository, and CI/CD Pipelines may be positioned together in private subnetbecause they form an integrated automation workflow: the source repository may store the infrastructure-as-code definitions, the CI/CD pipelines may execute these definitions, and secrets managermay provide the necessary credentials for pipeline execution. Placing these components in the private subnet may ensure that they are protected from direct internet access, while remaining accessible to Kubernetes nodes,and management infrastructure within the same subnet. Examplemay represent the central management system's core automation infrastructure, from which all distributed tenant platform deployments may be orchestrated. The components shown in the right-side availability zone may be replicated for high availability in the left-side zone, though only one instance is illustrated to maintain diagram clarity. Additionally, these components may be critically important to the centralized management system's ability to store credentials, maintain infrastructure code, and execute automated deployment pipelines.

an identity provider, a source control repository, a continuous integration automation system, a secrets manager, an automated application deployment system, a service mesh, a policy agent, a vulnerability scanner, and time-series and log observability systems. In some embodiments, the initial configuration of the central management system may be performed from the Kubernetes bastion. Application images for the platform's network driver, storage driver, ingress gateway, and internal artifact store may be deployed directly to the Kubernetes nodes, and each of the respective workloads may be deployed and configured. The remaining application images used by the system may be copied to the internal artifact store, and the remaining system components may be deployed, including:

7 FIG. 700 700 702 704 706 702 704 Referring now to, exampleof an initiated central management system and a secondary management system consistent with embodiments of the present disclosure is provided. The combined central and secondary management system of examplemay be hosted on a cloud region and include a management network (e.g., management network), a secondary network (e.g., secondary network), and an artifact store, where both management networkand secondary networkmay be identically composed. Both networks include two availability zones, wherein each zone may include public and private subnets.

704 708 710 708 710 702 704 700 4 712 714 712 716 712 718 Secondary networkmay include a first availability zone (e.g., first zone) and a second availability zone (e.g., second zone), such that first zoneand second zonemay also be identically composed. Between management networkand secondary network, the combined central and secondary management system of examplemay includeidentical availability zones. Each zone may include a public subnet (e.g., public subnet) and a private subnet (e.g., private subnet), where public subnetmay include a network load balancer (e.g., load balancer) to distribute incoming network traffic across multiple servers. Public subnetmay also include a NAT gateway (e.g., NAT gateway).

714 720 720 718 714 722 714 724 726 728 714 Private subnetmay include cloud provider endpoints (e.g., endpoint), such that endpoint, in conjunction with NAT gatewaymay be configured to facilitate egress. Private subnetmay further include a hosted Kubernetes control plane (e.g., control plane) configured to manage the overall state of a cluster of Kubernetes nodes, make global decisions, and ensure that the desired state of applications may be maintained. Private subnetmay also include managed autoscaling services (e.g., autoscaler) that may be configured to create and manage Kubernetes nodes (e.g., nodes,respectively). Network load balancers may be created to manage ingress into the Kubernetes cluster. At this point the temporary Kubernetes bastion (not shown) may have been decommissioned in private subnetand all infrastructure as code (IaC), libraries, and tools may have already been transferred over to the artifact store.

2 8 FIGS.and 800 222 10 800 802 804 806 808 810 Referring now to, where flowchartmay outline in greater detail the process of “creating () a secondary management system from the central management system via IaC and one or more automation pipelines” provided in management process. According to flowchart, the process of creating a secondary management system from the central management system may include creating () two or more secondary availability zones, wherein each secondary availability zone includes a private subnet, and a public subnet, creating() for each private subnet, a secondary cluster of network nodes and a load balancer, and creating () for each private subnet, a secondary control plane configured to manage connections to the cluster of networked nodes. The process of creating a secondary management system from the central management system may also include creating () for each private subnet, a secondary temporary bootstrap bastion and assigning a second set of administrator identities and permission policies to administer the cluster of networked nodes, and transferring () state data necessary to perform IaC automation to a secondary artifact store.

In some embodiments, once the secondary management system is created and configured, the management of the central management system, previously performed via manual processes performed from bastion systems, may be transferred to the secondary management system. The secondary management system's automated application deployment system may be configured to manage each of the central management system's subsystems, using images served by the secondary management system's internal artifact store and secrets provided by the central management system's secret manager. The infrastructure-as-code state for the central management system may be transferred to the secondary management system, and infrastructure pipelines may be created to automate all future infrastructure. At this stage, the central and secondary management systems may operate as a highly available pair, ensuring that the independent failure of either system may be repaired or recovered from the other system via auditable automation pipelines.

2 9 FIGS.and 900 230 10 900 902 904 Referring now to, where flowchartmay outline in greater detail the process of “transferring () management of the central management system to the secondary management system” provided in management process. According to flowchart, the process of transferring management of the central management system to the secondary management system may include using () a secondary automated application deployment system from the secondary management system to manage each subsystem of the central management system, and transferring () IaC state data for the central management system to the secondary management system, and creating one or more secondary automation pipelines to automate future infrastructure.

In some embodiments, the secondary management system's network and storage drivers, internal artifact store, identity provider, source control repository, continuous integration automation system, secrets manager, automated application deployment system, service mesh, policy agent, vulnerability scanner, and time-series and log observability systems may be deployed via the central management system's automated application deployment system, using images served by the central management system's internal artifact store and secrets provided by central management system's secret manager.

In some embodiments, the state data associated with the infrastructure-as-code (IaC) automation may be transferred to the artifact store, and the bootstrapping instance infrastructure may either be torn down (hosted infrastructure) or securely wiped (workstation). State data may essentially be a detailed record or “memory” of everything that Infrastructure-as-Code (IaC) tools have created and configured in a cloud environment. In other words, state data may be a comprehensive inventory list that keeps track of: (i) every resource created (servers, networks, databases, etc.), (ii) unique IDs assigned by the cloud provider, (iii) how each resource is configured, (iv) dependency relationships for each resource, and (v) the order in which the resources were created. This state data may come from the IaC tools (e.g., tools like: Terraform, CloudFormation, or Pulumi) when they first build the infrastructure. As these tools create each component by spinning up servers, setting up networks, configuring security rules, etc. the tools may record each action performed in a state-file. This state-file may act like a blueprint that shows a projected final schema, alongside the current state of the schema. The state data may be generated automatically as the IaC tools work, starting from the very first “terraform apply” or equivalent command. The state data may be updated every time a change is made to the infrastructure. Without this state data, the IaC tools may have no way of knowing what components have already been created, what components need to be updated, or how to deconstruct the bootstrap system safely without breaking dependencies. Before shutting down the bootstrap system that initially set everything up, this state data may be saved elsewhere (like in the artifact store) so future automation may continue managing the infrastructure without missing a step or losing track of what components have been created up to that point.

In some embodiments, the state data associated with the infrastructure-as-code automation may be transferred to the artifact store, and the bootstrapping instance infrastructure may either be torn down (hosted infrastructure) or securely wiped (workstation). The state data may include infrastructure configuration metadata that may track the current status and relationships of all provisioned resources within the managed environment. This state data may be generated during the initial infrastructure creation process, when IaC automation tools may provision resources such as networks, subnets, computer instances, storage volumes, and security policies. More specifically, the state data may include resource identifiers, configuration parameters, dependency mappings, and version tracking information that may enable the IaC automation system to maintain consistency between the desired infrastructure configuration and the actual deployed resources. During infrastructure provisioning, the IaC automation tools may generate and continuously update this state data to reflect resource creation, modification, and deletion operations. The state data may originate from the bootstrap bastion during initial setup and may subsequently be maintained by the automation pipelines, enabling infrastructure drift detection, resource lifecycle management, and automated rollback capabilities. Prior to decommissioning the bootstrap infrastructure, this critical state data may be securely transferred to ensure continuity of infrastructure management operations.

10 11 FIGS.- 1000 1100 1000 1100 1000 1002 1004 1006 1100 1102 1104 1106 1008 1108 1010 1110 Referring now to, exampleof an initiated central management system connected to an on-premises/Edge network, and exampleof a combined central management system and secondary management system connected to a customer/tenant network consistent with embodiments of the present disclosure are provided. Both examples,,, may represent instances where the central management system may provide a platform for creating and managing distributed tenant platforms. In examplethe central management system may be hosted on a cloud region and include a management network (e.g., management network) and an artifact store (e.g., artifact store) connected to a tenant's local area network (LAN) (e.g., LAN) via virtual private network (VPN) tunnel. In example, the combined central and secondary management system may also be hosted on a cloud region and include a management network (e.g., management network), a secondary network (e.g., secondary network), and an artifact store, connected to a customer/tenant network that may also be hosted on the same cloud region. The tenant networks in both examples may include public and private subnets (e.g., public subnets,, and private subnets,, respectively).

1000 1012 1100 11 FIG. 10 FIG. In some embodiments, in example, a management bastion (e.g., bastion) may be created and configured as an agent of the central management system's continuous integration automation system. This bastion may be necessary in the on-premises/edge scenario to bridge the gap between the cloud-based management system and the physically separated tenant infrastructure. In contrast, examplemay depict a fully cloud-hosted scenario where both the management systems and the tenant network reside within the same cloud region. In this configuration, the management systems may directly interface with the tenant infrastructure through cloud-native networking and API endpoints, eliminating the need for a separate bastion host. The managed autoscalers shown in the tenant network ofmay directly communicate with the central and secondary management systems through internal cloud provider networks, whereas the on-premises deployment inmay require the bastion to facilitate this communication across the VPN tunnel.

In some embodiments, identities and permission policies for managing tenant-location infrastructure may then be created and stored in the central management system's secrets manager. Infrastructure-as-code (IaC) repositories may be created in the central management system's source repository, and continuous integration pipelines may be created to apply the IaC in automated pipelines using credentials securely retrieved from the secrets manager. The integration pipelines may be executed from the agent on the management bastion, enabling the centralized management of edge infrastructure.

1014 1016 1018 1112 1114 1116 1118 1120 1020 1122 10 FIG. 11 FIG. In some embodidments, each private subnet may further include a cluster of Kubernetes nodes (e.g., nodes,,inand nodes,,,,inrespectively) and at least one hosted Kubernetes control plane (e.g., control plane,respectively).

2 12 FIGS.and 1200 232 10 1200 1202 1204 1206 1208 1210 1212 1214 1216 1218 Referring now to, where flowchartmay outline in greater detail the process of “creating () and managing distributed tenant platforms through the central management system” provided in management process. According to flowchart, the process of creating and managing distributed tenant platforms through the central management system may include installing () a firewall configured to create and manage virtual subnets at a tenant system location, establishing () a virtual private network tunnel between the tenant system location and the central management system, and creating a management bastion configured to act as an agent of an integration automation system of the central management system, and creating () identities and permission policies for managing tenant-location infrastructure and storing them in the secrets manager of the central management system. The process of creating and managing distributed tenant platforms through the central management system may further include creating () one or more IaC repositories in the central management system and one or more integration pipelines to apply the IaC using credentials securely retrieved from the secrets manager of the central management system, creating () a cluster of control plane nodes, via the integration pipelines, preloaded with application images from the control plane component and creating a cluster of worker nodes preloaded with application images from the secondary artifact store, and pushing () a plurality of application images from the control plane to the secondary artifact store. The process may also include creating () and transferring administrator credentials and permission policies to the secrets manager of the central management system, deploying () a plurality of components from the application deployment system central management system's using credentials securely provided by the secrets manager, and adding () tenant applications to the cluster of control plane nodes via the integration pipelines configured to push application images to the secondary artifact store of cluster of worker nodes and deploying the applications via the application deployment system of the central management system.

In some embodiments, the infrastructure pipelines may create Kubernetes control plane nodes preloaded with the control plane components'application images and Kubernetes worker nodes preloaded with the network driver, storage driver, and internal artifact store application images. The control plane may be configured, and the worker nodes may be connected. Application images for an ingress gateway, service mesh, policy agent, vulnerability scanner, and time-series and log observability systems are pushed to the internal artifact system. Administrator credentials and policies may be configured and transferred to the central management system's secrets manager. The ingress gateway, service mesh, policy agent, vulnerability scanner, and time-series and log observability systems may be deployed to the tenant Kubernetes cluster from the central management system's automated deployment system (e.g., Kuberntes Control Plane) using credentials securely provided by the secrets manager.

In some embodiments, tenant applications may be added to the cluster via automated pipelines that push application images to the tenant cluster's internal artifact store and may deploy the applications via the central management system's automated deployment system. Once deployed, the tenant cluster and its applications may be operationally stable in the event of a loss of connectivity to the central management system.

13 FIG. 1300 1300 1302 1304 1306 1308 1308 1310 Referring now to, exampleof system/application logging tools consistent with embodiments of the present disclosure is provided. Each tenant cluster may provide co-located and centralized observability of application logs, audit logs, and time-series metrics data via inwardly cascading streams managed by log aggregators and time-series data managers at each tenant cluster and in the centralized management system. Examplemay show a management network (e.g., management network) hosted on a cloud region alongside a central management system, and a local area network (e.g., LAN) of the tenant's physical location. In use local log forwarders (e.g., log forwarder) on each node in a Kubernetes cluster may collect application logs and forward them to a local aggregator (e.g., local aggregator), applications may then push audit logs to local aggregator, and time-series application metrics may be scraped from the logs by a local time-series data manager (e.g., local manager). Log and time-series data may then be optionally filtered or sampled to reduce their total size and written to co-located data stores with automated lifecycle policies to ensure data remains stored in a manner consistent with organizational policy.

1312 1314 In some embodiments, application logs, time-series metrics, and audit logs may be optionally filtered or sampled and pushed to the central management system's log aggregator (e.g., central aggregator) and time-series data manager (e.g., central manager), to ensure that any successfully pushed/forwarded data may remain available even during periods of loss of connectivity. During periods of connectivity, the tenant log aggregator and time-series data manager may be remotely queried from the central management system, allowing full-resolution observability data on demand.

530 In some embodiments, the central management system's ICAM (identity, credential, and account management) subsystem may provide centralized user management for all tenant application platforms. The ICAM may may be stored within the database and secret manager. However, to enable user login to on-premises and edge tenant applications while disconnected from the central management system, an edge ICAM subsystem may be deployed to each cluster, and user accounts may be distributed. The users of each tenant platform, or group of closely related platforms, may be managed in a realm within the central ICAM subsystem. The Central ICAM may maintain a table of which version of each user account has been successfully synchronized to each edge ICAM. Each edge ICAM may initiate period synchronization requests to the central ICAM. Upon receiving a synchronization request, the central ICAM queries for records that may have been created, updated, or deleted since the edge ICAM instance's last synchronization request. As the edge ICAM instance processes each change, it may send an acknowledgment back to the central ICAM, which then updates its synchronization data. In this way, the central ICAM subsystem may provide a single point of access for all user account management for platform services and tenant applications while enabling (denied, degraded, intermittent, and limited) DDIL-resilient user authentication.

In some embodiments, when initially deployed, the ICAM (identity, credential, and account management) system may be configured to require mTLS (mutual transport layer security) connections with client certificates signed by an explicitly trusted PKI certificate authority. The initial administrator account may be configured with credentials provided to the ICAM workloads via the platform's secret management. Platform administrators may access the ICAM systems'web interface and authenticate with a trusted mTLS certificate (e.g., via a smartcard) and the administrator account credentials. The administrators may provision user-specific administrator accounts and then disable the built-in administrator user.

In some embodiments, during the initial configuration stage, all systems may be configured using built-in administrator credentials provided to the repository's workloads via the platform's secret management. After the ICAM system is configured, the built-in administrator account may be used to configure single sign-on via the ICAM system, and then the built-in administrator accounts may be disabled.

In some embodiments, when the infrastructure provider supports external authentication, the central management system's ICAM provider may be configured to provide authentication to engineers accessing the infrastructure provider's API endpoints and user interface. Additionally, the CI Automation and ICAM provider may be configured to allow automation pipelines to request short-lived credentials for the infrastructure provider, enabling secure automated management of infrastructure via auditable pipelines.

In some embodiments, before decommissioning the Kubernetes bastion, all infrastructure as code, libraries, and tools may be transferred to the source repository, artifact repository, or CI automation runners, enabling fully automated management of infrastructure and systems.

In some embodiments, the secondary management system's identity, credential, and account management may be configured following the same process used with the central management system, including the management of short-term infrastructure-provider credentials.

The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

Although a few example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from the scope of the present disclosure, as described herein. Accordingly, such modifications are intended to be included within the scope of this disclosure as defined in the following claims. In the claims, means-plus-function clauses may cover the structures described herein as performing the recited function and not only structural equivalents, but also equivalent structures. Thus, although a nail and a screw may not be structural equivalents in that a nail employs a cylindrical surface to secure wooden parts together, whereas a screw employs a helical surface, in the environment of fastening wooden parts, a nail, and a screw may be equivalent structures. It is the express intention of the applicant not to invoke 35 U.S.C. § 112, paragraph (f) for any limitations of any of the claims herein, except for those in which the claim expressly uses the words ‘means for’ or ‘step for’ together with an associated function.

Having thus described the disclosure of the present application in detail and by reference to embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure defined in the appended claims.