Scalable key state for network encryption

This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63/440,586, filed on Jan. 23, 2023, the contents of which is incorporated herein by reference in its entirety.

This invention was made with Government support under Contract Number H98230-23-C-0350 awarded by the Maryland Procurement Office. The Government has certain rights in this invention.

Ensuring data privacy is an increasingly important element of supercomputing. Sensitive data, including healthcare patient data, data labeled as “Top Secret,” and other types of sensitive data, are transmitted and stored throughout these supercomputer systems. In traditional systems, the data may be accessed by system administrators of the supercomputer system. When the supercomputer is used to host multiple applications and users, the administrative users of the shared infrastructure could theoretically access other user's systems. However, users should be confident that other users and the system administrators cannot access their data.

The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed.

Providing strong guarantees as to authenticity and privacy can enhance the value and use of the supercomputer system. Some traditional systems, like Cloud Service Providers (CSPs), implement encryption ubiquitously. However, this still allows administrative users to access multiple systems when the encryption is removed. Scalable computing solutions within an interconnect fabric or other shared infrastructure are described throughout the disclosure to better implement ubiquitous encryption in these systems.

Examples of the technology described herein implement encryption of data-in-motion and/or otherwise stored data using a key server and a secure enclave of a Network Interface Card (NIC). The secure enclave of the NIC may store a Unique Device Secret (UDS) that can act as an immutable fingerprint confirming the identity of the NIC. The UDS may not be transmitted outside of the NIC's secure enclave and instead be used to generate a public key. The public key associated with the UDS can be added to a key request and, when the key request is received by the key server, the key server can look up the previously stored private/public key pair that can be provided to the authenticated NIC. Once authenticated, the key server can provide private/public key pair to the NIC to allow the NIC to later encrypt data-in-motion. The NIC can act as a passthrough between the client device and the shared infrastructure of the supercomputer system, referred to as the interconnect fabric.

The encryption of data-in-motion refers to encryption of data that are transmitted as data packets throughout the interconnect fabric, from a first device to a second device, rather than implementing encryption as the data is statically stored at a particular device, sometimes referred to as data-at-rest. The data-in-motion may be parsed and transmitted as data packets to the second device, such that a plurality of the data packets are encrypted as it is transmitted (e.g., in motion). The encryption of data-at-rest, comparatively, refers to implementing the encryption of the data before or after transmitting the data, so that the data are encrypted as it is statically stored at the particular device. The data packets transmitted while the data is in motion may be aggregated to form the data-at-rest. The aggregated data may be stored in the particular device rather than storing parsed data separately (e.g., corresponding with a plurality of the data packets in motion). In some examples, the data-at-rest and the data-in-motion may refer to mirror images of the same data, yet converted to different formats for transmission/storage purposes.

Data-in-motion and data-at-rest may be transmitted and encrypted through various methods discussed herein. For example, data-in-motion that is received by the system may be stored in a memory as data-at-rest. In some examples, the data-at-rest may be accessed for encrypting (e.g., using an AES-XTS encryption accelerator), encrypted, then returned to memory in its encrypted format. The encrypted data may be transmitted as data-in-motion and processed through an additional encryption process.

The encryption of the data-in-motion and the data-at-rest may be implemented differently from each other, while various methods may rely on the key server and a Unique Device Secret (UDS) stored in a secure enclave of the NIC to improve authenticity and privacy in the system. For example, data-in-motion may be transmitted from the first/client device. As discussed herein, the UDS may not be transmitted outside of the NIC's secure enclave. Other devices or users may not have access to view the UDS as well. The UDS may be provided to a silicon-level process (e.g., Caliptra or DICE) with other information to form composite certificates and private/public key pairs. These composite certificates and private/public key pairs can be used to establish secure communication with the key server. Once the secure communication channel is established, the NIC may request an encrypted key from the key server. The encrypted key may be transmitted from the key server to the NIC. The NIC may decrypt the key and subsequently use the key to encrypt the data packets. The encrypted data packet using the key from the key server may be transmitted to a target NIC. Once the target NIC receives the data packet, it may analyze the data packet to select the key (that it received from the key server) to decrypt the data packet.

In some examples, the analysis may be performed by a Root-of-Trust (RoT) in the target NIC, which integrates security functionality directly into the hardware level of the device. In some cases, the ROT implements the security functionality by making an immutable fingerprint (e.g., the NIC's own UDS) in the silicon of the NIC that provides advanced levels of authentication and protection (e.g., against firmware attacks). Using this functionality, the firmware in the RoT in the NIC may initiate an HTTPS connection to fetch the key from the key server, allowing the ROT to retrieve the encrypted key to add to the data packet without transmitting the UDS and improving the authentication process of the NIC.

Upon receiving the request to fetch the key from the NIC, the key server may generate a new key or recover an existing encrypted key from its own storage and decrypt it. Various methods may be implemented to generate the new key. In one example, the key server generates a random key. This random key generation may be a default process for the encryption and decryption of data-in-motion. The key may have a short lifetime for use (e.g., various time ranges, including limitations of a second or a day). This technical feature may limit the access to the data for data-in-motion. For example, when the data are fraudulently captured and a data decryption is attempted, the key may be unretrievable, causing the data to be in accessible. This may help obscure the data from users other than the intended recipient for an extended duration of time. In contrast, data-at-rest may be intended to be decrypted many times over various time frames (e.g., weeks, months, years, etc.).

In either of these implementations, the new key may be requested based on the UDS of the NIC. The received encrypted key from the key server may be used to encrypt new data packets. The key may be returned by the key server to the NIC via the HTTPS connection that was established between the key server and the NIC. The RoT in the NIC may decrypt the new key and store it in a crypt engine of the NIC. Storage key encryption follows a similar path, with the first/client device authenticating themselves and requesting a new key for use with a specific NIC. The new key may be encrypted with the NIC's public key and returned to the first/client device.

Data may be encrypted using the Advanced Encryption Standard's Galois Counter Mode (AES-GCM) algorithm for authenticating and encrypting/decrypting packet data. AES-GCM is a 128-bit block cipher that uses hashing over a binary Galois field to provide authenticated encryption and authenticated decryption. A particular request or service may associate a plurality of applications for the given request or service. The applications may share a key. Replay may be detected using the reliability protocol. Network encryption may have four inputs, including a 256-bit secret key, a 96-bit nonce, the plaintext to be encrypted, and additional data. The 96-bit nonce may comprise, for example, the 32-bit source identifier and a 64-bit unique initialization vector (IV). The format of the nonce may correspond with 96-bits, although other implementations are possible. In some examples, the nonce may be a public source ID that is unique to a plurality of NICs and transmitted with the packet. The output of the encryption may include the encrypted ciphertext and the 128-bit ICV (Integrity Check Value, used for authentication purposes).

Data-at-rest can also be encrypted with a user's key from the key server. The Storage Crypt Engine (SCE) may perform encryption using various methods described herein, including AES-GCM or AES-XTS (e.g., operating on 4K blocks). Storage encryption for AES-GCM is described above, and storage encryption for AES-XTS may have four inputs, including two 256-bit secret keys provided at the start of processing (e.g., one for block encryption of the plaintext and the other for encryption of the “tweak” value), a 128-bit tweak value that is unique for various segments processed for a given key, and the plaintext to be encrypted. The output of the encryption may include the encrypted ciphertext. The length of the encrypted ciphertext may be the same length as the input. For storage applications, the tweak value may be defined as a base value at the start of a file and incremented for subsequent contiguous data units within the file.

Technical improvements are realized throughout the disclosure. For example, the disclosed technology can improve data security of data-at-rest and data-in-motion for a scaled, distributed environment. The data may be transmitted between numerous computing systems of different entities without relying on the protocols and individual data security for those systems, so that data is authenticated and secure along a plurality of the data paths, to enable secure data transmissions upon scaling massive distributed systems. Some implementations of the system may include, for example, crypto shredding corresponding with a limited data retention or GDPR requirements, time-boxed sensitive data access (Classified, HIPAA, Legal, Evidentiary), data segmentation strategies (e.g., Classified, HIPAA, Census, Legal, Evidentiary labels), or implementing a managed “Need to Share” environment where data transmission boundaries are cryptographically enforced.

Technical improvements of the system also enhance scalability to provide technical improvements for large, distributed systems overall. For example, in many existing systems, one key is needed for a pair of clients. This means that a system with 80,000 NICs may maintain 80,000 keys for transmitting and 80,000 keys for receiving. In addition, an NIC may maintain a sequence number with the keys (e.g., 80,000 sequence numbers for transmit and 80,000 for receive). In the disclosed system, one key may be shared among the NICs for encrypting and decrypting all traffic, where the NICs may correspond with a unique UDS that is embedded with the NIC during the manufacturing process and not transmitted external to the NIC. This key sharing may be implemented by generating the packet using a single key (e.g., Key A) and a unique nonce. The NIC may also implement a counter of all generated packets. In some examples, the packets generated by the NIC use the single key with a nonce attached to it (e.g., consisting of a source identifier and the counter value). This may help ensure that every packet that the NIC sends to the network for that key has a unique nonce value and further improve data security. The value generated by the counter may be replaced with a timestamp, in some examples, without loss of generality as long as the timestamp is guaranteed to be unique for every packet generated by that NIC.

is an illustrative distributed computer system that implements a scalable key state for network encryption, in accordance with some examples of the disclosure. In example 100, key server, interconnect fabric, NIC, and client deviceare illustrated. One or more NICsand client devicesmay be implemented with the distributed computer system without diverting from the disclosure.

Key serveris configured to generate and transmit keys via interconnect fabric. The keys that result from such a process are used by NICto encrypt transmitted data packets that can originate with client device. Different data packets generated on the same NICmay be encrypted with different keys to enforce access control and different privilege levels.

The keys may be transmitted via an HTTPS channel between key serverand NIC. In a later stage, authorized client deviceobtains the group keys through key serverusing DTLS. In this example, Datagram Transport Layer Security (DTLS) corresponds with a type of mutual authentication and key exchange to establish a secure connection. During DTLS, the two devices in the connection (e.g., client deviceand key server) authenticate each other using the TLS protocol. The mutual authentication may be performed by the devices at the same time.

Key serveris also configured to generate a new key or recover an encrypted from its own storage and decrypt it. Key servercan then encrypt the key using the public key of NICand transmit the key to NIC, so that NICcan decrypt the key using its own private key.

As described herein, encryption in this system may take various forms. For example, for some data-in-motion, the encryption may use the AES-GCM algorithm for authenticating and encrypting/decrypting packet data. GCM is a 128-bit block cipher that uses hashing over a binary Galois field to provide authenticated encryption and authenticated decryption. The application processes that make up a given job (or using a service) share a key. Replay is detected using the reliability protocol. Network encryption has four inputs: a 256-bit secret key, a 96-bit nonce, the plaintext to be encrypted, and additional data. The 96-bit nonce may comprise, for example, the 32-bit source identifier and a 64-bit unique IV. The output of the encryption may include the encrypted ciphertext and the 128-bit ICV.

Interconnect fabricmay comprise a networking switch or head unit where computing components are attached, including key serverand each NIC. Access to networks and storage is then provided through interconnect fabric.

NICcomprises secure enclaveand is configured to act as a passthrough between client deviceand the shared infrastructure of the distributed computer system, which includes other NICs and key server.

Secure enclavemay comprise the ROT, fuse block, decrypt and encrypt engines (e.g., for network traffic encryption), and corresponding memories. UDS and some keys may be kept internally and not available outside of secure enclaveand client devicemay not have access to them. The fuse block may contain the UDS of NICand a secure mode bit. In some examples, ROT may adhere to an open source specification, like Caliptra®, or implement a Physically Unclonable Function (PUF)-based RoT. In these examples, the functionality of the ROT may further protect the identifier from being physically inspected as it is stored in secure enclave.

In some examples, secure enclavemay come out of reset mode in a secure mode. The secure mode may disable test and scan access. The secure mode can be disabled until manufacturing scan/test is complete, the UDS has been programmed and stored in secure enclave, and the secure mode bit has been burned on NIC. Secure enclavemay then be in secure mode forever after.

Once the UDS is stored and not transmitted, the UDS may be used to generate a public key during an enrollment process. The public key associated with the UDS can be added to a key request and, when the key request is received by the key server, the key server can look up the previously stored private/public key pair that can be provided to the authenticated NIC. For example, (a) the public key/certificate for the of the key server may be distributed during installation and (b) the public keys for each NIC may be installed in key server.

In some examples, NICuses connectionless protocols. Network encryption may be integrated into this model, with the set of processes sharing a Secure Domain Identifier (SDI) that identifies a secret key. Secure distribution of these keys may be implemented. The encryption engines can also perform line rate encryption for point-to-point traffic, e.g., IP traffic between an edge device and a process running within an application. NICis also configured to receive data-in-motion (e.g., as a stream of data packets) from client deviceand add an encrypted key to the data packet. Additional detail of the data packet format is provided withand.

NICis also configured to receive key material from key server. Firmware in the RoT in secure enclaveof NICinitiates an HTTPS connection to fetch the key material from key server. Key servercan send an encrypted key to NICusing the public key generated from the UDS of NIC. The RoT in NICcan decrypt the key and store it in the appropriate crypt engine.

Storage key encryption (e.g., for data-at-rest) follows a similar path, with client deviceauthenticating themselves and requesting a key for use with a specific NIC, including NIC. The key is returned to client device, encrypted with the private key of NIC.

In some examples, the data-in-motion that is received by the system is stored in a host memory as data-at-rest. The data may be accessed for encrypting (e.g., using an AES-XTS encryption accelerator), encrypted, then returned to memory in its encrypted format. In some examples, the host computer may be sent the encrypted data as data-in-motion. There may not be an extension header for the encryption for data-at-rest.

In some examples, data-at-rest encryption may be accelerated. The data encryption may be implemented for data-at-rest (e.g., AES-XTS) encryption “inline” as part of the network transfer.

In some examples, NICcan retrieve a key for storage encryption from key server. Key servercan provide a key to NICin an encrypted form. A similar process for retrieving and providing the key may be implemented for both data-in-motion and data-at-rest encryption.

In some examples, encryption for data-at-rest can also be integrated into the network pipeline. For example, a block of data may be retrieved from host memory and the system may break it into data packets, encrypt the data packets, and perform various actions. The actions may include, for example, storing an encryption state so that the system can retransmit the data packets at a later time or re-encrypt the data packets when they are retransmitted. The actions may also include an update to the “tweak” value (described below) to help allow transmission of any block of data aligned to any position in the file. The block of data may correspond with any size (e.g., within practical limits).

In some examples, encryption keys are referred to using a handle once they are installed at NIC. Network encryption handles may be stored within NICand may be bound to the queue through which a user process or system service submits the commands. Client devicemay install a key by presenting the encrypted key material supplied by key server. The key material may be decrypted by the ROT at NICand the handle may be returned. DMA commands may be tagged with these handles and may be passed with each packet to the Storage Crypt Engine and the Outbound Crypt Engine. The Inbound Crypt Engine may perform the decryption of network packets by selecting a key and an algorithm using a pre-programmed match on the packet header.

In some examples, key server, NIC, and client devicemay have valid certificates issued by a certification authority (not shown).

illustrate a key distribution process, in accordance with some examples of the disclosure. Key server, interconnect fabric, NIC, and client deviceinmay correspond with key server, interconnect fabric, NIC, and client devicein. As illustrated in example 200, the UDS is stored in the secure enclave and not transmitted, and the decrypted keys are accessible in the secure enclave of NICand key server, otherwise the keys are encrypted. One or more NICsand client devicesmay be implemented with the distributed computer system without diverting from the disclosure.

In this example, the secure enclave of NICmay store a UDS that acts as an immutable fingerprint confirming the identity of the NIC and is used to generate a public key. The UDS may be stored at the secure enclave of NIC, for example, during the manufacturing of NIC. The public key associated with the UDS can be added to a key request during the enrollment process.

At block, an enrollment process may be initiated. During the enrollment process, the ROT initiates an HTTPS connection to fetch key material from key server. Key servergenerates a new key or recovers one from its own storage and decrypts it. The key is then encrypted using the requesting NIC's public key and returned via an HTTPS connection. The RoT in NICdecrypts the key and stores it in the appropriate crypt engine. Storage key encryption follows a similar path, with the client deviceauthenticating themselves and requesting a key for use with NIC. The key is returned to client device, encrypted with the private key of NIC.

In some examples, the keys may be encrypted and distributed to the NIC when the client deviceinitiates communications with NIC(e.g., at job startup originating with client device). A large job might have 100,000 endpoints (e.g., with client devices) where the client device may need a copy. Once a key for the job has been created, the traffic transmitted via the HTTPS connection may be transmitted in parallel to/from the devices (e.g., to distribute the key, etc.).

In some examples, the enrollment process is performed during the manufacturing process to store device information, including a device ID, Unique Device Secret (UDS), or an initial unencrypted key in the secure enclave of NIC. This device information may be restricted from being transmitted using the interconnect fabric. The enrollment process may take place in a physically secure environment prior to new equipment being used. If the NIC keys are being burnt on-site, this burning portion of the enrollment process may occur first. The authentication certificate of key servermay be distributed to NIC. It may be encrypted using the private key of NICand stored for future use. The public keys of the NIC may be uploaded to key server. In some examples, a mutual authentication process (e.g., mutual TLS or mTLS) may be used to validate connections from the ROT of NICto key server.

At block, a data packet is received from client computerat NIC. For example, in response to the enrollment process, the decrypted key may be stored in a secure enclave of NIC. NICcan receive a data packet from client device. The data packet may correspond with the data packet format illustrated inor.

At block, a HTTPS connection may be initiated to transmit a key request to key serverfrom NICto fetch key material from the key server.

At block, the key material may be received at NICfrom key serverthat includes the encrypted key material from the key server via the same HTTPS connection.

At block, NICmay decrypt the key material from key serverusing the decrypted key stored in the secure enclave of NIC. The decrypted key material may be stored in the secure enclave of NICand correspond with client device. In some examples, the decrypt engine of NICmay decrypt network traffic in a process that is separate from decrypting the key material from key serverusing the decrypted key stored in the secure enclave of NIC.

At block, NICmay return handle to client device. DMA commands may be tagged with the returned handle and may be passed with the packets from client deviceto NIC(e.g., the Storage Crypt Engine and the Outbound Crypt Engine).

At block, once the encryption keys are installed at NIC, client devicemay refer to them using the handle received from NIC. Network encryption handles may be stored within NICand may be bound to the queue through which a user process or system service submits the commands. The storage key material is received by ROT from key server, decrypted, and the resulting key is encrypted and given to client device. Client devicecan install the key by sending ROT the encrypted key it previously received. The ROT can decrypt it and install the unencrypted key in the SCE. Client devicemay use the unencrypted key via the handle it received during the installation process. The key material may be decrypted by the RoT at NICand the handle may be returned. DMA commands may be tagged with these handles and may be passed with packets to the Storage Crypt Engine and the Outbound Crypt Engine. The Inbound Crypt Engine may perform the decryption of network packets by selecting a key and an algorithm using a pre-programmed match on the packet header.

may correspond with a data packet format for layer three encryption of data traffic (data-in-motion), in accordance with some examples of the disclosure. In example 300, the data packet format may be provided. In some examples, the headers and trailers may follow IPsec formats.

At, an initial data packet is provided. The data packet may be generated by client deviceand transferred to NIC. The data packet may comprise, for example, fabric identifier(e.g., an ethernet identifier, Internet Protocol (IP) identifier, User Datagram Protocol (UDP) identifier), transport identifier, payload information, and Flow Cytometry Standard (FCS) identifier.

At, NICmay add cryptographic data fields to the data packet received from client device. The cryptographic data fields may comprise, for example, cryptographic headerand cryptographic trailerbetween various the fields in the original data packet, including fabric identifier, transport identifier, payload information, and FCS identifier.

At, examples of cryptographic headerand cryptographic trailerare provided. For example, the inputs to generate cryptographic headermay comprise a 256-bit secret key, a 96-bit nonce (comprising source IDand IV), the plaintext to be encrypted, and additional data. Secret keymay correspond with the SDI protocol that uses a symmetric encryption key, similar to the RADIUS key, in order to encrypt sessions. In some examples, secret keycorresponds with the User Datagram Protocol (UDP). Secret keymay be saved in a node secret file and the file may be deployed manually or automatically. The 96-bit nonce may comprise, for example, the 32-bit source identifierand a 64-bit unique initialization vector (IV). Cryptographic trailermay comprise paddingand ICV.