Fly parameter compression and decompression to facilitate forward and/or back propagation at clients during federated learning

Federated learning of machine learning (ML) model(s) is a technique for training ML model(s) in which an on-device ML model is stored locally on a client device of a user, and a global ML model, that is a remote-based counterpart of the on-device ML model, is stored remotely at a remote system (e.g., a remote server or a cluster of remote servers). The client device, using the on-device ML model, can process user input detected at the client device to generate predicted output, and can generate a gradient based on the predicted output in a supervised or unsupervised manner.

In some implementations, the client device can transmit the gradient to the remote system. The remote system can utilize the received gradient, and, additional gradients generated in a similar manner at the client device or additional client devices, to update weights and/or other parameters of the global ML model, to generate an updated global ML model.

In some additional or alternative implementations, the client device can locally train on-device ML model based on the gradient and additional gradients generated in a similar manner at the client device. The client device can transmit the locally trained on-device ML model (e.g., at least updated weights and/or parameters thereof) to the remote system. The remote system can aggregate the locally trained on-device ML model, with other locally trained on-device ML models and/or the global ML model, to generate an updated global ML model. The remote system can transmit an updated global ML model, and/or updated weights and/or parameters of the global ML model, to the client device. The client device can then replace the on-device ML model with the updated global ML model, or replace any parameters of the on-device ML model with the updated parameters of the updated global ML model, thereby updating the on-device ML model.

However, federated learning of ML model(s) can cause computational resources to be unnecessarily consumed at the client device and/or the client device may not have sufficient resources to store and train full-sized ML model(s) locally at the client device. For example, assume an on-device automatic speech recognition (ASR) model is utilized to generate a transcription that is predicted to correspond to a spoken utterance captured in a stream of audio data via microphone(s) of the client device. In this example, the predicted output can correspond to the transcription, or a portion thereof, and the gradient can be generated based on the transcription. In generating the gradient, on-device memory would be consumed by storing activations at each layer of the on-device ASR model, and on-device computational resources would be consumed by: computing an attributable loss for each portion of the on-device ASR model, and causing the loss to be backpropagated appropriately across each portion of the on-device ASR model. As a result, consumption of on-device memory and other computational resources (e.g., network bandwidth, processing bandwidth, etc.) would be inefficiently consumed across the numerous client devices being employed for facilitating the federated learning process.

Implementations disclosed herein are directed to efficient federated learning of machine learning (ML) model(s) through on-the-fly decompression and compression of model parameters, of the ML model(s), when facilitating forward propagation and/or back propagation at client device(s). For example, implementations can transmit, from a remote system to a client device, a compressed on-device ML model that includes some compressed parameters. For example, a majority of the ML model parameters can be compressed, 90% or more of the ML model parameters can be compressed, and/or all of the ML model parameters can be compressed.

Further, the client device can, in performing forward propagation and/or back propagation using the on-device ML model, decompress those compressed parameters on-the-fly as the parameters are needed for the propagation. The propagation will utilize the decompressed parameters that were decompressed on the fly. Further, after the decompressed parameters are utilized, they can be deallocated from memory (while their compressed counterparts optionally remain in memory) to enable allocation of memory for further decompressed parameters that will be needed next and/or for other ongoing process(es). Yet further, in the case of back propagation, any decompressed parameters updated by the backpropagation can be compressed and stored (e.g., in memory and then in non-memory storage) in the compressed format prior to being deallocated from memory. A compressed updated parameter can replace a compressed pre-updated parameter in memory and/or in non-memory storage, and the compressed updated parameter utilized in a next forward propagation in lieu of the compressed pre-updated parameter.

In these and other manners, forward and/or back propagation using the on-device ML model is enabled, while mitigating the extent of memory utilization at the client device. This enables a client device to perform forward and/or back propagation without overly constraining memory resources of the client device. Further, and as referenced above, the compressed on-device ML model can be transmitted from the remote system to the client device. Yet further, an updated compressed on-device ML model can be transmitted from the client device to the remote system. Transmitting the compressed on-device ML model, in lieu of an uncompressed counterpart, can preserve limited network resources. Such preservation of limited network resources is especially impactful in many federated learning scenarios where hundreds or thousands of client devices can be involved.

In some implementations, processor(s) of a given client device can receive client input data and process the client input data using a given on-device ML model to generate predicted output. The processing of the client input data via forward propagation can involve decompressing, by a client device, client model data (e.g., parameter(s)) for at least a portion of a given layer, of the on-device ML model, that is currently being used in processing, while other client model data remains compressed but, optionally, allocated to memory (e.g., RAM or other memory) in the compressed state. When forward propagation using the decompressed client model data for the given layer is completed, the client device memory can be deallocated for the decompressed model data, and then reallocated for additional decompressed model data. For example, it can be reallocated for to be decompressed model data for a next portion of the given layer to be used in processing, or for a next layer to be used in processing. When forward propagation is completed, and a predicted output for the on-device ML model has been generated, a loss can be generated for updating model parameters during back propagation. For example, the loss can be generated in a supervised manner based on comparing the predicted output to ground truth output. For instance, the ground truth output can be generated based on implicit or explicit user feedback provided in response to rendering the predicted output to the user.

As a particular example, for a given layer of a given on-device ML model that is currently being used in back propagation, a subset of compressed client model data for the given layer can be decompressed and the decompressed client model data allocated to memory for use in back propagation for that given layer. When back propagation for that given layer is completed, the decompressed client model data for the given layer can be deallocated from memory, and the memory reallocated for a different decompressed subset of model data (e.g., a decompressed subset of parameters for a next layer for processing). After back propagation is completed for the on-device ML model and for the loss, an update can be provided, by the client device, to the remote system in a compressed format. For example, the update can be a compressed version of the client device trained on-device ML model (e.g., at least updated weights and/or parameters thereof). In these and other manners, a global ML model of the remote system to be updated based on the compressed update from the client device, and other compressed updates from any other client devices participating in the federated learning process. These compressed updates and/or other compressed model data from the client devices can be decompressed by the server remote system, and the decompressed data can then be utilized to update the global ML model. A subsequent update for all on-device ML models can then be generated and compressed at the remote system, and provided to each participating client device, thereby allowing each on-device ML model to be updated accordingly.

In some implementations, a portion of the client ML model data can be provided in a non-compressed state in which decompression is not necessary when utilizing that portion of the client ML model data for forward propagation and/or back propagation. For example, in some implementations, some model parameters characterized by the client ML model data can be transmitted to the client device in a non-compressed state and can remain in the non-compressed state during forward propagation and/or back propagation. For instance, model parameters for certain layers and/or certain type(s) of model parameters can be non-compressed. As a particular instance, model parameters for scaling factors and/or biases in normalization layers can be non-compressed. As a particular example, assume audio data capturing a spoken utterance of a human user of “Hey Assistant, turn on the kitchen lights” is generated via microphone(s) of a given client device of the human user. In this example, the audio data capturing the spoken utterance can correspond to the client input data. Further, the audio data can be processed using an on-device automatic speech recognition (ASR) model stored in on-device memory of the given client device to generate ASR data, such as one or more speech hypotheses that are predicted to correspond to the spoken utterance captured in the audio data. When generating the ASR data from the audio data using the client ML model (e.g., the ASR model), forward propagation can be performed using an on-device ML model previously received from a remote system. For some layers of the client ML model, parameters that were not previously compressed by the remote system and/or the client device can be utilized to facilitate forward propagation. Alternatively, or additionally, for other layers of the client ML model, a subset of the client ML model data can be decompressed to access other parameters for facilitating the forward propagation.

Continuing the above example, a gradient can be generated for any loss resulting from differences between an output predicted using the on-device ML model and ground truth data, and the gradient can be utilized to determine updated parameters. For example, a predicted output of “Hey Assistant, turn on the basement lights” can result in a loss relative to ground truth data that indicates the spoken utterance embodied the command, “Hey Assistant, turn on the kitchen lights.” Back propagation can be facilitated by determining the loss, in furtherance of generating updated parameters that can be provided to the remote system. For example, updating parameters for some initial layers during back propagation can be performed without decompressing any subset of data (e.g., a bias parameter matrix) of the client ML model data. Alternatively, updating those parameters for those initial layers during back propagation can necessitate decompressing certain subsets of data (e.g., a compressed weight parameter matrix) of the client ML model data. When parameters for a given layer have been updated, those parameters can be compressed or, optionally, remain non-compressed before being communicated to the remote system. In these and other manners, parameters that are more sensitive can be non-compressed or otherwise not be subject to any compression or decompression process. Otherwise, other parameters that are less sensitive can be compressed in furtherance of preserving memory and other resources during federated learning.

The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, and other implementations, are described in more detail below.

Other implementations can include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), digital signal processors (DSP(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations can include a system of one or more computers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Federated learning (FL) allows for training neural network models directly on edge devices (referred to as client devices) instead of transferring their data back to a server for centralized training, thereby allowing FL to preserve user privacy. FL is composed of multiple federated rounds. In a standard federated round, a server model is first transported to clients; then, the clients train the model (e.g., their client ML model) on their local data. The trained models are finally transported back to the server and aggregated to improve the server model (e.g., a global ML model). This process is repeated until the server model converges.

FL involves on-device training and model transportation between servers and clients, which lead to two main challenges. The first challenge is that edge devices usually have limited memory available for training. Given the fact that recent Automatic Speech Recognition (ASR) models can typically contain hundreds of millions of parameters or more, keeping these parameters in full precision in memory can exceed the available memory. Although there may be significant effort in the field on reducing memory usage of parameters during inference, such as quantization-aware training, it is usually at the cost of higher memory usage during training. The second challenge is the high communication cost. Communication can be much slower than computation, and transporting models in full precision also burdens the communication network.

In some implementations, Online Model Compression (OMC) addresses these challenges of on-device FL. Different from regular full-precision FL, where each client keeps, updates, and transports full-precision parameters, OMC keeps and transports at least some of the parameters in a compressed format. During training, when an operation needs the value of a compressed parameter, OMC decompresses it on-the-fly and deallocates memory for the decompressed value immediately after it is consumed. Therefore, OMC keeps the compressed parameters and a smaller number of transient decompressed parameters in memory, which uses less memory than the full-precision parameters.

A design challenge of some implementations OMC is achieving a favorable accuracy-efficiency trade-off since, with OMC, compression and decompression can optionally occur in every training iteration. As a result, the error introduced by compression can accumulate quickly and degrade model accuracy significantly. On the other hand, using a complicated algorithm to control the accumulated error can significantly slow down training and/or utilize a significant quantity of computational resources. In view of these and other considerations, various implementations of OMC seek to operate as simple and fast as possible while mitigating impact on model accuracy. Some of those various implementations of OMC achieve this goal by using quantization, per variable transformation, weight matrices only quantization, and/or partial parameter quantization—each of which is described in more detail herein.

Implementations of OMC reduce memory usage in forward and/or back propagation using a model. There are three main sources of memory usage in forward and/or back propagation using a model: model parameters, activations, and gradients. OMC aims to reduce the memory usage of model parameters. Implementations of OMC, that transmit a compressed model from a server and a client (and/or vice versa), additionally or alternatively reduce network traffic. Because models are transported between servers and clients, reducing the model size helps reduce communication cost. Implementations of OMC additionally or alternatively enable lightweight operation, which is particularly beneficial (or even necessary) on resource constrained client devices. In those implementations, OMC does not significantly slow down the training process even though compression and decompression occur frequently.

1 FIG. 100 102 illustrates a viewof the framework of an implementation Online Model Compression (OMC). OMC stores parameters in a compressed parameter datain a compressed format, such as floating-point numbers with reduced bitwidths, but performs computations in full precision or other hardware-supported formats. This design decouples compression formats and hardware-supported formats to provide higher flexibility for choosing the compression format and method to achieve better memory usage reduction.

112 104 110 108 114 106 102 1 FIG. 1 FIG. When performing forward propagation(represented by the dashed line path in), OMC decompresses parameters on an as-needed basis to generate decompressed layer parametersfor that layer (e.g., layer N, where N is any integer) for layer-N processing, on the fly, and deallocates the decompressed copies immediately after they are consumed. When performing back propagationfor a layer (represented by the non-dashed line path in), OMC provides decompressed layer parametersand applies the gradients to update them. The updated decompressed parametersare then compressed and discarded immediately. Therefore, OMC keeps the compressed parameter dataand a number of transient decompressed copies in memory. The transient decompressed copies, that are in memory at a given time during forward and/or back propagation, can include some (or all) parameter(s) of a layer, or first parameter(s) from a given layer and second parameter(s) from an adjacent layer.

Quantization reduces the number of bits (i.e., bitwidth) for representing a value. While full precision (e.g., 32 bits) quantization can be used in deep learning, neural networks have been shown to be error-resilient and allow using much lower bitwidths, such as 1 to 8 bits, without harming prediction accuracy. However, such low bitwidths are usually utilized for inference (i.e., forward propagation only). Reducing memory usage by quantization during training (e.g., which includes back propagation) is more difficult because training requires more bits to precisely accumulate smaller gradients across training iterations.

In some implementations, OMC adopts the floating-point format as a non-limiting example, although other formats, such as the fixed-point format, can also be used. The floating-point format can comprise three parts: the sign bits, the exponent bits, and the mantissa bits. For example, the format of FP32 (32-bit single-precision floating-point format) is composed of 1-bit sign, 8-bit exponent, and 23-bit mantissa. To quantize a floating-point value, the quantity numbers of bits can be reduced for the exponent and the mantissa, which are the two hyper-parameters of floating-point quantization.

202 204 204 206 200 208 2 FIG. Quantization can be a lossy operation and thus, can undesirably introduce quantization errors. As a result, quantizing parameters for every training iteration can lead to a larger accumulated error and prevent usage of fewer bits with the original accuracy maintained. To mitigate quantization error, implementations of OMC perform per variable transformation. Per variable transformation can include, on a per variable basis, decompressing the variablesto generate initial decompressed parameters, and applying a linear transformation on the initial decompressed parametersto generate transformed parameters, which is illustrated in viewof. This step is performed per variable, such as per weight matrices, so that all the model parameters in a variable can share a few transformation-related parameters to make the memory overhead negligible for the resulting layer-variable data. This can also be performed, in reverse, during compression (e.g., of updated variables). Generally, per variable transformation can utilize a per-variable scaling factor and/or bias, generated and stored along with the compressed data during compression, to refine an initially decompressed parameter (using the scaling factor and/or bias) to mitigate the extent of loss from the compression and decompression.

In some instances, some types of parameters are more sensitive to quantization than the others. These sensitive parameters can include the scaling factors and/or biasing values in normalization layers. In contrast, weight matrices in convolutional and feed-forward layers are less sensitive to quantization but dominate the model size. For example, the weight matrices in a streaming Conformer model can account for 99.8% of the model size. Hence, in some implementations, OMC quantization can be limited to weight matrices, while keeping the remaining variables in FP32. This method helps maintain accuracy while saving a large amount of memory.

OMC also leverages the feature of federated learning that there are many clients training a model in parallel to further reduce quantization errors. This feature provides an opportunity to quantize only a subset of parameters for each client and vary the selection from one client to another. As a result, the server or other remote system can receive high-quality and precise updates of each parameter from the clients that do not quantize this parameter.

3 FIG. 3 FIG. 3 FIG. 350 350 350 301 301 350 350 350 350 350 350 350 350 301 350 illustrates example process flows that demonstrate various aspects of the present disclosure. A client deviceis illustrated in, and includes at least the components that are encompassed within the box ofthat represents the client device. The client devicecan receive client datavia various components and/or sensors. For example, the client datareceived at the client devicecan include spoken utterances captured in audio data generated via one or more microphones of the client device, typed input captured in textual data generated via a touch-sensitive display of the client deviceor peripheral device in communication with the client device(e.g., a keyboard integral with the client deviceor an external keyboard), touch input based captured in touch data generated via the touch-sensitive display of the client deviceone or more buttons of the client device(e.g., one or more hardware buttons and/or software buttons), gesture input captured in vision data generated via one or more vision components of the client device, and/or any other type of client datathat can be received by the client devicevia other components and/or sensors.

350 350 354 301 301 301 350 350 360 354 360 350 360 3 FIG. Further, the client devicecan include one or more on-device machine learning (ML) models stored in on-device memory of the client device, such as an on-device ML model(s) databaseA. In some implementations, and based on processing the client data, the on-device ML models can be utilized to cause certain fulfillment of the client datato be performed. In additional or alternative implementations, and based on processing the client data, the on-device ML models can be updated locally at the client deviceand/or the on-device ML models can be utilized in generating one or more updates at the client devicefor counterpart global ML models that are stored remotely in remote memory of a remote system(e.g., a remote server or cluster of remote servers or other computing devices), such as in global ML model(s) databaseB. The remote system includes at least the components that are encompassed within the box ofthat represents the remote system. The client deviceand the remote systemare communicatively coupled over one or more networks (e.g., over one or more local area networks (LANs) and/or wide area networks (WANs)) for transmitting and receiving data therebetween.

332 354 301 302 332 301 350 301 350 332 301 302 302 301 As described herein, implementations are directed to improved efficiency of federated learning of ML model(s). Similar to traditional federated learning, on-device ML enginecan process, using one or more on-device ML models stored in the on-device ML model(s) databaseA, the client datato generate predicted output. The one or more on-device ML models utilized by the on-device ML enginemay depend on a type of the client datareceived at the client device. For example, assume the client datacorresponds to audio data generated via one or more microphones of the client device. In this example, the on-device ML enginecan utilize one or more audio-based ML models to process the client datain generating the predicted output, such as an automatic speech recognition (ASR) model, a hotword detection model, a continued conversation model, a voice identification model, a voice activity detection (VAD) model, an endpointing model, and/or other audio-based ML models. Further, the predicted output(and/or predicted values associated therewith, such as probabilities, log likelihoods, binary values, and/or any other predicted values) in this example can be based on the one or more audio-based ML models utilized in processing the client data.

301 350 332 301 302 302 301 301 350 332 301 302 302 301 Also, for example, additionally or alternatively assume the client datacorresponds to vision data generated via one or more vision components of the client device. In this example, the on-device ML enginemay utilize one or more vision-based ML models to process the client datain generating the predicted output, such as an object detection model, an object classification model, a face identification model, and/or other vision-based ML models. Further, the predicted output(and/or predicted values associated therewith, such as probabilities, log likelihoods, binary values, and/or any other predicted values) in this example can be based on the one or more vision-based ML models utilized in processing the client data. Also, for example, additionally or alternatively assume the client datacorresponds to textual data or touch data generated via one or more user interface input components of the client device(or generated based on processing audio data and/or vision data). In this example, the on-device ML enginecan utilize one or more text-based ML models and/or touch-based ML models to process the client datain generating the predicted output, such as natural language understanding (NLU) model, a fulfillment model, and/or other text-based ML models and/or touch-based ML models. Further, the predicted output(and/or predicted values associated therewith, such as probabilities, log likelihoods, binary values, and/or any other predicted values) in this example can be based on the one or more text-based ML models and/or touch-based ML models utilized in processing the client data.

301 332 337 354 337 337 350 301 302 In furtherance of facilitating efficient processing of client databy the on-device ML engine, compression enginecan be utilized to decompress on-device ML model parameters on the fly. The parameter(s) that are decompressed by the compression engine at a given time are a subset of the compressed on-device ML model parameter(s) and can be the next parameter(s) that are needed in performing a propagation. For example, the parameter(s) that are decompressed can include only some parameter(s) of a given ML model layer, all parameter(s) of a given ML model layer, or at least some parameter(s) of a given ML model layer and at least some parameter(s) of an additional given ML model layer. As a particular example, for processing an input to an Nth layer of the on-device ML model, compressed parameters from the on-device ML model databaseA can be decompressed by the compression engine. The compression enginecan allocate memory for the decompressed parameters for a particular layer of a given on-device ML model, process the decompressed parameters with an input to the Nth layer in furtherance of forward propagation, and deallocate the memory for the decompressed parameters. This process can be repeated for each layer of multiple layers of the on-device ML model, thereby preserving memory of the client devicewhen processing client dataand/or performing forward propagation to generate a predicted output.

350 350 350 302 336 336 302 303 304 303 334 334 303 350 301 301 350 332 334 303 350 In various implementations, to cause the on-device ML models to be updated locally at the client deviceand/or the on-device ML models to be utilized in generating the one or more updates at the client devicefor the counterpart global ML models, the client devicecan cause the predicted outputto be provided to a gradient engine. The gradient enginecan compare the predicted outputwith a supervision signalto generate a gradient. In some implementations, the supervision signalcan be an implicit supervision signal generated using an unsupervised learning engine. The unsupervised learning enginecan utilize one or more self-supervised learning techniques and/or semi-supervised learning techniques to generate the supervision signalin instances where a supervision signal is not available based on explicit user feedback (e.g., feedback from a human user of the client deviceand/or feedback from a human annotator of the client data), which is typical in federated learning of ML model(s) due to privacy considerations. For example, assume the client datacorresponds to audio data generated via one or more microphones of the client device, and further assume that the on-device ML engineprocesses the audio data, using an on-device ASR model, to generate and select a given speech hypotheses that is predicted to correspond to a spoken utterance captured in the audio data. In this example, the unsupervised learning enginecan utilize the one or more self-supervised learning techniques and/or semi-supervised learning techniques described herein to generate the supervision signaleven if a human user of the client devicedoes not provide any express feedback with respect accuracy of the given speech hypotheses.

303 301 334 350 301 350 332 350 301 303 334 334 In additional or alternative implementations, the supervision signalcan be an express supervision signal generated based on the client dataand the unsupervised learning enginecan optionally be omitted (e.g., feedback from a human user of the client device). For example, again assume the client datacorresponds to audio data generated via one or more microphones of the client device, and again assume that the on-device ML engineprocesses, using an on-device ASR model, the audio data to generate and select a given speech hypotheses that is predicted to correspond to a spoken utterance captured in the audio data. However, further assume that a human user of the client devicecorrects one or more terms or phrases of the given speech hypotheses. In this example, the correction may be captured in the client dataand utilized as the supervision signalwithout using the unsupervised learning engine. However, in this example, the unsupervised learning enginecan additionally or alternatively utilize the one or more self-supervised learning techniques and/or semi-supervised learning techniques described herein to generate an additional or alternative supervision signal.

304 303 302 303 302 336 303 302 336 303 302 304 304 302 In some implementations, the gradient(and other gradients) can be derived from a loss function used to train the ML model(s) (e.g., the on-device ML model(s) and/or the global ML model(s)), such that the gradient represents a value of that loss function (or a derivative thereof) obtained from a comparison of the supervision signalto the predicted output. For example, when the supervision signaland the predicted outputmatch, the gradient enginecan generate a zero gradient. Also, for example, when the supervision signaland the predicted outputdo not match, the gradient enginecan generate a non-zero gradient that is optionally dependent on the extent of the mismatching. The extent of the mismatching can be based on an extent of mismatching between deterministic comparisons of the supervision signalto the predicted output. In additional or alternative implementations, the gradient(and other gradients) may be derived from a loss function used to train the ML model(s), such that the gradientrepresents a value of that loss function (or a derivative thereof) determined based on the predicted output(e.g., without considering any supervision signals).

350 370 336 304 350 350 304 360 336 304 340 340 306 304 306 306 306 340 304 3 FIG. Traditionally, in federated learning, a plurality of client devices (e.g., the client deviceand one or more of the additional client devices) each generate a corresponding gradient for a corresponding on-device ML model based on corresponding client data received a given one of the plurality of client devices. For example, each of the plurality of client devices can have a corresponding counterpart on-device ML model stored in corresponding on-device memory, and process the corresponding client data to generate gradients for the corresponding instances of the corresponding on-device ML models. In some implementations, the gradient enginecan cause the gradientto be stored in on-device memory of the client devicealong with other gradients generated at the client deviceuntil one or more client device conditions are satisfied (e.g., described with respect to) to cause the gradient(and other gradients) to be transmitted to the remote system. In additional or alternative implementations, the gradient enginecan cause the gradientto be provided to ML update engine. The ML update enginecan generate a ML updatebased on the gradientthat optionally includes an indication of the type of global ML model to be updated based on the ML update(e.g., an ASR update when the ML updateif generated based on a gradient for an ASR model, a hotword update when the ML updateif generated based on a gradient for a hotword model, and so on). In additional or alternative implementations, the ML update enginecan cause the corresponding counterpart on-device ML model to be updated based on the gradient.

340 304 304 339 339 350 339 339 306 360 360 The ML update enginecan cause the gradientto be backpropagated across the one or more on-device ML layers to update one or more corresponding on-device weights, and/or one or more other parameters, of the one or more on-device ML layers. In some implementations, the gradientcan be utilized during back propagation as a basis for updating parameters of the on-device ML model, and this back propagation can involve using decompressed parameters that have been decompressed by a compression engineon the fly. For example, when determining updated parameters for an Nth layer of the on-device ML model, the compression enginecan decompress a subset of compressed ML model parameters, while leaving other ML model parameters compressed and stored at the client device. Memory can be allocated for the decompressed parameters during processing to determine the updated parameters for the Nth layer and then deallocated for the decompressed, in furtherance of allowing other parameters for an Mth layer to be decompressed by the compression engineand buffered into allocated memory. Any updated parameters can be selectively compressed, or left as decompressed, by the compression engineand collectively stored as an ML updatethat can be shared with the remote systemand/or utilized to further refine an ML update that is shared with the remote systemwhen certain optional criteria is met.

304 306 307 In traditional federated learning, the corresponding counterpart global ML model can be updated in a distributed manner in that the corresponding counterpart global ML model is updated based on gradients (e.g., the gradientand/or one or more additional gradients) or updates (e.g., the ML updateand/or one or more of the additional ML updates) generated at a plurality of client devices. However, a considerable amount of computational and network resources may be consumed at each of the plurality of client devices in generating these gradients or updates. Accordingly, by utilizing techniques described herein, both computational and network resources that would otherwise be consumed at each of the plurality of client devices may be conserved by enabling each of the client devices to generate and transmit less data compared to traditional federated learning.

342 306 350 307 354 306 350 307 380 360 306 350 307 342 342 306 350 307 370 The remote update enginecan utilize at least the ML updatereceived from the client deviceand/or one or more of the additional ML updatesto update one or more of the global ML models stored in the global ML model(s) databaseB. In some implementations, the ML updatereceived from the client deviceand/or one or more of the additional ML updatescan be received in a compressed format and stored in an update(s) databasewhen they are received at the remote system. Alternatively, or additionally, the ML updatereceived from the client deviceand/or one or more of the additional ML updatescan be received in a format in which less than all of the parameters are compressed, thereby allowing other parameters that are sensitive to quantization to remain non-compressed. In some versions of those implementations, the remote update enginecan obtain the updates to update one or more of the global ML models when one or more remote system conditions are satisfied. In additional or alternative implementations, the remote update enginemay update one or more of the global ML models as the ML updateis received from the client deviceand/or as one or more of the additional ML updatesis received from one or more of the additional client devices.

342 306 350 307 370 342 354 342 306 307 380 342 342 306 307 Notably, the remote update enginecan update the corresponding global ML models in various manners based on a type of the ML updateis received from the client deviceand/or as one or more of the additional ML updatesis received from one or more of the additional client devices. For example, the remote update enginecan identify a particular global ML model, from among the global ML models stored in the global ML model(s) databaseB, to update global layers and/or global parameters thereof. In some implementations, the remote update enginecan identify the particular global ML model based on the indication of the type of global ML model to be updated based on the ML updateand/or one or more of the additional ML updates. For example, if a plurality of updates for an ASR model are received and/or stored in the update(s) database, the remote update enginecan identify the corresponding counterpart global ASR model to be updated. Further, how the remote update engineupdates the corresponding counterpart global ASR model can be based on content of the ML updateand/or one or more of the additional ML updates.

342 342 342 342 350 370 342 342 In some implementations, the remote update enginecan assign the updates to specific iterations of updating of one or more of the global ML models based on one or more criteria. The one or more criteria can include, for example, the types of updates available to the remote update engine, a threshold quantity of updates available to the remote update engine(and optionally for the disparate portions of the one or more global ML models), a threshold duration of time of required to perform the updates, and/or other criteria. In particular, the remote update enginecan identify multiple sets or subsets of updates generated by the client devicesand/or. Further, the remote update enginecan update one or more of the global ML models based on these sets or subsets of the updates. In some further versions of those implementations, a quantity of updates in the sets or subsets may be the same or vary. In yet further versions of those implementations, each of the sets or subsets of updates can optionally include updates from at least two unique client devices. In other implementations, the remote update enginecan utilize the updates to update one or more of the global ML models in a first in, first out (FIFO) manner without assigning any update to a specific iteration of updating of one or more of the global ML models.

344 308 350 370 344 308 360 350 370 308 350 370 308 308 308 350 370 350 370 350 370 360 360 Update distribution enginecan transmit one or more of the updated global ML models(e.g., including the updated global ML model itself, updated layers thereof, and/or updated weights and/or other parameters thereof) to the client deviceand/or one or more of the additional client devices. In some implementations, the update distribution enginecan transmit one or more of the updated global ML modelsin response to one or more remote system conditions being satisfied at the remote systemand/or one or more client device conditions being satisfied at the client deviceand/or one or more of the additional client device. Notably, one or more of the updated global ML modelscan be transmitted to the client deviceand/or one or more of the additional client devicein a synchronous manner or asynchronous manner. Upon receiving one or more of the updated global ML models, one or more of the updated global ML modelscan replace the corresponding on-device ML model counterpart stored in the on-device memory thereof with one or more of the updated global ML models. Further, the client devicesand/orcan subsequently use one or more of the updated on-device ML model(s) to make predictions based on further client data received at the client deviceand/or one or more of the additional client device. The client deviceand/or one or more of the additional client devicecan continue transmitting the updates to the remote systemin the manner described herein. Further, the remote systemcan continue updating one or more of the global ML models in the manner described herein.

4 FIG. 400 400 illustrates a methodfor performing, on the fly, decompression and compression of model parameters at edge devices during federated learning, and sharing updated compressed parameters with a remote system for updating a corresponding global ML model. The methodcan be performed by one or more client computing devices, applications, and/or any other apparatus or module that can assist with federated learning.

400 402 400 402 404 400 The methodcan include an operationof determining whether a client ML model has been received at a client device. For example, determining whether the client ML model has been received at a client device from a remote server system that is managing a federated learning process. The client ML model can include client ML model data that characterizes various model parameters (e.g., weight matrices, biasing value matrices, etc.). The client ML model includes some compressed parameters. For example, a majority of the ML model parameters can be compressed, 90% or more of the ML model parameters can be compressed, and/or all of the ML model parameters can be compressed. When a client ML model has been received, the methodcan proceed from the operationto an operation, otherwise the methodcan await receipt of a client ML model.

404 400 404 406 400 404 The operationcan include determining whether client data is to be processed using the client ML model. The client data can include, but is not limited to, image data, audio data, video data, and/or any other data that can be captured or otherwise processed by a client device. For example, the client ML model can be an ASR model, and the client data can include audio data that captures a spoken utterance from a user. When the client data is determined to be processed using the client ML model, the methodcan proceed from the operationto an operation. Otherwise, the client data can be processed otherwise and/or the methodcan return to the operation.

406 406 406 406 The operationcan include decompressing a compressed subset of parameters of the client ML model. For example, some or all of the compressed parameters for an initial layer of the client ML model can be decompressed in an initial iteration of operation(without decompressing any other compressed parameters in the initial iteration), the remaining compressed parameters for the initial layer and/or parameter(s) for a next layer can be decompressed in a second iteration of operation(without decompressing any of the compressed parameters in the second iteration), etc. Put another way, at each iteration of operationthe compressed subset of parameters, that are decompressed in that iteration, can be a next subset needed in performing the forward propagation.

In some implementations, the compressed subset of parameters can include values from a weight matrix and/or array, and/or values from a biasing value matrix and/or array. When the subset of parameters is decompressed, other subsets of parameters of the client ML model data can remain compressed (but optionally still in memory, in a compressed format), such as other parameters that may be used to facilitate forward propagation at other layers of the client ML model. Memory can be allocated for the decompressed parameters, and, upon completion of any calculations for the current decompressed parameter(s), the memory can be de-allocated for those decompressed parameters and re-allocated for another subset of to be decompressed parameters.

400 406 408 400 408 410 400 410 412 400 410 406 The methodcan then proceed from the operationto an operation, which can include performing processing using the decompressed subset of parameters. For example, decompressed weight values for a particular layer and non-compressed biasing values for the particular layer can be processed in furtherance of facilitating forward propagation. When processing is completed using the decompressed subset of parameters, the methodcan proceed from the operationto an operation, which can include determining whether forward propagation is complete for all layers of the client ML model. When forward propagation is completed, the methodcan proceed from the operationto an operation. Otherwise, when forward propagation is not complete, the methodcan proceed from the operationto the operation, and another compressed subset of parameters can be decompressed for further facilitating forward propagation.

412 The operationcan include determining any loss based on a model output. For example, a loss can be determined based on comparing the model output to ground truth data (e.g., based on feedback from the user, indication of what was said by the user, etc.). The loss can then be utilized to facilitate back propagation and determine updated parameters for the client ML model, which can then be shared with the remote system or otherwise provide a basis for other data that is shared with the remote system.

400 412 414 412 412 412 400 414 416 For example, the methodcan proceed from the operationto an operation, which can include decompressing a compressed subset of parameters of the client ML model. The subset of parameters can be decompressed for facilitating back propagation, and allocated to memory until back propagation is completed for those parameter(s). For example, some or all of the compressed parameters for a terminal layer of the client ML model can be decompressed in an initial iteration of operation(without decompressing any other compressed parameters in the initial iteration), the remaining compressed parameters for the terminal layer and/or parameter(s) for a preceding layer can be decompressed in a second iteration of operation(without decompressing any of the compressed parameters in the second iteration), etc. Put another way, at each iteration of operationthe compressed subset of parameters, that are decompressed in that iteration, can be a next subset needed in performing the back propagation. When the subset of parameters have been decompressed, the methodcan proceed from the operationto an operation.

416 400 416 418 The operationcan include determining updated parameters based on the loss and corresponding decompressed subset of parameters. The updated parameters can include, but are not limited to, updated parameter(s) of the subset, one or more updated weight values, one or more updated biasing values, and/or any other updated value or function that can be utilized by an ML model. The updated parameters can then be compressed and/or stored in memory until back propagation is completed. For example, the methodcan proceed from the operationto an operation, which can include determining whether back propagation has been completed for all layers and/or for the client ML model.

400 418 420 400 418 414 420 When back propagation is completed, the methodcan proceed from the operationto an operation, otherwise the methodcan proceed from the operationto the operationfor furthering back propagation. The operationcan include providing compressed and/or non-compressed updated parameters to a remote system for updating a global ML model.

Note that some parameters for some layers can be compressed, while some parameters for other layers may remain non-compressed when shared with the remote system. In this way, layers and/or parameters (e.g., biasing values) that may be susceptible to errors during quantization can remain non-compressed, while other layers and/or other parameters that are less susceptible to errors during quantization can be compressed for preserving computational resources such as memory and network bandwidth.

412 414 416 418 420 In some implementations, operations,,,, andcan be omitted. In those implementations, compression and decompression techniques can be utilized herein for forward propagation/at inference, without any utilization for back propagation.

5 FIG. 500 510 510 514 512 524 525 526 520 522 516 510 516 is a block diagramof an example computer system. Computer systemtypically includes at least one processorwhich communicates with a number of peripheral devices via bus subsystem. These peripheral devices may include a storage subsystem, including, for example, a memoryand a file storage subsystem, user interface output devices, user interface input devices, and a network interface subsystem. The input and output devices allow user interaction with computer system. Network interface subsystemprovides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

522 510 User interface input devicesmay include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computer systemor onto a communication network.

520 510 User interface output devicesmay include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computer systemto the user or to another machine or computer system.

524 524 400 350 360 3 FIG. Storage subsystemstores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystemmay include the logic to perform selected aspects of method, and/or to implement one or more of, client device, remote system, and/or any other application, device, apparatus, and/or module discussed herein.

514 525 524 530 532 526 526 524 514 These software modules are generally executed by processoralone or in combination with other processors. Memoryused in the storage subsystemcan include a number of memories including a main random access memory (RAM)for storage of instructions and data during program execution and a read only memory (ROM)in which fixed instructions are stored. A file storage subsystemcan provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystemin the storage subsystem, or in other machines accessible by the processor(s).

512 510 512 Bus subsystemprovides a mechanism for letting the various components and subsystems of computer systemcommunicate with each other as intended. Although bus subsystemis shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

510 510 510 5 FIG. 5 FIG. Computer systemcan be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer systemdepicted inis intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer systemare possible having more or fewer components than the computer system depicted in.

In situations in which the systems described herein collect personal information about users (or as often referred to herein, “participants”), or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed. For example, a user's identity may be treated so that no personal identifiable information can be determined for the user, or a user's geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and/or used.

While several implementations have been described and illustrated herein, a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein may be utilized, and each of such variations and/or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings is/are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.

In some implementations, a method implemented by processor(s) of a client device is provided and includes receiving, from a remote system, a client machine learning (ML) model in a compressed format. The remote system hosts a global ML model that corresponds to the client ML model and the client ML model includes compressed model parameters for processing model input data. The method further includes, for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed: decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation. The method further includes determining, based on the predicted output from the forward propagation, a loss in furtherance of updating the compressed model parameters. The method further includes, for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate back propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed: decompressing the compressed subset of parameters to generate the corresponding decompressed subset of parameters; processing a remaining portion of the loss and the corresponding decompressed subset of parameters to generate corresponding decompressed updated parameters for the corresponding decompressed subset of parameters; compressing the corresponding decompressed updated parameters to generate corresponding updated compressed parameters for the client ML model; and storing the corresponding updated compressed parameters. The method further includes providing, to the remote system from the client device, the corresponding compressed updated parameters for the client ML model. The global ML model is updated by at least decompressing, by the remote system, the corresponding compressed updated parameters from the client device.

In some implementations, decompressing the compressed subset of parameters, for a given one of the compressed subsets, to facilitate forward propagation, includes deallocating client device memory for an additional one of the decompressed subsets utilized in an immediately preceding iteration of the forward propagation.

In some implementations, decompressing the compressed subset of parameters, for the given one of the compressed subsets, to facilitate back propagation, includes deallocating client device memory for a further one of the decompressed subsets utilized in an immediately preceding iteration of the back propagation.

In some implementations, the client ML model includes non-compressed model parameters that are also utilized to facilitate forward propagation and/or back propagation.

In some implementations, the client ML model is utilized by the client device to facilitate automatic speech recognition (ASR) and an initial instance of the corresponding data is based on audio captured by the client device or another computing device.

In some implementations, decompressing the compressed subset of parameters includes applying a corresponding scaling factor and/or a corresponding biasing value to each parameter of the decompressed subset of parameters.

In some implementations, determining the loss is based on comparing the predicted output to a ground truth value.

In some implementations, a method implemented by processor(s) of a client device is provided and includes receiving, from a remote system, a client machine learning (ML) model in a compressed format. The remote system hosts a global ML model that corresponds to the client ML model and the client ML model includes compressed model parameters for processing model input data. The method further includes, for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed: decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation. The method further includes determining, based on the predicted output from the forward propagation, a loss in furtherance of updating the compressed model parameters. The method further includes providing, to the remote system, compressed updated parameters that are for the client ML model and that are generated based on the loss. Providing the compressed updated parameters to the remote system causes the remote system to update the global ML model based on a decompression of the compressed updated parameters.

In some implementations, a method implemented by processor(s) of a client device is provided and includes receiving, from a remote system, a client machine learning (ML) model in a compressed format. The method further includes, for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed: decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation. The method further includes performing one or more actions based on the predicted output from the forward propagation.

In some implementations, decompressing the compressed subset of parameters, for a given one of the compressed subsets, to facilitate forward propagation, includes deallocating client device memory for an additional one of the decompressed subsets utilized in an immediately preceding iteration of the forward propagation.

In some implementations, the client ML model, received from the remote system, includes non-compressed model parameters that are also utilized to facilitate forward propagation. In some of those implementations, the non-compressed model parameters include a plurality of bias parameters, and the compressed model parameters include a plurality of weight parameters.

In some implementations, the client ML model is utilized by the client device to facilitate automatic speech recognition (ASR) and an initial instance of the corresponding data is based on audio captured by the client device or another computing device. In some of those implementations, the one or more actions include: rendering a transcription, of the audio data, determined based on the predicted output and/or controlling one or more applications based on the transcription determined based on the predicted output.

In some implementations, decompressing the compressed subset of parameters includes applying a corresponding scaling factor and/or a corresponding biasing value to each parameter of the decompressed subset of parameters.