Fast Quantization for Neural Networks

Machine learning (ML) models, which include artificial intelligence (AI) models and techniques, operate on numerical representations of data such as text, audio, and images. A typical numerical representation is an embedding such as a vector embedding that encodes features of the data in a series (or vector) of numerical values. An ML model is trained to transform the numerical representation of the input data into one or more output values such as a value indicating the identity of a feature in the data, a classification of the feature, or a value that is predicted based upon the input data. Neural networks are often used to implement ML models. A typical neural network includes an input layer, one or more hidden layers, and an output layer formed of interconnected sets of nodes. The nodes are connected so that the output of nodes in one layer becomes the input to nodes in a subsequent layer. As data is passed from one node to the next, the data is multiplied by a weight associated with the connection between the two nodes. The weight therefore represents the strength of the connection between the two nodes. Nodes combine the weighted inputs, typically as a linear combination, to determine a value that is provided to an activation function that determines whether the node should “activate.” Examples of activation functions include a sigmoid function, a rectified linear unit (ReLU) function, a softmax function, and a hyperbolic tangent function. Training the ML model includes, for example, determining values of the weights and activation functions based on training data.

The weights and activation functions that represent an ML model are typically represented as floating point numbers in a format such as FP32 or FP16. However, the memory requirements and computational complexity of the neural network can become intractably large as the size of the neural network increases. Quantization therefore can be used to reduce the precision of the weights and activation functions from the floating-point values to a smaller representation such as fixed point, integer formats INT8 or INT4. Quantization can be performed by introducing a quantization algorithm into the training process for the ML model (e.g., quantization aware training, QAT) or applying quantization to a previously trained ML model (e.g., post training quantization PTQ, which can also include fine-tuning). Both elements of the model quantization process—the ML model and the quantizer software—change relatively frequently. The ML model can change to optimize or increase the floating-point accuracy; the quantizer software can change to support new enhancements, optimizations, or bug fixes. A conventional ML model can be re-quantized in response to a change in either the ML model or the quantizer software. In many cases, the re-quantization process incurs significant overhead that can impact the time to market for the application.

1 6 FIGS.- illustrate systems, apparatuses, and methods of reducing the overhead for re-quantizing an ML model by selectively reusing quantization parameters for a first portion (or subgraph) of a previously quantized ML model and re-quantizing a separate second portion (or subgraph) of the ML model. The previously quantized ML model is generated by quantizing a previous version of the ML model using a previous version of the quantization algorithm. As discussed herein, re-quantization can be performed in response to a change in the previous version of the ML model, a change in the previous version of the quantization algorithm, or both. Changes in the ML models are detected by identifying mismatches between corresponding layers in the previous version of the ML model and the (current) ML model. As used herein, two layers “match” when the two layers implement the same functionality and produce the same output in response to the same input. For example, a layer in a previous version of the ML model that implements 4×4 convolution matches a layer in the current ML model that also implements 4×4 convolution. There is a “mismatch” between two layers if they implement different functionality that produces different output in response to the same input. For example, there is a mismatch between a layer in a previous version of the ML model that implements 4×4 convolution and a layer in the current ML model that implements pooling. For another example, there is a mismatch between a layer in the previous version of the ML model that implements 4×4 convolution and a layer in the current ML model that implements 1×1 convolution.

In some embodiments, a processing unit (such as a parallel processor or graphics processing unit, GPU) detects a change in the ML model based on a comparison of topologies of a current ML model and a previously quantized ML model. For example, the processing unit can compare a first topology of a first graph representing the current ML model to a second topology of a second graph representing the previously quantized ML model. The comparison proceeds layer-by-layer beginning at a first layer (such as the input layer), and the processing unit bypasses the quantization/dequantization layers in the previously quantized ML model. If the processing unit detects a mismatch between corresponding layers of the first topology and the second topology, the processing unit determines a percentage of layers of the first graph and the second graph that matched. If the percentage is less than a threshold (such as 40%), then the entire ML model is re-quantized. Otherwise, if the percentage is greater than the threshold, the first graph (of the previously quantized ML model) is partitioned into a first subgraph that includes the matching portion and a second subgraph that includes the remaining, mismatched portion. The processing unit generates a second (re-quantized) subgraph by re-quantizing the mismatched portion based on the corresponding portion of the ML model and calibration data associated with a corresponding portion. The first subgraph and the second sub graph are then combined to form an updated quantized ML model. In some cases, the re-quantization is performed in response to changes in characteristics of the quantization technique, which can be indicated in metadata associated with the quantization technique.

1 FIG. 1 FIG. 100 100 102 100 100 104 105 100 104 102 104 100 102 is a block diagram of a processing systemthat updates quantized ML models by combining portions of previously quantized versions of an ML model with re-quantized portions of the ML model, according to some embodiments. The processing systemincludes a busimplemented with circuitry that supports communication between entities implemented in the processing system. Some implementations of the processing systeminclude other buses, bridges, switches, routers, and the like, which are not shown inin the interest of clarity. An input/output (I/O) engineis implemented with circuitry that handles input or output operations associated with display, as well as other elements of the processing systemsuch as keyboards, mice, printers, external disks, and the like. The I/O engineis coupled to the busso that the I/O enginecan communicate with other entities in the processing systemby exchanging signals over the bus.

100 106 106 106 100 106 108 110 108 112 108 Processing systemalso includes or has access to a memoryor other storage component implemented using a non-transitory computer-readable medium such as a dynamic random-access memory (DRAM). However, some embodiments of the memoryare implemented using other types of memory including, for example, static random-access memory (SRAM), nonvolatile RAM, and the like. Some embodiments of the memoryinclude an external memory implemented external to the processing units implemented in the processing system. The memorycan store information representing instructions such as program codefor one or more applications (e.g., graphics applications, compute applications, ML models or applications), datathat is consumed by the program code, and resultsproduced by executing the program code.

100 114 102 100 104 106 102 114 116 116 1 116 2 116 116 116 114 116 114 108 106 114 110 106 112 1 FIG. The processing systemincludes a central processing unit (CPU)that is connected to the busto communicate with other entities in the processing system, such as the I/O engine, the memory, or other entities connected to the bus. The CPUis implemented with circuitry including one or more processor cores, such as the illustrated plurality of processor cores-,-, . . .-M that execute instructions concurrently or in parallel. Although three processor coresare shown in, more or fewer processor corescan be implemented in other embodiments of the CPU. The processor coresinclude circuitry to implement one or more compute units such as single-instruction, multiple data (SIMD) units that perform the same operation on different data sets to produce one or more results. The CPUis configured to execute instructions such as the program codefor one or more applications (e.g., graphics applications, compute applications, machine learning applications), which is stored in the memory. The CPUcan consume dataand store information in the memorysuch as the resultsof the executed instructions.

100 118 118 118 120 1 118 120 120 118 120 1 FIG. The processing systemalso includes a parallel processor. The parallel processorcan include, for example, a GPU, a general-purpose GPU (GPGPU), a neural processing unit (NPU), an intelligence processing unit (IPU) or other vector processor or type of parallel processor. The parallel processorincludes circuitry to implement one or more processor cores-. . . N that each operate as a compute unit configured to perform one or more operations based on one or more instructions received by the parallel processor. Although three processor coresare shown in, more or fewer processor corescan be implemented in other embodiments of the parallel processor. The compute units in the processor coresare implemented as circuitry for one or more single-instruction, multiple data (SIMD) units that perform the same operation on different data sets to produce one or more results.

118 122 120 122 106 108 122 110 122 122 118 120 120 122 118 1 FIG. In the illustrated embodiment, the parallel processorincludes circuitry for storing data such as one or more high bandwidth memories (HBM)that are used to store instructions or data that are accessed by one or more of the processor coresvia high bandwidth connections (not shown inin the interest of clarity). Some embodiments of the HBMstore instructions or information that has been retrieved or fetched from memory. For example, instructions can be fetched from the program codeand stored in HBMand data can be fetched from the dataand stored in the HBM. Although the HBMis shown internal to the parallel processorand external to the processor cores, some embodiments of the processor coresinclude additional, internal HBM or caches and some embodiments of the HBMare implemented external to the parallel processor.

118 120 118 106 106 122 118 118 118 124 124 124 The parallel processoris configured to implement selective updating of an ML model by re-quantizing portions of the ML model based on a comparison of a graph of the ML model with the graph of a quantized representation of a previous version of the ML model. One or more of the processor coresof the parallel processorare configured to compare a first graph that represents a machine learning (ML) model and a second graph that represents a previously quantized ML model generated by quantizing a previous version of the ML model. In some embodiments, information representing the ML model, the previously quantized ML model, calibration data, and metadata indicating characteristics of the quantization technique used to generate the previously quantized ML model are stored in a memory such as the memory. Portions of the stored information can be fetched from the memoryinto the HBM. Based on the comparison, the parallel processorcan selectively reuse a first subset of quantization parameters from the previously quantized ML model and re-quantize a second portion of the ML model to form a second subset of the quantization parameters. The first subset was previously formed by quantizing a first portion of the previous version of the ML model. The parallel processorcan then combine the first subset and the second subset to form an updated quantized ML model. In some embodiments, the parallel processorprovides information representing the updated second ML model to a devicethat is configured to perform inference using the updated second ML model based on input values such as a feature map. For example, the devicecan be configured to generate a value indicating on or more of an identity of a feature in the feature map, a classification of a feature in the feature map, or a value that is predicted based upon the feature map. The devicecan be a smart phone, an Internet of Things (IoT) device, a neural processing unit (NPU), or other low power device.

2 FIG. 200 202 200 200 200 202 200 illustrates an ML modelthat uses a floating-point representation and a quantized ML modelthat uses a fixed point representation formed by quantizing the ML model, according to some embodiments. The floating-point representation of the modelcan use representations having different precisions such as an FP32 representation that uses 32 bits or an FP16 representation that uses 16 bits to represent the floating-point numbers such as the weights or activation functions of nodes in the ML model. The fixed-point representation of the quantized ML modelcan also have different precisions such as INT8 that uses eight bits or INT4 that uses four bits to represent the fixed point integers that indicate quantized values of the weights or activation functions of nodes in the ML model.

200 204 200 200 206 206 208 206 208 208 210 204 200 200 In the illustrated embodiment, the ML modelincludes an input layermade up of one or more nodes that receive input values of features provided to the ML modelfor training or inference. The ML modelalso includes a convolution layerthat convolves the input values with a filter associated with the convolution layer. A ReLU layerrepresents an activation function that is applied to values output from the convolution layer. The activation function represented by the ReLU layeris a piecewise linear function that outputs the input directly if it is positive; otherwise, it outputs zero. Values generated by the ReLU layerare then provided to an output layer. In some embodiments, the input layerreceives data such as a tensor representing a feature map and the ML modeluses this information to classify one or more features in the received feature map. For example, the ML modelcan perform computer vision tasks such as image classification, object detection, object tracking, and content-based image retrieval.

202 200 202 200 202 212 214 216 218 220 222 200 202 212 214 216 218 220 222 200 202 212 214 216 218 220 222 The quantized ML modelis formed by applying a quantization technique or algorithm to the ML model. The quantized ML modelhas the same topology as the ML modelexcept for additional nodes or layers associated with the quantization or de-quantization technique. In the illustrated embodiment, the quantized ML modelincludes quantize layers,and dequantize layers,,,. The (first) topology of the ML modelis therefore the same as a (second) topology of the quantized ML modelas long as the additional quantize layers,and dequantize layers,,,are bypassed or ignored. A comparison of the topologies of the ML modeland the quantized ML modelare therefore performed after removing, bypassing, or ignoring the quantize layers,and dequantize layers,,,.

3 FIG. 2 FIG. 2 FIG. 300 302 300 300 200 302 200 300 300 200 300 300 302 204 206 208 210 300 302 304 illustrates an updated version of an ML modelthat uses a floating-point representation and a quantized ML modelthat uses a fixed-point representation formed by quantizing the ML model, according to some embodiments. The ML modelrepresents a newer version of the ML modelshown inand the quantized ML modelrepresents a newer quantized version of the ML modelshown in. The ML modelcan therefore be referred to as new, later, updated, modified, changed, or other terms to indicate that the ML modelis different than the ML model, which can be referred to as a previous, earlier, older, or prior version of the ML model. The ML modeland the one post ML modelinclude the input layer, the convolution layer, the ReLU layer, and the output. However, the ML modeland the quantized ML modelalso include a pooling layerthat is configured to downsample, aggregate, or otherwise combine information that is dispersed among many vectors into fewer vectors or scalars.

302 300 202 200 302 300 302 306 308 310 312 314 316 300 302 306 308 310 312 314 316 2 FIG. In a conventional system, the quantized ML modelis formed by applying a quantization technique or algorithm to the ML model. As discussed herein, the quantization technique can be the same or different than the quantization technique used to generate the quantized ML modelbased on the ML modelshown in. The quantized ML modelhas the same topology as the ML modelexcept for additional nodes or layers associated with the quantization or de-quantization technique. In the illustrated embodiment, the quantized ML modelincludes quantize layers,and dequantize layers,,,. The topology of the ML modelis therefore the same as the topology of the quantized ML modelas long as the additional quantize layers,and dequantize layers,,,are bypassed or ignored.

302 300 100 202 302 202 300 212 214 216 218 220 222 202 202 300 304 300 212 216 218 220 306 310 312 314 304 308 316 214 222 1 FIG. 2 FIG. However, as discussed herein, generating the quantized ML modelby re-quantizing the entire ML modelcan be prohibitively resource intensive. Systems such as the processing systemshown inare therefore configured to compare a previously quantized ML model (e.g., the quantized ML model) to the current or updated ML modelprior to applying the quantization technique. Based on the comparison, the processing system can selectively reuse portions of the previously quantized ML modeland re-quantize complementary portions of the updated ML model. In the illustrated embodiment, the processing system performs the comparison by removing, ignoring, or bypassing the quantize layers,and dequantize layers,,,in the quantized ML modeland then comparing the topologies of the quantized ML modeland the updated ML model. The two topologies differ because of the additional pooling layerin the updated ML model. Thus, quantize layersand dequantize layers,,can be reused so that they are equal to the corresponding quantize layersand dequantize layers,,. Re-quantization is performed from the pooling layerdownstream so that the quantize layerand the dequantize layerare different than the quantize layerand the dequantize layerin.

302 302 302 302 Some embodiments of the quantized ML modelare trained to perform tasks in response to receiving input data such as a feature map. For example, a feature map can be provided to the quantized ML model. Based on the quantized ML model, a value indicating an identity of a feature in the feature map, a classification of a feature in the feature map, a value that is predicted based upon the feature map, or other characteristic or result and be generated by the quantized ML model.

4 FIG. 4 FIG. 400 400 402 404 406 408 410 412 414 416 418 420 400 400 402 412 422 402 404 406 410 412 416 400 408 414 418 400 illustrates layers of an ML modelthat are selectively re-quantized using a quantization algorithm to convert a floating-point representation of weights and activation functions in the layers to a fixed-point representation, according to some embodiments. The ML modelincludes an input layer, convolution layers,,, pooling layers,,, ReLU layers,, and output layer. The topology of the ML modelis compared to a topology of a quantized version of a previous version of the ML model (not shown inin the interest of clarity) to determine whether the topology of the ML model has changed. In some embodiments, the topologies are compared by comparing a first graph that represents the ML modeland a second graph that represents the previously quantized version. The comparison is performed layer-by-layer beginning at a first layer such as the input layer. In the illustrated embodiment, a mismatch between the first graph and the second graph is detected after the pooling layer, as indicated by the dashed line. Thus, quantization parameters that represent the input layer, the convolution layers,, the pooling layers,, and the ReLU layercan be reused in the quantized version of the ML model. Weights and activation functions in the convolution layer, the pooling layer, and the ReLU layerare re-quantized and then combined with the reused parameters to form the quantized version of the ML model.

5 FIG. 1 FIG. 500 501 501 118 501 502 502 504 504 502 illustrates the first portionof a methodfor selectively reusing previously determined quantization parameters for an ML model and re-quantizing remaining quantization parameters for the ML model, according to some embodiments. The methodis implemented by a processing system or a processing unit such as some embodiments of the parallel processorshown in. To implement the method, the processing unit accesses information representing a pre-trained ML model, which can be referred to as “the current ML model,” and information representing a reference quantized model, which can be referred to as “the previously quantized ML model” because the reference quantized modeland represent a quantized version of a previous version of the current ML model.

502 504 500 501 The processing unit implements a predetermined quantization technique or algorithm to quantize parameters of ML models. As discussed herein, the quantization technique or algorithm can change following quantization of a previous version of the ML model, in which case re-quantizing the entire ML model could be preferable. Consequently, the quantization technique that the processing unit uses to re-quantize the current ML modelcan differ from the quantization technique that was used to quantize a previous version of the ML model to generate the reference quantized model. Some embodiments of the processing unit therefore access metadata associated with the quantization technique prior to initiating the first portionof the method. The metadata can indicate parameters used during quantization such as the calibration method, quantization type, whether fast finetune is used, and other related settings.

504 500 501 506 500 501 506 502 501 502 The processing unit can compare the settings for the current quantization technique to the settings or the quantization technique that was used to generate the reference quantized model. In response to determining that the settings are the same, a first portionof the methodproceeds to the block. In response to determining that there are minor changes or differences between the two sets of settings, the first portionof the methodproceeds to blockafter providing a warning that the changes or differences in the settings will be ignored. In some embodiments, the processing unit can provide an option for a user to override this decision and force the processing unit to perform re-quantization of the entire ML model. Examples of minor changes include changes to the calibration method, the version of operators, and the like. In response to determining that there are major changes or differences between the two sets of settings, the methodends, and the processing unit reverts to performing re-quantization of the entire ML model. Examples of major changes include data type changes, quantization strategy changes such as changing to a weights-only quantization, quantization scheme changes such as changing from per-tensor to per-channel quantization, symmetric changes, changes to the quantization algorithm or parameters of the quantization algorithm, and the like.

506 502 504 504 502 504 2 FIG. 3 FIG. At block, the processing unit compares a first graph that represents the current ML modeland a second graph that represents the reference quantized model. As discussed herein with regard toand, the processing unit bypasses or ignores quantization or dequantization layers or nodes in the reference quantize modelwhen performing the comparison between the first graph and the second graph. In some embodiments, the processing unit compares the first graph and the second graph layer-by-layer beginning at a first layer such as an input layer of the ML modeland the previously quantized ML model.

508 502 504 200 202 300 302 508 501 510 504 502 504 508 501 512 2 FIG. 3 FIG. At decision block, a processing unit determines whether there is a mismatch between any layer or node in the ML modeland the previously quantized ML model. For example, there is no mismatch between the ML modeland the quantized ML modelshown in. However, there is a mismatch between the ML modeland the quantized ML modelshown in. If there is no mismatch detected at decision block, methodflows to the blockand the reference quantized modelis reused to represent a quantized version of the ML model, in which case no re-quantization is necessary. Some embodiments, the reference quantized modelis sent to the next pass in the overall quantization process to perform optimizations and the like. If the processing unit detects a mismatch at decision block, the methodflows to the block.

512 502 504 502 504 At block, the processing unit determines how much of the ML modeland the previously quantized ML modelmatch. In the illustrated embodiment, the processing unit determines a percentage of the nodes or layers in the ML modeland the previously quantized ML modelthat match, beginning from a first layer (such as the input layer) and proceeding layer-by-layer until a mismatch is detected.

514 502 504 502 504 501 516 502 502 504 501 518 At decision block, a processing unit determines whether the percentage match between the ML modeland the previously quantized ML modelis greater than a threshold value. For example, the processing unit can determine whether the percentage match is greater or less than 40%. If the percentage match is less than the threshold, which indicates that a relatively large percentage or fraction of the ML modelshould be re-quantized and a relatively small percentage or fraction of the previously quantized ML modelcan be reused, the methodflows to the blockand the processing unit reverts to the default flow so that the entire ML modelis re-quantized. If the percentage match is greater than the threshold, which indicates that a relatively small percentage or fraction of the ML modelshould be re-quantized and a relatively large percentage or fraction of the previously quantized ML modeland the reused, the methodflows to the block.

518 502 200 300 502 504 502 501 1 500 600 2 FIG. 3 FIG. 6 FIG. At block, a processing unit partitions the graph that represents the ML model. Examples of graphs of ML models are the graph of the ML modelshown inand the graph of the ML modelshown in. Partitioning the graph creates two subgraphs: a first subgraph that represents nodes or layers of the ML modelthat can be reused (i.e., the corresponding nodes or layers of the previously quantized ML modelare used to represent the updated version of the quantized ML model) and a second subgraph that represents nodes or layers of the ML modelthat are to be re-quantized to generate the updated version of the quantized ML model. The methodthen flows to the node, which connects the first portionto a second portionthat is shown in.

6 FIG. 1 FIG. 5 FIG. 5 FIG. 600 501 600 118 600 501 1 602 604 602 504 600 501 606 502 608 606 604 606 illustrates the second portionof the methodfor selectively reusing previously determined quantization parameters for an ML model and re-quantizing remaining quantization parameters for the ML model, according to some embodiments. The second portionis implemented by a processing system or a processing unit such as some embodiments of the parallel processorshown in. In the illustrated embodiment, a second portionof the methodreceives (at the node) information representing the reuse subgraph(e.g., the first subgraph described in) and the re-quantize subgraph(e.g., the second subgraph described in). The reuse subgraphis used to identify a portion of the previously quantized ML modelthat can be reused to form the updated version of the quantized ML model. The second portionof the methodalso receives or accesses calibration dataassociated with the ML model. A subsetof the calibration datacorresponding to the re-quantize subgraphis extracted or accessed from the calibration data.

610 604 608 606 At block, the processing unit performs a calibration of the re-quantize subgraphbased on the subsetof the calibration data.

612 604 610 602 604 614 604 614 504 At block, the processing unit quantizes the re-quantize subgraphusing the calibrated information provided by the block. In some embodiments, the processing unit taps the outputs from the reuse subgraphand uses these to quantize the re-quantize subgraphusing a default quantization technique or algorithm. The resulting quantize subgraphrepresents quantized parameters (e.g., weights and activation functions) for the nodes or layers in the re-quantize subgraph. The parameters in the quantized subgraphare used to replace parameters in the corresponding portion of the previously quantized ML model.

616 504 602 614 604 502 At block, the processing unit accesses the portion of the previously quantized ML modelcorresponding to the reuse subgraph. The processing unit also accesses the quantize subgraphcorresponding to the re-quantize subgraph. The processing unit then combines or merges the quantized subgraphs to form the updated quantized ML model that represents a quantized version of the ML model.

Selectively re-quantizing portions of an ML model has several advantages over the conventional practice of re-quantizing the entire ML model. In some cases, a runtime improvement of a factor of 10 or more is achieved by reusing portions of previously quantized ML models instead of re-quantizing these portions. If a subgraph that extends from input to the longest comparison point the graph can be reused, the processing system only needs to operate on the remaining subgraph, which is significantly faster relative to the conventional re-quantization technique. Furthermore, additional incremental optimizations can be enabled if there are fewer modifications to the model, e.g., because only a subset of the nodes or layers are being re-quantized. In addition to long runtimes, large ML models place correspondingly large commands on the available memory resources. Some embodiments of the techniques disclosed herein reduce consumption of memory resources by factors corresponding to the reuse percentage.

In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software can include instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

Note that not all the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.