A method and system for decoding speech based on recorded brain signals is provided. The method can include receiving recorded brain signals via a microelectrode array. The method can include extracting one or more features from the recorded brain signals. The method can include converting the one or more extracted features into one or more feature embeddings. The method can include transforming, by one or more encoders, the one or more feature embeddings. The method can include predicting, by one or more decoders, phonemes based on the one or more transformed feature embeddings. The method can include predicting speech based on the predicted phonemes.
Legal claims defining the scope of protection, as filed with the USPTO.
receiving recorded brain signals via a microelectrode array, the microelectrode array comprising non-penetrating cortical surface microelectrodes; extracting spatiotemporal features from the recorded brain signals; transforming, using a neural foundation model, the spatiotemporal features to generate one or more transformed feature embeddings, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with at least one of speech or motor tasks; and predicting speech or motor tasks based on the one or more transformed feature embeddings. . A method comprising:
claim 1 . The method of, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with a plurality of tasks.
claim 1 receiving one or more subject-specific embeddings at the encoder; transforming, by the encoder, the one or more subject-specific embeddings; and applying a weighted sum based on the transformed feature embeddings and the transformed subject-specific embeddings. . The method of, wherein the neural foundation model comprises an encoder, the method further comprising:
claim 3 concatenating the encoded representation of the transformed feature embeddings and the subject-specific embeddings with a user embedding representing a profile of the user. . The method of, wherein the weighted sum outputs an encoded representation of the transformed feature embeddings and the subject-specific embeddings, the method further comprising:
claim 1 dividing the plurality of frames into a plurality of patches; wherein the spatiotemporal features are extracted from the plurality of patches. . The method of, wherein the recorded brain signals comprise a plurality of frames ordered temporally in a three-dimensional array, the frames comprising recorded brain signals at different time steps, the method further comprising:
claim 1 tuning the encoder by freezing one or more parameters of the encoder and training one or more layers of the encoder. . The method of, wherein the neural foundation model comprises an encoder, the method further comprising:
claim 1 aligning, by a beam search decoder, predicted phonemes based on one or more language constraints; and generating a predicted transcription based on the aligning. . The method of, wherein predicting speech comprises:
a microelectrode array comprising non-penetrating cortical surface microelectrodes; and a processor; and receive recorded brain signals via the microelectrode array, the microelectrode array comprising non-penetrating cortical surface microelectrodes; extract spatiotemporal features from the recorded brain signals; transform, using a neural foundation model, the spatiotemporal features to generate one or more transformed feature embeddings, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with at least one of speech or motor tasks; and predict speech or motor tasks based on the one or more transformed feature embeddings. a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the computer system to: a computer system communicably coupled to the microelectrode array, the computer system comprising: . A system comprising:
claim 8 . The system of, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with a plurality of tasks.
claim 8 the neural foundation model comprises an encoder; and receive one or more subject-specific embeddings at the encoder, transform, by the encoder, the one or more subject-specific embeddings, and apply a weighted sum based on the transformed feature embeddings and the transformed subject-specific embeddings. the memory stores further instructions that, when executed by the processor, cause the computer system to: . The system of, wherein:
claim 10 the weighted sum outputs an encoded representation of the transformed feature embeddings and the subject-specific embeddings; and concatenate the encoded representation of the transformed feature embeddings and the subject-specific embeddings with a user embedding representing a profile of the user. the memory stores further instructions that, when executed by the processor, cause the computer system to: . The system of, wherein:
claim 8 the recorded brain signals comprise a plurality of frames ordered temporally in a three-dimensional array, the frames comprising recorded brain signals at different time steps; and divide the plurality of frames into a plurality of patches, wherein the one or more features are extracted from the plurality of patches. the memory stores further instructions that, when executed by the processor, cause the computer system to: . The system of, wherein:
claim 8 the neural foundation model comprises an encoder; and tune the encoder by freezing one or more parameters of the encoder and training one or more layers of the encoder. the memory stores further instructions that, when executed by the processor, cause the computer system to: . The system of, wherein:
claim 8 aligning, by a beam search decoder, predicted phonemes based on one or more language constraints; and generating a predicted transcription based on the aligning. . The system of, wherein the memory stores further instructions that, when executed by the processor, cause the computer system to predicting speech by:
Complete technical specification and implementation details from the patent document.
This application is a Continuation of U.S. application Ser. No. 19/062,356, filed Feb. 25, 2025, the entire contents of which are incorporated herein by reference.
Brain-computer interfaces (BCIs) have shown promise as systems for restoring, replacing, and augmenting lost or impaired neurological function in a variety of contexts, including paralysis from stroke and spinal cord injury, blindness, and some forms of cognitive impairment. Multiple innovations over the past several decades have contributed to the potential of these neural interfaces, including advances in the areas of applied neuroscience and multichannel electrophysiology, mathematical and computational approaches to neural decoding, power-efficient custom electronics and the development of application-specific integrated circuits, as well as materials science and device packaging. Nevertheless, the practical impact of such systems remains limited, with only a small number of subjects worldwide having received highly customized interfaces through clinical trials.
High bandwidth brain-computer interfaces are being developed to enable the bidirectional communication between the nervous system and external computer systems in order to assist, augment, or replace neurological function lost to disease or injury. A brain-computer interface should be able to accurately decode electrophysiologic signals recorded from individual neurons, or populations of neurons, and correlate such activity with one or more sensory stimuli or intended motor response. For example, such a system may record activity from the primary motor cortex in an animal or a paralyzed human subject and attempt to predict the actual or intended movement in a specific body part.
Prior attempts to utilize brain-computer interfaces for speech decoding have been limited due to the low signal-to-noise ratio (SNR) and the low resolution of the recorded neural signals. Prior attempts have required guessing which signal to follow from noisy data prior to trying to decode the signal. Due to these limitations, decoding speech based on neural recordings captured using neural interfaces has been extremely challenging.
The present disclosure is directed to systems and methods for utilizing neural foundation models (NFMs) to facilitate communication using brain-computer interfaces. NFMs are a category of deep learning models that have been trained on recordings captured from a BCI device across a variety of tasks, users, and sessions. NFMs captures both a low and high-level representation of the brain recordings which we have called neural embeddings. Neural embeddings can be adopted for several BCI use cases across subjects through fine-tuning a smaller model using only a few examples making them more data efficient and generalizable.
The present disclosure is directed to a method comprising: receiving recorded brain signals via a microelectrode array, the microelectrode array comprising non-penetrating cortical surface microelectrodes; extracting spatiotemporal features from the recorded brain signals; transforming, using a neural foundation model, the spatiotemporal features to generate one or more transformed feature embeddings, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with at least one of speech or motor tasks; and predicting speech or motor tasks based on the one or more transformed feature embeddings.
In some embodiments, the neural foundation model has been trained to associate spatiotemporal features from brain signals with a plurality of tasks.
In some embodiments, the neural foundation model comprises an encoder, and the method further comprises: receiving one or more subject-specific embeddings at the encoder; transforming, by the encoder, the one or more subject-specific embeddings; and applying a weighted sum based on the transformed feature embeddings and the transformed subject-specific embeddings.
In some embodiments, the weighted sum outputs an encoded representation of the transformed feature embeddings and the subject-specific embeddings, and the method further comprises: concatenating the encoded representation of the transformed feature embeddings and the subject-specific embeddings with a user embedding representing a profile of the user.
In some embodiments, the recorded brain signals comprise a plurality of frames ordered temporally in a three-dimensional array, the frames comprising recorded brain signals at different time steps, and the method further comprises: dividing the plurality of frames into a plurality of patches; wherein the spatiotemporal features are extracted from the plurality of patches.
In some embodiments, the neural foundation model comprises an encoder, and the method further comprises: tuning the encoder by freezing one or more parameters of the encoder and training one or more layers of the encoder.
In some embodiments, predicting speech comprises: aligning, by a beam search decoder, predicted phonemes based on one or more language constraints; and generating a predicted transcription based on the aligning.
The present disclosure is directed to a system comprising: a microelectrode array comprising non-penetrating cortical surface microelectrodes; and a computer system communicably coupled to the microelectrode array, the computer system comprising: a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the computer system to: receive recorded brain signals via the microelectrode array, the microelectrode array comprising non-penetrating cortical surface microelectrodes; extract spatiotemporal features from the recorded brain signals; transform, using a neural foundation model, the spatiotemporal features to generate one or more transformed feature embeddings, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with at least one of speech or motor tasks; and predict speech or motor tasks based on the one or more transformed feature embeddings.
The present disclosure is directed to a computer system communicably connected to a microelectrode array comprising non-penetrating cortical surface microelectrodes, the computer system comprising: a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the computer system to: receive recorded brain signals via the microelectrode array, the microelectrode array comprising non-penetrating cortical surface microelectrodes; extract spatiotemporal features from the recorded brain signals; transform, using a neural foundation model, the spatiotemporal features to generate one or more transformed feature embeddings, wherein the neural foundation model has been trained to associate spatiotemporal features from brain signals with at least one of speech or motor tasks; and predict speech or motor tasks based on the one or more transformed feature embeddings.
In some embodiments, the neural foundation model has been trained to associate spatiotemporal features from brain signals with a plurality of tasks.
In some embodiments, the neural foundation model comprises an encoder; and the memory stores further instructions that, when executed by the processor, cause the computer system to: receive one or more subject-specific embeddings at the encoder, transform, by the encoder, the one or more subject-specific embeddings, and apply a weighted sum based on the transformed feature embeddings and the transformed subject-specific embeddings.
In some embodiments, the weighted sum outputs an encoded representation of the transformed feature embeddings and the subject-specific embeddings; and the memory stores further instructions that, when executed by the processor, cause the computer system to: concatenate the encoded representation of the transformed feature embeddings and the subject-specific embeddings with a user embedding representing a profile of the user.
In some embodiments, the recorded brain signals comprise a plurality of frames ordered temporally in a three-dimensional array, the frames comprising recorded brain signals at different time steps; and the memory stores further instructions that, when executed by the processor, cause the computer system to: divide the plurality of frames into a plurality of patches, wherein the one or more features are extracted from the plurality of patches.
In some embodiments, the neural foundation model comprises an encoder; and the memory stores further instructions that, when executed by the processor, cause the computer system to: tune the encoder by freezing one or more parameters of the encoder and training one or more layers of the encoder.
In some embodiments, the memory stores further instructions that, when executed by the processor, cause the computer system to predicting speech by: aligning, by a beam search decoder, predicted phonemes based on one or more language constraints; and generating a predicted transcription based on the aligning.
The present disclosure is generally directed to systems and methods for enabling people who cannot speak to communicate. In particular, the present disclosure is directed to using a NFM to decode speech based on brain signals recorded via a brain-computer interface.
Disclosed herein are a system and methods for decoding speech from brain signals utilizing NFMs. In conventional BCI systems, translating brain signals to intuitive communications is hindered by variability in array placement (array refers to electrodes responsible for capturing brain signals), variability in brain signals, low SNR, and subject-specific differences due to the anatomy of the brain, age, underlying conditions, gender, etc. Through harnessing the computational and linguistic capabilities of NFMs, BCIs can decode and generate natural language from brain signals in real-time, thereby facilitating seamless bidirectional communication between a subject and other individuals. Further, BCIs can decode multiple tasks using NFMs, including generating speech and video from brain signals in real-time based on multi-task learning. Multi-task learning is a set of techniques wherein one machine learning model learns to decode multiple tasks similar to how human brain works.
Conventional BCI/neural devices typically include electrode arrays that penetrate a subject's brain to sense and/or stimulate the brain. However, the present disclosure is directed to the use of non-penetrating BCI devices, i.e., BCI devices having electrode arrays that do not penetrate the cortical surface. Such non-penetrating BCI devices are minimally invasive and minimize the amount of impact on the subject's cortical tissue. BCI devices can sense and record brain activity, receive instructions for stimulating the subject's brain, and otherwise interact with a subject's brain as generally described herein.
1 3 FIGS.- 100 110 130 130 110 130 170 172 130 130 140 140 130 140 Referring now to, there is shown a diagram of an illustrative systemincluding a neural devicethat is communicatively coupled to an external device. The external devicecan include any device to which the neural devicecan be communicatively coupled, such as a computer system or mobile device (e.g., a tablet, a smartphone, a laptop, a desktop, a secure server, a smartwatch, a head-mounted virtual reality device, a head-mounted augmented reality device, or a smart inductive charger device). The external devicecan include a processorand a memory. In some embodiments, the external devicecan include a server or a cloud-based computing system. In some embodiments, the external devicecan further include or be communicatively coupled to storage. In one embodiment, the storagecan include a database stored on the external device. In another embodiment, the storagecan include a cloud computing system (e.g., Amazon Web Services or Azure).
180 110 180 180 180 180 180 In some embodiments, the electrode arrayof the neural devicecan have electrodes that are sufficiently small and spaced at sufficiently small distances in order to define a high-density electrode arraythat can, accordingly, capture high resolution electrocortical data. Such high-resolution data can be used to resolve electrographic features that can otherwise not be identified using lower resolution electrode arrays. In some embodiments, the electrodes of the electrode arraycan be from about 10 μm to about 500 μmin width. In one illustrative embodiment, the electrodes of the electrode arraycan be about 50 μm in width. In some embodiments, the electrodes of the electrode arraycan be spaced by about 200 μm (i.e., 0.2 mm) to about 3,000 μm (i.e., 3 mm). In illustrative one embodiment, adjacent electrodes of the electrode arraycan be spaced by about 400 μm.
110 212 180 110 212 180 204 3 FIG. The neural devicecan further include a flexible substratesupporting the electrode arrayand/or other components of the neural device, as shown in. In some embodiments, the flexible substratecan be flexible enough to permit the electrode arrayto be inserted through an osteotomy into the subdural space, then along the cortical surface.
110 110 112 114 116 118 120 112 180 102 114 102 114 116 118 112 130 118 130 120 110 130 The neural devicecan include a range of electrical or electronic components. In the illustrated embodiment, the neural deviceincludes an electrode-amplifier stage, an analog front-end stage, an analog-to-digital converter (ADC) stage, a digital signal processing (DSP) stage, and a transceiver stagethat are communicatively coupled together. The electrode-amplifier stagecan include an electrode array, such as is described below, that is able to physically interface with the brainof the subject in order to sense brain signals and/or apply electrical signals thereto. The analog front-end stagecan be configured, amplify signals that are sensed from or applied to the brain, perform conditioning of the sensed or applied analog signals, perform analog filtering, and so on. The front-end stagecan include, for example, one or more application-specific integrated circuits (ASICs) or other electronics. The ADC stagecan be configured to convert received analog signals to digital signals. The DSP stagecan be configured to perform various DSP techniques, including multiplexing of digital signals received via the electrode-amplifier stageand/or from the external device. For example, the DSP stagecan be configured to convert instructions from the external deviceto a corresponding digital signal. The transceiver stagecan be configured to transfer data from the neural deviceto the external devicelocated outside of the body of the subject.
110 119 180 119 119 119 In some embodiments, the neural devicecan include a controllerthat is configured to perform various functions, including compressing electrophysiologic data generated by the electrode array. In various embodiments, the controllercan include hardware, software, firmware, or various combinations thereof that are operable to execute the functions described below. In one embodiment, the controllercan include a processor (e.g., a microprocessor) executing instructions stored in a memory. In another embodiment, the controllercan include a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC).
110 110 130 100 100 100 1 FIG. In various embodiments, the stages of the neural devicecan provide unidirectional or bidirectional communications (as indicated in) by and between the neural deviceand the external device. In various embodiments, one or more of the stages can operate in a serial or parallel manner with other stages of the system. It can further be noted that the depicted architecture for the systemis simply intended for illustrative purposes and that the systemcan be arranged differently (i.e., components or stages can be connected in different manners) or include additional components or stages.
110 110 110 200 202 205 204 110 200 205 110 180 112 200 180 182 112 114 116 118 120 130 2 FIG. 3 FIG. In some embodiments, the neural devicedescribed above can include a brain implant, such as is shown in. The neural devicecan be a biomedical device configured to study, investigate, diagnose, treat, and/or augment brain activity. In some embodiments, the neural devicecan be positioned between the brainand the scalpor between the brain and the durain the subdural space, as shown in. In some embodiments, the neural devicecan be positioned between the brainand the dura. The neural devicecan include an electrode array(which can be a component of or coupled to the electrode-amplifier stagedescribed above) that is configured to record and/or stimulate an area of the brain. The electrode arraycan be connected to an electronics hub(which can include one or more of the electrode-amplifier stage, analog front-end stage, ADC stage, and DSP stage) that is configured to transmit via wireless or wired transceiverto the external device(in some cases, referred to as a “receiver”).
180 180 200 110 180 110 110 110 180 The electrode arraycan include non-penetrating cortical surface microelectrodes (i.e., the electrode arraydoes not penetrate the brain). Accordingly, the neural devicecan provide a high spatial resolution, with minimal invasiveness and improved signal quality. The minimal invasiveness of the electrode arrayis beneficial because it allows the neural deviceto be used with larger population of subjects than conventional brain implants, thereby expanding the application of the neural deviceand allowing more individuals to benefit from brain-computer interface technologies. Furthermore, the surgical procedures for implanting the neural devicesare minimally invasive, reversible, and avoid damaging neural tissue. In some embodiments, the electrode arraycan be a high-density microelectrode array that provides smaller features and improved spatial resolution relative to conventional neural implants.
110 110 110 120 130 110 180 180 In some embodiments, the neural deviceincludes an electrode array configured to stimulate or record from neural tissue adjacent to the electrode array, and an integrated circuit in electrical communication with the electrode array, the integrated circuit having an analog-to-digital converter (ADC) producing digitized electrical signal output. In some embodiments, the ADC or other electronic components of the neural devicecan include an encryption module, such as is described below. The neural devicecan also include a wireless transmitter (e.g., the transceiver) communicatively coupled to the integrated circuit or the encryption module and an external device. The neural devicecan also include, for example, control logic for operating the integrated circuit or electrode array, memory for storing recordings from the electrode array, and a power management unit for providing power to the integrated circuit or electrode array.
3 FIG. 110 110 180 110 204 205 200 110 204 205 200 180 180 190 200 192 20 190 192 180 Referring now to, there is shown a diagram of an illustrative embodiment of a neural device. In this embodiment, the neural devicecomprises an electrode arraycomprising nonpenetrating microelectrodes. As generally described above, the neural deviceis configured for minimally invasive subdural implantation using a cranial micro-slit technique, i.e., is inserted into the subdural spacebetween the duraand the surface of the subject's brain. In some embodiments, the neural deviceis inserted into the subdural spacebetween the duraand the surface of the brain. Further, the microelectrodes of the electrode arraycan be arranged in a variety of different configurations and can vary in size. In this particular example, the electrode arrayincludes a first groupof electrodes (e.g.,μm microelectrodes) and a second groupof electrodes (e.g.,μm microelectrodes). Further, example stimulation waveforms in connection with the first groupof electrodes and the resulting post-stimulus activity recorded over the entire array is depicted for illustrative purposes. Still further, example traces from recorded neural activity recorded by the second groupof electrodes are likewise illustrated. In this example, the electrode arrayprovides multichannel data that can be used in a variety of electrophysiologic paradigms to perform neural recording of both spontaneous and stimulus-evoked neural activity as well as decoding and focal stimulation of neural activity across a variety of functional brain regions.
Additional information regarding brain-computer interfaces described herein can be found in Ho et al., The Layer 7 Cortical Interface: A Scalable and Minimally Invasive Brain—Computer Interface Platform, bioRxiv 2022.01.02.474656; doi: https://doi.org/10.1101/2022.01.02.474656, which is hereby incorporated by reference herein in its entirety.
As generally described above, BCIs can be useful for a variety of different applications for restoring, replacing, and/or augmenting lost or impaired neurological function. However, conventional BCIs have challenges related to speed, accuracy, and the feasibility of the use of the communication channels. Conventional BCI systems translate brain signals typically captured from electroencephalography (EEG) or electrocorticography (ECoG) into a predetermined set of commands to enable a form of limited communication. Such conventional BCI system are based on EEG/ECoG data captured from test subjects that has been translated into the aforementioned commands; however, such techniques are limited because they rely on signal processing and neurophysiological signals that are ultimately derived from a small number of subjects. Translating brain signals from new subjects based on such pre-characterized data is hindered by variability in neural array placements between different subjects, the inherent variability in brain signals, low SNR ratio, and user-specific differences (e.g., differences in subjects'brain anatomy), which in turn leads to the issues with speed, accuracy, and the feasibility of the communication channels.
In order to address these issues with conventional BCI systems, embodiments of systems are described herein that incorporate NFMs with large language models (LLMs). The integration of NFMs with BCI system can be used to address the limitations of conventional BCI systems to enable a more intuitive, efficient, and natural form communication between users and computer systems. Through harnessing the computational and linguistic capabilities of NFMs, a BCI system can decode and generate neutral language from brain signals in real-time, thereby facilitating seamless bidirectional communication. Further, techniques for incorporating NFMs with a BCI system to improve the performance of the BCI system are described herein. For example, techniques for incorporating NFMs into a BCI system for engineering relevant features that improve the decoding of speech and movements from brain recordings are described herein.
100 110 130 110 In some embodiments, a BCI system (e.g., system) can include a neural devicethat is communicatively coupled to a client device (e.g., external device). The neural devicecan be used to capture electrocortical data that is translated into text and/or speech for facilitating communication via the client device. The client device can, for example, be a desktop computer equipped with applications configured to record and analyze signals. The client device can execute various NFMs and/or other algorithms to decode speech from the signals. The client device can be equipped with one or more modules configured to execute the NFMs and/or other algorithms.
4 FIG.A 400 410 440 440 440 110 440 130 440 110 130 400 410 440 440 is a block diagram illustrating a general illustrative computing environmentfor decoding speech from brain signals captured by a BCI, according to example embodiments. The neural devicecan be connected to one or more NFM modules. As used herein, the term “module” refers to hardware, software, firmware, or various combinations thereof that are operable to execute the described functions. The NFM software modulescan be stored on at least one of a local area network, an external network, a computing system (e.g., a tablet, a smartphone, a laptop, a desktop, a secure server, a smartwatch, a head-mounted virtual reality device, a head-mounted augmented reality device, or a smart inductive charger device), and a cloud computing network. In one embodiment, the one or more NFM modulescan be executed onboard the neural device. In another embodiment, the one or more NFM modulescan be executed by the external device. In yet another embodiment, the one or more NFM modulescan be executed across the neural device, the external device, and/or other devices in a parallel or sequential manner. In some embodiments, the computing environmentis capable of bidirectional communication between the neural interfaceand the NFM modules. In some embodiments, the NFM modulescan include one or more LLMs.
4 FIG.B 400 40 435 480 400 410 435 410 435 400 is a block diagram illustrating an illustrative computing environmentfor decoding speech from brain signals captured by a BCI, according to example embodiments. As shown, the computing environmentcan be based on a client-server model, with a serverconnected to at least one client device. The computing environmentcan include a neural interfacecommunicatively coupled to a server. In some embodiments, the neural interfacecan be considered a client to the server. It should, however, be understood that the client-server model is just for illustration and ease of explanation and should not be considered limiting. Therefore, any type of computing environment performing the functionality disclosed herein should be considered within the scope of this disclosure. Furthermore, the individual components of the computing environmentare just illustrative and computing environments with alternative, additional, or fewer number of components should be considered within the scope of this disclosure.
400 435 440 480 410 430 480 440 480 440 410 440 410 410 435 480 The computing environmentcan be generally in a clinical setting to monitor brain signals of a subject. In some illustrative use cases, the servercan store different NFM modulesthat can be accessed by the clientsand neural interfaceusing the network. The clientsthemselves can have standalone applications (not shown) to access the NFM modules. Alternatively, the clientscan access the NFM modulesthrough a browser application, for example. Similarly, the neural interfacecan access the NFM modulesthrough any type of firmware and/or software installed in the neural interface. In some embodiments, the neural interfacecan communicate with the serverusing one or more of the clients.
435 440 435 435 435 435 The hardware of the serverstoring the NFM modulescan include any kind of computing device. For example, the servercan include a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone. The servermay not necessarily be at a single location and can be realized by a network of computers. Furthermore, the servermay not necessarily be co-located within the clinical setting itself and can be hosted by a third-party cloud computing provider. Therefore, any kind of servershould be considered within the scope of this disclosure.
480 410 435 430 430 430 480 480 480 480 480 435 430 435 480 a b c d As described above, the clientsand/or the neural interfacecan access the serverthrough the network. The networkcan include any combination of one or more packet switching networks (e.g., an internet protocol-based network) and/or one or more circuit switching networks (e.g., a cellular telephony network). Some non-limiting examples of the networkinclude a local area network, a metropolitan area network, and/or a wide area network such as the Internet, etc. Similarly, non-limiting examples of the clientscan include a desktop terminal (e.g., desktop terminal), a laptop computer (e.g., a laptop computer), a tablet computer (e.g., a tablet computer), a smartphone (e.g., a smartphone), etc. Any type of computing device that allows an access to the serverthrough the networkshould be considered within the scope of this disclosure. Furthermore, the functionality described within this disclosure can be distributed in any fashion, i.e., functionality of the servercan be performed by one or more clientsand vice versa.
435 450 452 454 456 458 460 462 464 466 468 470 435 4 FIG.B As described above, the servercan include multiple software modules.shows some non-limiting illustrative software modules: a feature extraction module, an encoder module, a self-attention module, a decoder module, a feedforward network module, a subject-specific encoder module, a speech synthesis module, an attention module, a beam search decoder module, a training module, and a reinforcement learning module. It should be understood that this described modularization of the serverfunctionality is provided simply for the ease of explanation and should not be considered limiting. Therefore, any kind of alternative modularization should be considered within the scope of this disclosure.
450 410 450 410 410 180 450 The feature extraction modulecan extract features of the brain signals from the recorded brain signals captured by the neural interface. In some embodiments, the extracted features include a spatial representation and a temporal representation of the recorded brain signals. The neural device represented in this patent includes 1024 electrodes in a 33 by 31 shape (where one electrode is dropped out after reshaping) which can be employed to extract spatial information across electrodes. The feature extraction modulecan select features of the recorded brain signals using convolutional neural network models or through analysis of different frequency bands e.g., one feature may contain high frequency signals that correlates with the intent to moving the hands or uttering vowels. The neural interfacecan be a high-resolution BCI which provides a high-resolution brain signal with decreased SNR. In some embodiments, the neural interfacecan include the neural devicedescribed above. In some embodiments, user profile data and/or digital history can be concatenated with the feature extraction module.
452 450 502 501 452 502 501 501 452 454 458 454 502 501 452 454 e The encoder modulecan generate embeddings based on the feature representations generated by the feature extraction module. For example, if the recorded brain signals are separated into patches (i.e., a group of neighboring electrodes)and frames, the encoder modulecan capture spatial relationships between the patchesin the framesas well as temporal relationships between frames. The encoder modulecan include one or more self-attention modulesand/or a feedforward network modules. Wherein the self-attention module is a mechanism that allows the model to learn which patches and frames are most important for a given decoding task. For example, the one or more self-attention modulescan learn both the spatial representation between patchesand the temporal representation between frames. The encoder modulecan be tailored to speech production and can include at least one temporal transformer block and at least one spatial transformer block. These transformer blocks can include self-attention modulesand can separate the spatial representation and the temporal representation from the spatiotemporal features. The separate spatial and temporal representations can be used to determine speech, as a spatial representation, in addition to the temporal representation, is needed to decode speech. This is in contrast to text being able to be decoded based on a temporal representation.
452 452 452 452 452 452 452 In some embodiments, encoder modulecan be trained using multitask learning. Multitask learning can create a robust encoder moduleby learning to generalize its understanding of data to multiple tasks. As such, encoder modulecan be used in inference using zero-shot and/or few-shot classification. Encoder modulecan generalize across tasks and subjects such that for a given dataset, encoder modulecan extract representations consistent with the appropriate task. For example, for neural data representing a subject's attempt to move, encoder modulecan extract the representations consistent with movement and determine that a particular motion is attempted. The representation can be based on a k-nearest neighbor determination. For example, if encoder modulegenerates a neural embedding from a movement by a subject, then the k-nearest neighbor can determine if it is near a certain pre-determined class of movements.
454 440 452 454 The self-attention modulecan be used by at least one of the other NFM modulesto weigh the importance of elements in an input and adjust their influence on the output. The importance of each element can determine how much attention is paid to it. The elements can include past events which can enable long-term dependencies. For example, the encoder modulecan use the self-attention modulesto incorporate past elements that can be beneficial to the prediction.
456 452 452 456 464 456 452 458 The decoder modulecan receive the output from the encoder moduleto decode the spatial and temporal embeddings generated by the encoder module. In some embodiments, the decoder modulecan include a causal masked attention modulewherein masking control which sequence can see each other in the attention computation. A causal mask ensures that the model can only attend to tokens at the current or earlier positions, not future positions. The decoder modulecan be used to predict phonemes based on the spatial and temporal embeddings received from the encoder moduleand/or the feedforward network module.
458 458 468 458 The feedforward network modulecan include an efficient model comprising a reduced number of parameters to reduce training time for personalization of a model for a subject. For example, during personalization the number of trainable parameters can be reduced from w*h to w*r+h*r, where w is the number of layers of the feedforward network module, h is number of units in each layer, and rank r is a value greater than one determined through hyperparameter tuning. The reduced feedforward network parameter is then weighted average with the original feedforward network, i.e., w*h+alpha*(w*r+h*r) where alpha controls the weighting between the two networks. In some embodiments, the reduced feedforward network modulecan enable the training moduleto efficiently fine-tune the feedforward network moduleat least once per day.
460 460 460 452 460 460 458 1 2 1 2 w*feedforward network module+w*subject specific encoder module, where wand ware relative weightings. The subject-specific encoder modulecan learn from subject-specific representations based on data gathered for the specific subject while keeping the rest of the model frozen. The subject-specific encoder modulecan be fine-tuned regularly to ensure that the subject-specific encoder moduleis properly encoding the recorded brain signals. In some embodiments, the encoder modulecan include a subject-specific encoder module. The output of the subject-specific encoder modulecan be weighted summed with the output of the feedforward network moduleto generate a single output embedding of the recorded brain signals. For example, the weighted sum can be represented by the following equation:
462 462 456 462 466 The speech synthesis modulecan utilize a subject's past speech utterances and corresponding transcription to model the subject's voice characteristics. The speech synthesis modulecan receive the output from the decoder moduleand, based on the decoding and predicted speech, generate speech for the subject such that others can hear what the subject wanted to say. In some embodiments, the speech synthesis modulecan utilize a subject's past video recordings to generate a video representation of the subject. The video representation of the subject can be based on predicted motor commands based on the recorded brain signals. In some embodiments, the beam search decoder modulecan also predict motor commands.
462 462 452 462 462 In some embodiments, the speech synthesis modulecan generate speech based on the neural embeddings. Speech synthesis modulecan receive the neural embeddings from encoder moduleand perform stable diffusion on the neural embedding and a latent time-frequency representation of underlying speech to generate new speech. In some embodiments, speech synthesis modulecan guide the speech and/or video generation by using Contrastive Language-Image Pre-training (CLIP) loss. Speech synthesis modulecan generate avatar movement based on neural embeddings of movement and video embeddings of a subject's prior recorded movements.
464 452 456 464 464 464 The attention modulecan be utilized by the encoder moduleand the decoder moduleto attend to selected input elements. The attention modulecan include at least one transformer block. In some embodiments, attention modulecan include specialized transformer blocks. For example, attention modulecan include a spatial transformer block and a temporal transformer block.
468 468 440 468 456 468 452 456 The training modulecan train portions of the NFM. The training modulecan train at least one of the analytical modules in the NFM moduleswith a plurality of subjects. In some embodiments, the training modulecan train the decoder moduleusing a transcription corresponding to the speech segment as a ground truth with a Connectionist Temporal Classification (CTC) loss. The speech transcription corresponds to phonemes which is a distinct units of sound uttered in the speech. The CTC loss aligns the predicted phoneme probabilities using a blank token with the ground truth phoneme transcriptions by automatically aligning them hence reducing the need for detecting and aligning every single phoneme in the neural data. For example, this can be done by instructing a subject what to say prior to recording the brain signals. In some embodiments, the training modulecan train the encoder moduleand the decoder module.
466 456 466 466 The decoder module predicts the phoneme probabilities associated with each time step. The beam search modulecan convert the phoneme probabilities predicted by the decoder moduleinto a transcription. The beam search decoder takes in the phonemes predicted at each time step and evaluates multiple paths (i.e., beam width) to the most likely sequence of phonemes using a language model. The language model is trained on a corpus of text data tokenized to phonemes wherein its tasked to predicting the next phoneme given previous phonemes. In some embodiments, the beam search decoder modulecan include a CTC beam search decoder. The CTC aligns the phoneme probabilities using a blank token by collapsing repeated phonemes. For example, if the decoder predicted “hellllooo”, then the CTC would collapse the repeated phonemes to “hello”. The beam search decoder modulecan additionally predict motor commands based on decoded brain signals by incorporating possible kinematics at each time step.
468 462 468 470 In some embodiments, the language model is trained using masked language modelling wherein a portions of phonemes are randomly removed and prompting the language model to predict the removed phonemes. This enables the language model to be contextually aware when predicting the most likely phoneme. In some embodiments, the training modulecan train the speech synthesis moduleto generate speech that sounds like the user. In some embodiments, the training modulecan include a reinforcement learning module.
468 468 468 458 452 468 452 456 452 468 458 468 452 The training modulecan perform fine-tuning to ensure that the NFM is performing as expected. Given the high-resolution data and the costs associated with the training, the training modulecan freeze one or more layers and/or parameters prior to performing the fine-tuning. In some embodiments, the training modulecan train at least one of the feedforward network moduleand the encoder moduleduring the fine-tuning. During fine-tuning, the training modulecan train the encoder moduleon a downstream task to enable the NFM to learn a higher-level structure of brain recordings while freezing the decoder module. This training can enable neural representations that learn contextual flow and coherence of language. The encoder modulecan be fine-tuned on a plurality of subjects through gradual unfreezing of layers. In some embodiments, the training modulecan fine-tune a reduced feedforward network module. The training moduletraining a smaller set of parameters enables fine-tuning with a smaller number of samples. The encoder modulecan also be fine-tuned through the parameter-efficient fine-tuning to create new representations of the neural embeddings.
470 470 470 466 470 470 The reinforcement learning modulecan be used to fine-tune the NFM using provided feedback. In some embodiments, the reinforcement learning modulecan include a reinforcement learning human feedback (“RLHF”) model. The reinforcement learning modulecan receive transcriptions generated by the beam search decoder moduleand prompt a user and/or caretaker to rank the transcriptions based on the user's true intent. The reinforcement learning modulecan use the user's response to improve the performance of the NFM. In some embodiments, the reinforcement learning modulecan freeze one or more parameters and/or layers that are not part of the transcription generation process in order to fine-tune fewer parameters allowing for decreased cost and processing power.
440 435 440 After one or more analytical models have been developed and validated, clinicians can use the NFM moduleswithin the serverto decode the speech of the subject on which the NFM moduleswere trained.
400 410 435 400 400 400 4 FIG.B In various embodiments, the stages of the illustrative computing environmentcan provide unidirectional or bidirectional communications (as indicated in) by and between the neural interfaceand the server. In various embodiments, one or more of the stages can operate in a serial or parallel manner with other stages of the computing environment. It can further be noted that the depicted architecture for the computing environmentis simply intended for illustrative purposes and that the computing environmentcan be arranged differently (i.e., components or stages can be connected in different manners) or include additional components or stages.
5 16 FIGS.-B Various embodiments of NFMs that can be integrated into the BCI systems described above are illustrated in. Various features, components, and/or techniques from the embodiments described below can be used in different combinations with each other. Before discussing each of the embodiments in detail, various features that can be incorporated into the embodiments will be discussed.
In some aspects, the feature extraction module can utilize a transformer-based architecture to learn neural representations from the captured brain signals. This transformer-based feature extraction module can include a transformer encoder block. The transformer encoder block can be configured to extract neural representations from the raw signals for a specific task, such as phoneme classification. In certain embodiments, the transformer model can comprise an encoder and a decoder. The encoder can be configured to process the neural embeddings. The processing of the neural embeddings can involve various techniques, such as self-attention mechanisms, to capture spatial and temporal relationships in the neural embeddings. The decoder, on the other hand, can be configured to generate output based on the processed neural embeddings. The output can represent a decoded version of the neural embeddings, which can be used to generate a form of communication, such as speech or text.
Various embodiments of NFM architectures can also include a speech synthesis module. This module can be configured to generate audible speech based on the output of the transformer model. The speech synthesis module can utilize various techniques to generate speech that is natural and intelligible to the user.
In some cases, the NFM architecture can be configured to perform multi-task learning. This can involve processing neural embeddings for multiple tasks in parallel. For example, the system can be configured to decode both speech and motor commands simultaneously. This can enhance the utility of the BCI function, allowing the user to perform multiple tasks concurrently.
502 501 501 502 501 501 502 501 502 501 501 502 502 501 501 In some embodiments, the captured brain signals used for training and/or during execution time of the NFM architecture can be divided into a series of patchesand frames. Each framecan be collected over a specific time period, which can range from a few milliseconds to several seconds, depending on the classification task. The captured brain signals can be processed using various signal processing techniques. For example, the raw signals can be downsampled and filtered to be within a specific frequency range. This can be done to reduce the computation power required for real-time decoding of the brain signals. The processed signals can then be put into a predetermined analysis window. In some cases, the processed signals can be further processed by extracting the envelope of the signal from every electrode at different frequency bands. The processed signals from different envelopes can be concatenated to form a new feature vector that can be fed to the model. This can help to preserve the spatiotemporal representation that is inherent to the brain signals. In some cases, the transformer encoder block can process the captured brain signals as a series of patchesand frames. Each framecan be collected over a specific time period, which can range from a few milliseconds to several seconds, depending on the classification task. The patchesand framescan then be linearly projected to a lower dimension and flattened to form input embeddings. In some cases, the transformer encoder block can add positional encoding to each patchand framein the sequence to consider the order of framesand patchesin the data. As used in the technical field, positional encoding is used to provide a relative position for each token or word in a sequence. Once the input embeddings and the position encodings are added, the output can be fed to transformer encoder blocks. The transformer encoder blocks can employ self-attention to capture spatial relationships between all patchesin each frameas well as temporal relationships between frames.
502 501 Various embodiments of NFM architectures can include a feature extraction module configured to process the captured brain signals and generate neural embeddings. The neural embeddings can represent a data-driven neural representation of the captured brain signals. The feature extraction module can use a transformer encoder block to extract neural representations from the captured brain signals. The transformer encoder block can be configured to process the captured brain signals as a series of patchesand frames, preserving the spatial and temporal representation of the signals.
In some embodiments, the feature extraction module can be trained or otherwise configured to perform multi-task learning. This approach can involve processing neural embeddings for multiple tasks in parallel. For example, the NFM architecture can be configured to decode both speech and motor commands simultaneously. This can enhance the utility of the BCI function, allowing the user to perform multiple tasks concurrently. The multi-task learning approach can involve sharing a set of parameters across multiple tasks, thereby enabling the system to learn a global representation of brain recordings. In addition, a task-specific class (CLS) token could be used to learn the brain recording representation for any given tasks. A classification token is a vector of numbers that is learned by the NFM encoder during training. This approach can allow the system to tailor the feature extraction process to the specific requirements of each task, thereby improving the accuracy and versatility of the BCI system.
The feature extraction module can be trained using data obtained from a number of different subjects. This approach can be used to make the feature extraction process robust to subject variability. By training the feature extraction module on brain signals from multiple subjects, the system can be able to learn a more generalized representation of brain recordings. This can allow the system to better handle the variability in brain signals between different subjects, thereby improving the performance of the BCI system across a wider range of users. The training process can involve adjusting the parameters of the feature extraction module based on the brain signals from each subject, thereby fine-tuning the feature extraction process to the specific characteristics of each subject's brain signals. This approach can allow the system to adapt to the unique characteristics of each subject's brain signals, thereby improving the accuracy and personalization of the BCI system.
In some embodiments, the NFM architecture can employ a unified attention mechanism to learn spatiotemporal representations of brain recordings. This approach can involve decoupling the spatial and temporal representation of the brain signals using separate spatial and temporal transformer blocks. Each of these transformer blocks can be configured to process the neural embeddings in a specific manner. For instance, the spatial transformer block can be configured to process the spatial representation of the neural embeddings, while the temporal transformer block can be configured to process the temporal representation of the neural embeddings. This approach can allow the system to capture both the spatial and temporal characteristics of the brain signals, thereby improving the accuracy and versatility of the BCI system.
502 501 502 501 502 501 In some embodiments, the NFM architecture can employ a unified attention mechanism that considers both the spatial and temporal representation of the brain recordings. This unified attention mechanism can involve processing the neural embeddings using a single transformer block that is configured to capture both the spatial and temporal characteristics of the brain signals. This approach can allow the system to learn a more comprehensive representation of the brain signals, thereby improving the performance of the BCI system. The unified attention mechanism can involve processing the neural embeddings in a manner that preserves the inherent spatiotemporal pattern of the brain signals. This can involve processing the neural embeddings as a series of patchesand frames, with each patchand framerepresenting a specific spatial and temporal characteristic of the brain signals. The unified attention mechanism can then use self-attention to capture the relationships between the patchesand frames, thereby learning a comprehensive spatiotemporal representation of the brain signals. This approach can allow the system to better understand and interpret the brain signals, thereby improving the accuracy and personalization of the BCI system.
The transformer model can be trained or otherwise configured to decode the neural embeddings into phonemes or other similar outputs. The encoder of the transformer model can be configured to process the neural embeddings, while the decoder can be configured to generate output based on the processed neural embeddings. The output can represent a decoded version of the neural embeddings, which can be used to generate a form of communication, such as speech or text.
In some embodiments, the transformer model can be fine-tuned using reinforcement learning from human feedback to improve transcription generation. This can involve generating multiple transcriptions from the decoded neural embeddings, receiving user feedback on the quality of the generated transcriptions, constructing a reward function based on the user feedback, and adjusting parameters of the transformer model to maximize the expected cumulative reward.
In some embodiments, the transformer model can be fine-tuned using parameter-efficient fine-tuning techniques to selectively update a subset of the transformer model parameters. This can involve freezing a subset of the parameters of the transformer model and training selected layers. This approach can reduce the number of parameters needed for fine-tuning, allowing the transformer model to be fine-tuned on a smaller number of samples and enabling few-shot learning.
The transformer model can further be trained or otherwise configured to process user embeddings along with the neural embeddings to generate personalized output. The user embeddings can be generated based on user-specific information, such as the user's age, gender, handedness, array placement in the brain, or device information. This can help to personalize the speech or text output to each individual user.
5 FIG. 500 500 510 520 530 500 505 110 500 505 1024 505 510 505 501 180 180 510 520 520 520 530 520 500 Turning now to specific embodiments of NFM architectures,shows a general diagram of an NFM architectureadapted to predict speech phonemes, in accordance with example embodiments. The NFM architecturecan include a one-dimensional convolutional layer, one or more convolutional layers, and a softmax layer. The NFM architecturecan be programmed or otherwise configured to receive a downsampled, filtered neural signalfrom a neural interface, such as the neural devicedescribed above. The NFM architecturecan further include algorithms or other processes for generating the downsampled, filtered signal. The downsampled filtered signals from theelectrodescan be fed into the one-dimensional convolutional layerwherein the layer extracts temporal features. The downsampled filtered signalscan be arranged as a three-dimensional array of signals. As generally described above, the electrode arrayconsists of a two-dimensional array of electrodes, which is configured to rest against the subject's cortical surface and capture a corresponding two-dimensional set of data of the electrical activity along the cortical surface. Successive sets of two-dimensional electrocortical data can further be arranged temporally to form a three-dimensional array of electrocortical data. In some embodiments, the array of signals can include brain signals captured from a high-resolution electrode array, such as is described above. The output of the one-dimensional convolutional layercan be input to one or more convolutional layers. The one or more convolutional layerscan include one or more feedforward networks. The one or more convolutional layerscan use signal processing techniques for extracting an envelope of the signal from a plurality of electrodes at different frequency bands. The processed signals from different envelopes can be concatenated to form a new feature vector that can be fed to the NFM. The softmax layercan output phoneme probabilities based on the representations learned by the one or more convolutional layers. However, such general techniques tend to overfit quickly and the NFM performance can plateau quickly. The spatial representation of the signal can also be lost as the signal is vectorized over the frequency channels. Thus, modules can be added to the NFM architecturesuch that spatial and temporal representations of the signal can be preserved.
450 500 500 620 504 503 502 501 620 505 506 505 458 530 6 FIG. In some embodiments, the NFM can include a feature extraction moduleto preserve the spatial and temporal representation of the brain signal.shows an NFM architectureadapted to extract features from recorded brain signals prior to speech prediction, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes a set of transformer blocksthat receive input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The transformer blockscan generate neural embeddings, which can in turn be fed through feedforward networkconsisting of a convolutional layer, a feedforward network body, and a softmax layerto output phoneme probability associated with the neural data.
501 502 180 501 501 180 501 180 501 502 501 502 502 501 502 502 The feature extraction module can extract features from the recorded brain signals by dividing the signal into framesand patchesto enable separation of the spatial and temporal representation. The brain signals can be recorded for a set length of time. A multidimensional array can be generated including the signal captured from each electrode at each time step in a predetermined frequency band. As such, the dimensions of the array can be the spatial arrangement of the electrocortical data (which corresponds to the spatial arrangement of the electrodes that make up the electrode array) relative to the number of time steps elapsed at one or more predetermined frequency bands. The recorded brain signals can include a time stamp at each time step. A four-dimensional array can be generated by reshaping the recorded brain signals at each time step into a three-dimensional frame, such that the brain signals are formed into a time-ordered array of the frames. For example, the brain signals recorded by an embodiment of the electrode arraydescribed above can form a 33×31×4 framecorresponding to the spatial arrangement of the electrodes that make up the electrode arrayfor four distinct frequency bands at a given time step. Each framecan be divided into a plurality of patches. In some embodiments, each framecan be divided into four patches. The plurality of patchescan vary in size and extend over the framein which case the patchcan be filled with values from the nearest electrodes. The patchescan be linearly projected to a lower dimension and flattened.
502 501 610 502 501 620 620 454 The projected patchescan be used as the input embeddings. In some embodiments, each patch in a framecan correspond to at least one input embedding. A task specific classification (CLS) tokencan be added to the input embedding to serve as a representation of the patchesand framesfor a given window. Transformer blockscan be used to extract neural representations that are useful in a phoneme classification task wherein the NFM learns a latent representation of the neural recording that is useful in predicting the correct phonemes associated with a neural representation. Each transformer blockcan include one or more self-attention modules. In some embodiments, the CLS token representation from the last transformer block can be used to represent the neural embeddings for the corresponding neural data.
502 501 502 501 501 620 454 620 502 502 501 622 456 510 458 530 458 A positional encoding of the patcheswithin each framecan be added to consider the order and position of patcheswithin a frameand relative to the sequence of framesin the data. These positional encodings of patches can also enable the model to perform electrode selections e.g., attending to at least one or more electrodes that convey more information about the underlying neural signal. The transformer blockscan receive the input embeddings and the positional encodings. The one or more self-attention modulesof the transformer blockcan capture spatial relationships between patchesand temporal relationships between patchesin multiple frames. In some embodiments, the CLS token from the last transformer blockcan indicate a spatiotemporal representation, hereafter called neural embeddings, of the brain signals. These neural embeddings can be used as an input to decoding tasks hence reducing the amount of data that is needed for training task-specific decoder models. In some embodiments, the neural embeddings can be passed to a one-dimensional convolutional layerand a feedforward network module. A softmax layercan then generate task specific probabilities for a variety of classification and regression tasks. For example, in speech decoding the task specific head can return phoneme probabilities based on the output of the feedforward network module.
7 7 FIGS.A-B 6 FIG. 500 600 620 610 500 610 610 610 610 610 a c a c a b c In some embodiments, the NFM can perform a plurality of tasks and/or perform decoding on a plurality of subjects, i.e., the NFM can be configured for multi-tasking. The NFM can use spatial and temporal transformer blocks to preserve the spatiotemporal features in the recorded brain signals.show embodiments of an NFM architectureadapted to perform a plurality of tasks in parallel or perform decoding on a plurality of subjects via parameter sharing, rather than being adapted to perform a single task as in. The NFM architecturecan be adapted to perform a plurality of tasks in parallel through sharing a set of parameters. In some embodiments, the CLS tokencan be shared by the plurality of tasks to learn a global representation of the brain recordings. The NFM architecturecan utilize a plurality of tailored CLS tokens-that correspond to separate tasks. For example, the CLS tokens-can correspond to tasks including phoneme decoding, motion decoding, and/or classifying the positions of a given stimuli. The different tasks can correspond to separate subjects and/or subjects and/or decoding tasks.
620 620 The neural embedding generated by the shared transformer blocks modulecan be fed to each task as an input. For example, the neural embedding can be used to train a model for the task of phoneme decoding and/or a variety of motion decoding such as hand gestures and finger movements. Note that each task can have different architectures and requirements. The neural representation that is learned from the shared transformer blockscan enable new task learning across a variety of subjects especially when the data specific tasks are rare.
7 FIG.A 7 FIG.A 500 620 504 503 502 501 620 505 506 506 506 620 454 710 720 620 a b c In the embodiment depicted in, the NFM architectureincludes a set of transformer blocksthat receive input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The transformer blockscan generate neural embeddings, which can in turn be fed through multiple feedforward networks (e.g., three feedforward networks,,as shown in) that each provide outputs for different tasks associated with the neural data (e.g., phoneme probability or motion probability). The shared transformer blocksrely on the self-attention moduleto learn both the spatial and temporal representations. The separate spatial transformer blockand temporal transformer blockcan provide a more clearly preserved spatiotemporal pattern from the brain signals than the shared transformer blockswhile reducing the number of operations needed to calculate the attention across the space and time axes.
620 710 720 710 720 454 710 720 620 500 710 720 504 503 502 501 620 505 506 506 506 500 710 720 7 FIG.B 7 FIG.A 7 FIG.A 7 FIG.B 7 FIG.A 7 FIG.B 7 FIG.A a b c In some embodiments, the transformer blockscan include a spatial transformer blockand a temporal transformer blockto detangle the representation learned through one shared transformer block. Each transformer block,can use a self-attention modulethat can be specialized to perform the given function. The separate spatial transformer blockand temporal transformer blockofcan be a portion of the plurality of shared transformer blocksshown in. As with, the embodiment of the NFM architecturedepicted inincludes a set of transformer blocks,that receive input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The transformer blockscan generate neural embeddings, which can in turn be fed through multiple feedforward networks (e.g., three feedforward networks,,as shown in) that each provide outputs for different tasks associated with the neural data (e.g., phoneme probability or motion probability). However, the embodiment of the NFM architectureindiffers from the embodiment shown inin that it utilizes a separate spatial transformer blockand a separate temporal transformer block.
8 FIG. 7 FIG.A 500 500 620 710 720 504 503 502 501 620 505 506 506 506 620 454 710 720 a b c In some embodiments, the NFM can execute the spatial transformer and the temporal transformer in parallel to optimize the preservation of the spatiotemporal features from the recorded brain signals.shows an embodiment of an NFM architectureadapted to encode recorded brain signals via parallel, specialized attention blocks, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes a transformer blockthat implements spatial attentionand temporal attentionin parallel and receives input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The transformer blockcan generate neural embeddings, which can in turn be fed through multiple feedforward networks (e.g., three feedforward networks,,as shown in) that each provide outputs for different tasks associated with the neural data (e.g., phoneme probability or motion probability). The transformer blockscan include a unified self-attention modulethat operates a spatial attentionand a temporal attentionin parallel to learn an aggregate representation of the brain signals. The concatenated aggregate representation can be linearly mapped prior to being output as the output embeddings.
18 FIG. 19 FIG. 19 FIG. 1801 1802 454 458 1803 1901 In some embodiments, the NFM is applied for decoding data captured from rostrums. For example, different spots on the rostrum can be stimulated using different mechanism, for example using a pressurized air puff or a mechanical arm. Each spot on the rostrum would then correspond to a class that the NFM needs to decode, i.e., this a classification problem wherein the NFM needs to learn from the neural data which spot on the rostrum was stimulated across multiple sessions and subjects. Due to the high variabilities in the collected data, e.g., different rostrum shapes, time of a session in the day, and the type of stimuli used in each session and subject these variabilities are all treated as a new task when training the NFM. For example, a task may correspond to a rostrum X that was stimulated at eight different locations randomly for several minutes in the morning and another to rostrum Y stimulated at ten different locations in the afternoon. In this experiment, the NFM was trained on eleven tasks and five subjects, each task corresponding to a different classification problem. As shown in, the rostrum data corresponding to each taskare preprocessed through a normalization module wherein the values of the electrodes are set to be within a specific pre-determined range. The multitask dataset are then fed to the encoder portion of the NFM, wherein the parameters of the NFM are shared across all tasks. The output from the NFM, the neural embeddings, are fed to a shallow SoftMax layers, i.e., task-specific headthat predicts the class location for a given task using a cross-entropy loss function.depicts the accuracy performance of the training and validation dataset for several tasks as the NFM training was in progress. The reference charactersin the bottom right portion of each plot represent the type of stimuli applied (“A” for air puff, “P” for piston, and so on) and the number of positions a rostrum was stimulated. For example, P14 indicates a 14-class decoding using a piston stimulus. As shown in, the performance accuracy on the validation dataset rises as training progresses for all tasks, albeit the rise is slower for some tasks than others.
454 2001 20 FIG. In some embodiments, the output from the shared NFM, i.e., the neural embeddings corresponding to each task and class, were projected down to three dimensions for visualization as shown in. Each cluster of points represents the neural embeddings corresponding to one subject, one task and one class. The neural embeddings from the same subject and stimulus tend to cluster more closely together. These clusters represent how the NFM can find shared representations that is robust across subjects, stimulus, and sessions.
21 FIG. In some embodiments, the encoder portion of the NFM was used to encode a new subject neural data that it had not seen before as shown in. The training and validation accuracy performance of these new tasks during fine-tuning the NFM started at very high accuracies. This enables the NFM to quickly adapt its representations on new tasks using only a few examples.
9 FIG. 500 500 452 504 503 502 501 452 505 456 452 450 452 464 452 454 454 452 458 454 e In some embodiments, an encoder and a decoder both comprise of transformer blocks.shows a diagram of an NFM architectureincluding an encoder and a decoder attention mechanism, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, have been divided into patchesfrom framesof the neural data. The encodercan generate neural embeddings, which can in turn be fed through a decoderto output provide an output associated with the neural data (e.g., phoneme probability). Encoder modulecan receive spatiotemporal features from the feature extraction module. Encoder modulecan include one or more attention modules. In some embodiments, the transformer encoder modulecan include one or more self-attention modules. The output of the one or more self-attention modulescan be normalized. The encoder modulecan include a feedforward network module, which can receive the normalized output of the one or more self-attention modulesand generate encoded representations of the brain signals, i.e., neural embeddings.
456 456 464 458 464 458 452 464 458 456 464 530 d e d The transformer decoder modulecan receive the neural embeddings along with the ground truth. The transformer decoder modulecan include one or more attention modulesand a feedforward network module. In some embodiments, the one or more attention modulescan include a masked multi-head attention block. The output of the feedforward network moduleof the encoder modulecan be input to the one or more attention modules. The feedforward network moduleof the decoder modulecan receive an output from the one or more attention modulesand generate a decoded representation of the brain signals. A softmax layercan receive the decoded representation and generate a probability of the most likely phonemes at each time step based on the decoded representation. The advantages of having a transformer decoder block are enabling the NFM to learn which neural embeddings to attend during decoding.
500 452 500 500 452 504 503 502 501 452 505 456 452 460 9 FIG. 10 FIG. 9 FIG. 9 FIG. In some embodiments, the NFM architecturecan include a subject-specific encoder adapted to enable personalization and reduce the number of parameters required to be fine-tuned, as opposed to training all the parameters for the generalized encoderas in the embodiment shown in.shows an embodiment of an NFM architectureadapted to encode recorded brain signals with reduced parameters incorporating a subject specific embedding, in accordance with example embodiments. As with the embodiment shown in, the depicted embodiment of the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The encodercan generate neural embeddings, which can in turn be fed through a decoderto output provide an output associated with the neural data (e.g., phoneme probability). This embodiment differs from the embodiment shown inin that the encoderincludes a subject-specific encoder module, which is described in greater detail below.
452 454 458 460 458 454 460 458 460 452 458 460 452 e e e e The encoder modulecan include one or more self-attention modules, a feedforward network module, and a subject-specific encoder module. The feedforward network modulecan receive the output of the one or more self-attention modulesand generate an encoded representation of the brain signals. For example, the subject-specific encoder modulecan be trained from data gathered for a specific subject attempting to speak. The output of the feedforward network modulecan be weighted summed with the output of the subject-specific encoder moduleto generate the embedded output of the encoder module. The outputs of the feedforward network moduleand the subject-specific encoder modulecan be weighted summed to avoid catastrophic forgetting and to hold to the contextualized representation of what the encoder modulehas learned from a plurality of subjects and tasks prior to being used for the specific subject.
11 FIG. 9 10 FIGS.and 500 500 452 504 503 502 501 452 505 456 500 452 1110 1110 1110 468 1110 In some embodiments, the NFM can include a subject profile embedding concatenated to the output of the encoder.shows an embodiment of an NFM architectureadapted to include a subject profile in the neural embedding, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The encodercan generate neural embeddings, which can in turn be fed through a decoderto output provide an output associated with the neural data (e.g., phoneme probability). The depicted embodiment differs from the embodiments shown inin that the NFMcan be tailored to a specific subject by concatenating the output of the encoder modulewith a user embedding. The user embeddingcan include user profile and digital history that can be used to create a personalized experience for the user. The user profile can include, for example, one or more of the user's age, gender, handedness, array placement in the brain, and other static information about the device or the subject. A separate network can transform the user profile and/or digital history into user embeddings. During fine-tuning, the training modulecan freeze the user embedding.
12 FIG. 500 1210 500 452 504 503 502 501 452 505 456 456 464 456 452 1210 1210 464 456 452 1210 464 458 456 458 456 458 456 530 d d d In some embodiments, the NFM can include a decoder adapted to include a subject embedding.shows an embodiment of an NFM architectureadapted to include a subject embeddingin a decoder, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data that, in some cases, can have been divided into patchesfrom framesof the neural data. The encodercan generate neural embeddings, which can in turn be fed through a decoderto provide an output associated with the neural data (e.g., phoneme probability). In this embodiment, the decoderthe one or more attention modulesof the decoder modulecan receive the output from the encoder modulealong with a subject embedding. The subject embeddingcan include the subject's preferred vocabulary and personalized sentences. Personalized sentences can include common command-like speech. One or more attention modulesof the decoder modulecan attend to the input sequence from the encoder moduleand the subject embeddingin order to achieve the personalization. The outputs of the attention modulescan be concatenated prior to the feedforward network moduleof the decoder modulereceiving them. In some embodiments, the concatenation can be linearly mapped prior to the feedforward network moduleof the decoder modulereceiving it. The feedforward network moduleof the decoder modulecan generate a decoded representation which can be input to a softmax layerto generate phoneme probabilities.
13 FIG. 500 500 452 504 503 505 456 500 466 456 466 466 466 466 In some embodiments, the NFM can include a beam search decoder adapted to predict a most likely transcript based on the phoneme probabilities.shows an embodiment of an NFM architectureadapted to predict speech via a beam search decoder, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data, which can generate neural embeddingsthat are fed through a decoderto provide an output associated with the neural data (e.g., phoneme probability), as described above. In this embodiment, the NFM architecturefurther includes a beam search decoder modulethat can receive a number of parameters in addition to the emission from the decoder moduleto create the most likely transcription. In some embodiments, the beam search decoder modulecan be a CTC beam search decoder. The parameters can include one or more of the following: a lexicon model, a small language model, along with the beam size, and an associated vocabulary. The lexicon model can provide prior knowledge on most likely phoneme sequences. The small language model can be an N-gram mode that has been tailored to the user's language history. The beam size adjusts the number of ways a decoder would evaluate to generate the most likely sequence. The vocabulary parses the emission matrix by mapping token to indices. Once the transcription has been predicted, the output of the beam search decoder modulecan have several functions. In some embodiments, the output of the beam search decoder modulecan be used to command effector devices such as opening an app on a smartphone. The output of the beam search decoder modulecan be connected to an effector device such that the effector device can respond to the brain signals of the subject.
466 In some embodiments, the encoder and decoder transformer blocks along with the beam search decoder modulecan predict motor commands based on decoded brain signals. The motor commands can be used to command effector devices to move a cursor on a computer device, decode handwritings to texts, and typing on a keyboard.
462 15 FIG. In some embodiments, the output from the transformer model can be used to generate speech using the speech synthesis module, as shown in. This can also be extended to an avatar of the subject displaying motor movements such as hand gestures and facial expressions based on the prediction of the motor commands.
14 FIG. 500 1410 500 452 504 503 505 456 500 1410 466 1410 1410 466 466 470 1410 470 470 470 In some embodiments, the NFM can include a reward model adapted to correct predicted transcripts.shows an embodiment of an NFM architectureadapted to correct predicted speech via a reward model, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data, which can generate neural embeddingsthat are fed through a decoderto provide an output associated with the neural data (e.g., phoneme probability), as described above. In this embodiment, the NFM architecturefurther includes a reward modelthat is fed the output from the beam search decoder module. In some embodiments, the reward modelcan be determined through reinforcement learning from human feedback, RLHF. The reward modelcan be used to rank the most likely transcription determined by the beam search decoder module. When the beam search decoder modulegenerates transcriptions, the reinforcement learning modulecan prompt the user to rank them based on the user's true intent. In some embodiments, the user can provide feedback in a plurality of categories including quality, coherence, and relevance. The feedback from the user can be used to train the reward model. The reinforcement learning modulecan guide the learning based on a mapping of the user feedback to a numerical reward. In some embodiments, the reinforcement learning modulecan include at least one of a policy gradient method, actor-critic, and proximal policy optimization. The reinforcement learning modulecan adjust at least one of the parameters of the NFM to maximize the expected cumulative reward, thus improving the performance of the NFM. In the constraint of a reinforcement learning problem, a policy can be the NFM that takes in user neural data and returns a transcription. An action space of this policy can be all the vocabulary of the NFM. An observation space can be a distribution of possible input sequences. The reward model can output a scalar reward. A reinforcement policy gradient method can then optimize the original NFM with respect to the reward model.
1510 500 500 452 504 503 505 456 500 470 470 1510 1510 462 462 462 462 1510 462 470 15 FIG. In some embodiments, the NFM can include a speech generation modeladapted to generate speech based on the transcription or directly from the embeddings generated by the transformer decoder blocks.shows an embodiment of an NFM architectureadapted to generate speech based on the predicted speech transcriptions, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data, which can generate neural embeddingsthat are fed through a decoderto output provide an output associated with the neural data (e.g., phoneme probability), as described above. In this embodiment, the NFM architecturefurther includes a reinforcement learning module. The transcription with the highest reward determined by the reinforcement learning modulecan be input to the speech generation model. The speech generation modelcan include a speech synthesis module. In some embodiments, the speech synthesis modulecan receive portions of the subject's digital history to generate speech in the style of the subject. For example, the portions of the subject's digital history can include speech utterances and/or videos of the subject speaking. The speech synthesis modulecan generate speech sounds and/or videos based on the recorded brain signals and the subject's digital history. The speech synthesis modulecan include at least one of a pre-trained diffusion speech generative model and a user-specific speech generative model. The speech generation modelcan be connected to a speaker device through which the speech synthesis modulecan vocalize the transcription with the highest reward as determined by reinforcement learning module.
16 FIG.A 500 500 452 504 503 1614 500 1613 1614 1612 1611 1602 1601 1611 1611 1612 In some embodiments, the NFM can include a generative speech module adapted to generate speech based on aligning neural embeddings with speech embeddings.shows an embodiment of an NFM architectureadapted to align neural embeddings with speech embeddings to generate speech, in accordance with example embodiments. In the depicted embodiment, the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data, which can generate neural embeddings. In this embodiment, the NFM architecturegenerates speech embeddingsfrom the neural embeddingsthat are fed into a speech encoder model, which is used to generate a time-frequency representationfrom the neural data, which are then utilized to guide a generative speech module. The aligned neural embeddings can be used to generate speech using stable diffusion. In some embodiments, the neural encoder modelcan receive neural data and the corresponding speech data of a user's past speech utterances (e.g., audio recordings). The speech data can be transformed to a time-frequency representation, such as a mel-spectrogram. Speech embeddings can then be extracted by passing the time frequency representationto a automatic speech recognition module(e.g., wav2vec2).
452 1601 The neural embeddings can be extracted using one of the previously discussed encoder modules (e.g., encoder module). The parameters of the neural encoder modelcan be updated using a contrastive loss that increases the similarity between the neural embeddings and the speech embeddings. Increasing the similarity between certain neural embeddings and speech embeddings can align them closely such that the neural embeddings can be interpreted based on proximity to certain speech embeddings.
1601 1602 1602 1603 The resulting neural embeddings from neural encoder modelthat have been closely aligned to speech embeddings can be used to guide a generative speech module. The generative speech modulecan use stable diffusion techniques to generate a time-frequency representation of the corresponding speech by denoising a latent time-frequency representation of a randomized speech using the neural embeddings over several iterations. The final generated speech time-frequency representation can then be inversely transformedback to the original domain and played back from a speaker device.
1601 1601 1601 1601 1601 1601 1601 1601 In some embodiments, the neural encoder modelcan be a multitask neural encoder model. The multitask neural encoder modelcan be trained on a predetermined set of tasks at each time step. The multitask modelcan have a loss function associated with every task for which it has to optimize. During training, similar tasks that are easier to decode can be grouped together and introduced to the multitask modelearly on. More difficult tasks can be grouped and introduced later in the training of the multitask model. This staged introduction can enable the multitask modelto learn a neural embedding representation that can be generalized over a variety of different tasks by mimicking how a human learns. For example, the neural encoder modelcan be trained on tasks that have distinct differences before being trained on tasks that have subtle differences.
1601 1601 1601 1601 The multitask modelcan receive a mixed set of tasks such that the modelcan optimize via a weighted loss function over the set of tasks. The weights of each loss function can be adjusted adaptively by reducing the gradients on easier tasks so that the modelcan learn more difficult tasks. When the gradient is reduced on an easier task, the error between the predicted speech and the actual speech is reduced, which can help the neural encoder modelto proceed to more difficult tasks.
1601 1601 1601 1601 2101 In some embodiments, the multitask modelcan have more parameters than the number of data samples used to train it. This can lead to the multitask modelbecoming overconfident in its predictions and not generalizing to new unseen tasks. To combat the overconfidence of the multitask model, a variety of regularization and data augmentations can be applied to improve the generalization of the multitask model. The amount of regularization can also be increased as the training progress to avoid overfitting to the training dataset (). The data augmentation can include adding noise to the data and learning how to denoise it prior to generating speech or movement.
1601 1601 In some embodiments, the data can be augmented by replacing a random portion of the microelectrode array data with its noisier self. This can help make the multitask modelrobust to variabilities in the electrode array. For example, a robust multitask modelcan predict a representation based on the electrode array with noisy electrode data.
1601 1601 In some embodiments, original microelectrode array data corresponding to a certain action, such as speaking the word “father” can be augmented by adding a weighted mix of one or more samples with different labels wherein the subject was speaking different words. This can lower the confidence of the multitask modelin producing the correct label for the original sample and increase its generalization. As such, the confidence of the neural encoder modelcan be lowered to mitigate the potential of overfitting to noise in the neural training data.
1601 1601 1601 1601 In some embodiments, a portion of the original microelectrode array data can be replaced by a portion of a different sample. This can introduce more intricate confusion to the multitask modelthat can make it more difficult for the multitask modelto overfit to noise in the neural data. With portions from other neural samples, the multitask modelcan learn the important portions of the neural signals instead of the irrelevant portions which increases the generalization of the multitask model.
1601 1601 1601 1601 1601 1601 1601 1601 1601 1601 In some embodiments, the neural encoder modelcan be used to perform zero-or few-shot classification. Zero-shot classification is when a model that previously trained on several tasks can be used as an encoder to extract representations from unseen data for unseen tasks. Few-shot classification is when a model that previously trained on several tasks can be used as an encoder to extract representations from data for tasks seen a few times. As the neural encoder modelcan be more robust, the representations can be stronger which enables deploying the neural encoder modelwithout having to fine-tune it on new data. For example, neural encoder modelcan have learned the neural embeddings for a few words. Instead of teaching the neural encoder modelnew words, a vocabulary of new words can be created using the existing neural encoder model. At inference, the new words can be compared against the existing vocabulary and the additional new words using k-nearest neighbors. This can be extended to motor decoding wherein the neural encoder modelcan be used to extract new gestures that it has not seen before. Through creating a database of new labeled gestures, the neural encoder modelcan find nearest gestures to an unseen gesture at inference. For example, considering gesture states including up, down, left, and right, if the neural encoder modeldetermines that the neural embedding is nearest the “up” gesture state than any of the other gesture states, then neural encoder modelcan predict the “up” gesture state is being selected.
1601 1601 1601 1601 1601 In some embodiments, the knowledge learned from the multitask neural encoder modelcan be taught to a smaller model that replicates one or more of the behaviors of the neural encoder modelusing knowledge distillation. This smaller model can be useful in deploying the neural encoder modelon mobile and edge devices where computation resources can be limited and, in some cases, can also help improve the neural encoder modelgeneralizations. A subset of the samples that correspond to desired tasks can be used for distillation of the smaller model. During training, the samples can be passed to both the neural encoder modeland the smaller model, where a loss function can be defined as the difference between the predictions of the two models. During distillation, the parameters of the smaller model can be updated accordingly.
500 500 500 500 452 504 503 1614 500 1653 1614 1652 1652 1602 1652 1653 1601 1614 1653 16 FIG.B 16 FIG.B 16 FIG.A 16 FIG.A The NFM architecturecan be generalized to video generation as well.shows an embodiment of an NFM architectureadapted to align neural embeddings with video embeddings to generate video, in accordance with example embodiments. The embodiment of the NFM architectureshown indiffers from the embodiment shown inin that it is adapted for video. In particular, the depicted embodiment of the NFM architectureincludes an encoderthat receives input embeddingsgenerated from a low-dimension projectionof neural data, which can generate neural embeddings. In this embodiment, the NFM architecturegenerates video embeddingsfrom the neural embeddings, which are fed into a video encoder model. The output of the video encoder modelcan be utilized to guide a generative speech module. The generalization can be done by using video data from the subject's digital history making hand gestures or speaking. Video encoder modelcan extract video representations from the subject's digital history to generate one or more video embeddings. The neural encoder modelcan perform contrastive loss on the neural embeddingand the video embeddingto align neural embeddings with certain video embeddings. The aligned neural embeddings can be used to generate video of the subject based on predictions of movement using stable diffusion, as described in. For example, neural embedding can be used alongside video embeddings to generate an avatar of the subject's face speaking displayed on a monitor. In another embodiment, neural embeddings can be used alongside video embedding to generate a video of the subject's moving their arms.
17 FIG. 1 FIG. 1700 1700 1700 1700 110 130 1700 1700 shows an illustrative methodof decoding speech. In various embodiments, the methodcan be embodied as software, hardware, firmware, and various combinations thereof. In various embodiments, the methodcan be executed by and/or between a variety of different devices or systems. For example, various combinations of steps of the methodcan be executed by the neural device, the external device, and/or other devices communicatively coupled to the aforementioned components. In various embodiments, the system(s) executing the methodcan utilize distributed processing, parallel processing, cloud processing, and/or edge computing techniques. For brevity, the execution of the methodis generally described below; however, it should be understood that the functions can be individually or collectively executed by one or multiple devices or systems described in connection with.
1710 180 Stepincludes receiving recorded brain signals via a microelectrode array. The microelectrode array can include non-penetrating cortical surface microelectrodes. In some embodiments, the microelectrode array can be electrode array.
1720 502 450 Stepcan include extracting one or more features from the recorded brain signals. In some embodiments, the extracting can include dividing the three-dimensional array of recorded brain signals into a plurality of patches. The extracting can be performed by the feature extraction module.
1730 450 Stepcan include converting the one or more extracted features into one or more feature embeddings. In some embodiments, the one or more feature embeddings can include at least a temporal representation and a spatial representation of the recorded brain signals. The conversion can be performed by the feature extraction module.
458 452 1700 encoders, the one or more feature embeddings. The one or more encoders can include one or more attention modules. The one or more attention modules can weigh the importance of elements in an input and adjust their influence on the output. The transforming can be performed by a feedforward network module. In some embodiments, the one or more encoders can include encoder module. In some embodiments, the transforming can include applying a temporal attention block to the temporal representation and a spatial attention block to the spatial representation. In some embodiments, the methodcan further include concatenating at least the temporal attention and the spatial attention.
460 In some embodiments, the one or more encoders can receive one or more subject-specific embeddings. In some embodiments, the subject-specific embeddings can be generated by the subject-specific encoder module. In some embodiments, the one or more encoders can transform the one or more subject-specific embeddings. In some embodiments, a weighted sum can be applied to the transformed extracted feature embeddings and the transformed subject-specific embeddings. In some embodiments, the output of the weighted sum can be concatenated with a user embedding representing a profile of the user.
1750 456 Stepcan include predicting, by one or more decoders, phonemes based on the one or more transformed feature embeddings. In some embodiments, the one or more decoders can include decoder module.
1760 466 466 466 462 Stepcan include predicting speech based on the predicted phonemes. In some embodiments, the predicted speech can be generated by the beam search decoder module. In some embodiments, the predicting speech can include aligning, by a beam search decoder, the predicted phonemes based on one or more language constraints. In some embodiments, the beam search decoder modulecan generate a predicted transcription based on the aligning. In some embodiments, the beam search decoder modulecan be a connectionist temporal classification beam search decoder. In some embodiments, the speech can be vocalized by a speech synthesis module.
1700 200 It should further be noted that although the functions and/or steps of the methodare depicted in a particular order or arrangement, the depicted order and/or arrangement of steps and/or functions is simply provided for illustrative purposes. Unless explicitly described herein to the contrary, the various steps and/or functions of the processcan be performed in different orders, in parallel with each other, in an interleaved manner, and so on.
18 FIG. 1800 1800 435 480 410 1800 500 1800 1700 1800 1800 1807 1804 1806 1808 1812 1810 shows a block diagram of an example computing devicethat implements various features and processes, according to example embodiments of this disclosure. For example, computing devicecan function as the server, the clients, the neural interface, or a portion or combination thereof in some embodiments. Additionally, the computing devicecan partially or wholly host and deploy NFM architecture. The computing devicecan also perform one or more steps of the method. The computing devicecan be implemented on any electronic device that runs software applications derived from compiled instructions, including without limitation personal computers, servers, smart phones, media players, electronic tablets, game consoles, email devices, etc. In some implementations, the computing deviceincludes one or more processors, one or more input devices, one or more display devices, one or more network interfaces, and one or more computer-readable media. Each of these components can be coupled by a bus.
1806 1807 1804 1810 1812 1807 Display deviceincludes any display technology, including but not limited to display devices using liquid crystal display (LCD) light-emitting diode (LED) technology. Processor(s)uses any processor technology, including but not limited to graphics processors and multi-core processors. Input deviceincludes any known input device technology, including but not limited to a keyboard (including a virtual keyboard), mouse, track ball, and touch-sensitive pad or display. Busincludes any internal or external bus technology, including but not limited to ISA, EISA, PCI, PCI Express, USB, Serial ATA or FireWire. Computer-readable mediumincludes any non-transitory computer readable medium that provides instructions to processor(s)for execution, including without limitation, non-volatile storage media (e.g., optical disks, magnetic disks, flash drives, etc.), or volatile media (e.g., SDRAM, ROM, etc.).
1812 1814 1804 1806 1812 1810 1816 Computer-readable mediumincludes various instructionsfor implementing an operating system (e.g., Mac OS®, Windows®, Linux). The operating system can be multi-user, multiprocessing, multitasking, multithreading, real-time, and the like. The operating system performs basic tasks, including but not limited to: recognizing input from input device; sending output to display device; keeping track of files and directories on computer-readable medium; controlling peripheral devices (e.g., disk drives, printers, etc.) which can be controlled directly or through an I/O controller; and managing traffic on bus. Network communications instructionsestablish and maintain network connections (e.g., software for implementing communication protocols, such as TCP/IP, HTTP, Ethernet, telephony, etc.).
1818 1820 Speech decoding instructionscan include instructions that implement the disclosed process for decoding speech based on recorded brain signals, as described throughout this disclosure. Application(s)can comprise an application that uses or implements the processes described herein and/or other processes. The processes can also be implemented in the operating system.
This disclosure is not limited to the particular systems, devices, and methods described, as these can vary. The terminology used in the description is for the purpose of describing the particular versions or embodiments only and is not intended to limit the scope of the disclosure.
The following terms shall have, for the purposes of this application, the respective meanings set forth below. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art. Nothing in this disclosure is to be construed as an admission that the embodiments described in this disclosure are not entitled to antedate such disclosure by virtue of prior invention.
As used herein, the singular forms “a,” “an,” and “the” include plural references, unless the context clearly dictates otherwise. Thus, for example, reference to a “module” is a reference to one or more modules and equivalents thereof known to those skilled in the art, and so forth.
As used herein, the term “about” means plus or minus 10% of the numerical value of the number with which it is being used. Therefore, about 50 nm means in the range of 45 nm to 55 nm.
As used herein, the term “consists of” or “consisting of” means that the device or method includes only the elements, steps, or ingredients specifically recited in the particular claimed embodiment or claim.
In embodiments or claims where the term “comprising” is used as the transition phrase, such embodiments can also be envisioned with replacement of the term “comprising” with the terms “consisting of” or “consisting essentially of.”
As used herein, the term “subject” includes, but is not limited to, humans and non-human vertebrates such as wild, domestic, and farm animals.
While the present disclosure has been illustrated by the description of example embodiments thereof, and while the embodiments have been described in certain detail, it is not the intention of the Applicants to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. Therefore, the disclosure in its broader aspects is not limited to any of the specific details, representative devices and methods, and/or illustrative examples shown and described. Accordingly, departures can be made from such details without departing from the spirit or scope of the Applicant's general inventive concept.
With respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity.
In addition, even if a specific number is explicitly recited, those skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number (for example, the bare recitation of “two recitations,” without other modifiers, means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, et cetera” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (for example, “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, et cetera). In those instances where a convention analogous to “at least one of A, B, or C, et cetera” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (for example, “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, et cetera). It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, sample embodiments, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”
In addition, where features of the disclosure are described in terms of Markush groups, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual member or subgroup of members of the Markush group.
Various of the above-disclosed and other features and functions, or alternatives thereof, can be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art, each of which is also intended to be encompassed by the disclosed embodiments.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
September 5, 2025
August 27, 2026
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.