System and Method for Preparing Practice Tests Using Machine Learning Models

This application claims the benefit under 35 U.S.C. § 120 as a continuation of application Ser. No. 18/972,582, filed Dec. 6, 2024, the entire content of which is hereby incorporated by reference for all purposes as if fully set forth herein. The applicant hereby rescinds any disclaimer of claim scope that may have occurred in the prosecution of the priority application and advises the USPTO that the claims of this application may be broader than in any priority application.

A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright or rights whatsoever. ©2024 Quizlet, Inc.

One technical field of the present disclosure is computer-implemented systems and methods for generating practice tests using machine learning models. Another technical field is data science, using a collection of machine learning models, each trained for specific purposes.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

Traditional online learning systems have made targeted instructions for students in a wide variety of subjects and learning modes more accessible than ever before. Outside of traditional educational institutions, diverse groups of users worldwide can learn almost anything without ever setting foot in a classroom. While online platforms have made significant advancements in delivering educational content, creating customized assessments and practice tests remains a challenge. Current systems lack effective software or technical means to generate tests for computer-based presentations over networks, often relying on static question banks or requiring significant manual effort to design tests that align with specific user-provided study materials. This limitation results in a lack of personalized and contextualized test preparation resources for users, requiring excessive use of computing resources and storage to draft and revise tests.

In conventional methods, generating practice tests typically involves educators or content creators manually selecting or writing questions and answers. This process can be time-consuming and may not adequately reflect the specific focus of a learner's study material. Consequently, students may practice questions that are either too generic or irrelevant to their particular needs. Moreover, manually crafted tests lack the dynamic adaptability to quickly incorporate new information or adjust to the user's evolving understanding of a subject.

Automated methods have been proposed to address these challenges. Still, they often involve simple rule-based systems or limited keyword-matching techniques that fail to capture the nuances of complex educational content. These traditional techniques do not generate questions and answers that are contextually appropriate and accurately aligned with the source material. As a result, there is a need for a more sophisticated solution that can automatically generate high-quality practice tests based on user-provided study materials.

Based on the foregoing, the relevant technical fields have developed a pressing need for improved techniques in creating customized practice tests.

The appended claims may serve as a summary of the invention.

In the following description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

The text of this disclosure, in combination with the drawing figures, is intended to state in prose the algorithms that are necessary to program the computer to implement the claimed inventions at the same level of detail that is used by people of skill in the arts to which this disclosure pertains to communicate with one another concerning functions to be programmed, inputs, transformations, outputs and other aspects of programming. That is, the level of detail set forth in this disclosure is the same level of detail that persons of skill in the art normally use to communicate with one another to express algorithms to be programmed or the structure and function of programs to implement the inventions claimed herein.

One or more different inventions may be described in this disclosure, with alternative embodiments to illustrate examples. Other embodiments may be utilized, and structural, logical, software, electrical, and other changes may be made without departing from the scope of the embodiments that are specifically described. Various modifications and alterations are possible and expected. Some features of one or more of the inventions may be described with reference to one or more embodiments or drawing figures, but such features are not limited to usage in the one or more embodiments or figures with reference to which they are described. Thus, the present disclosure is neither a literal description of all embodiments of one or more of the inventions nor a listing of features of one or more of the inventions that must be present in all embodiments.

Headings of sections and the title are provided for convenience but are not intended as limiting the disclosure in any way or as a basis for interpreting the claims. Devices that are described as in communication with each other need not be in continuous communication with each other unless expressly specified otherwise. In addition, devices that communicate with each other may communicate directly or indirectly through one or more intermediaries, logical or physical.

A description of an embodiment with several components in communication with one another does not imply that all such components are required. Optional components may be described to illustrate various possible embodiments and to illustrate one or more aspects of the inventions more fully. Similarly, although process steps, method steps, algorithms, or the like may be described in sequential order, such processes, methods, and algorithms may generally be configured to work in different orders unless specifically stated to the contrary. Any sequence or order of steps described in this disclosure is not a required sequence or order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously. The illustration of a process in a drawing does not exclude variations and modifications, does not imply that the process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred. The steps may be described once per embodiment but need not occur only once. Some steps may be omitted in some embodiments or occurrences, or some steps may be executed more than once in each embodiment or occurrence. When a single device or article is described, more than one device or article may be used in place of a single device or article. Where more than one device or article is described, a single device or article may be used instead of more than one device or article.

The functionality or features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other embodiments of one or more of the inventions need not include the device itself. Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be noted that embodiments include multiple iterations of a technique or multiple manifestations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of embodiments of the present invention in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved.

Systems and methods disclosed herein provide a specific technical solution for automatically generating practice tests from various educational content using machine learning models and advanced natural language processing (NLP) techniques. Thus, the systems and methods disclosed herein address the inefficiencies and complexities associated with manually creating practice tests, providing a practical application that saves time and enhances learning experiences.

Traditional practice test creation methods involve significant manual effort and subjectivity, resulting in inconsistent quality and relevance. Additionally, educators or students may review and organize large amounts of study material to create meaningful questions and answers. The proposed system addresses this problem by automating the extraction, generation, and validation of practice test questions, ensuring that the content is aligned with the source material and tailored to the learning objectives.

Systems and methods disclosed herein utilize text segmentation and contextual embedding analysis to determine questions and answers from provided content automatically. Such questions and answers are then stored in a question/answer (Q/A) bank database and can be used for practice tests.

In one embodiment, a system is configured to receive input data containing material for a practice test. The received input data can then be segmented into distinct sections based on syntactic and semantic analysis, allowing for precise identification of question and answer segments. This segmentation is enhanced by context-aware embeddings, which can capture the nuanced relationships between words and phrases. Using multi-level embeddings (e.g., word-level, sentence-level, and paragraph-level embeddings), the system can accurately differentiate between various text parts, such as questions, answers, and supporting explanations. This detailed segmentation provides a robust foundation for further processing and eliminates ambiguity in content classification.

Once the content is segmented, the system leverages fine-tuned large language models (LLMs) such as BERT or GPT to classify within segmented content questions and answers, thereby extracting such questions and answers from the content. Alternatively, if it is determined that content contains concepts but does not explicitly include questions and answers, the system is configured to generate questions and answers dynamically based on the content. The LLMs are trained on a specialized corpus of educational texts, ensuring that the generated questions maintain high relevance and accuracy. Furthermore, the system incorporates contextual attention mechanisms, allowing it to focus on the most significant parts of the text when generating questions. This dynamic generation process significantly reduces manual effort and produces high-quality questions that are aligned with the intended learning objectives.

Furthermore, in some implementations, the system is not only configured to extract answers from the content but also to generate answers based on the extracted questions and an available corpus of educational text. The system is then configured to perform a similarity comparison between the extracted answers and the generated answers. The comparison uses multi-level embeddings and a cosine similarity measure with attention-based weighting to ensure that the generated answers not only match the extracted content in terms of meaning but also reflect the context in which the questions are posed. This detailed similarity analysis provides a technical improvement over traditional methods, which often lack such fine-grained validation and rely on keyword matching or basic similarity measures.

The system also incorporates a FAISS (Facebook AI Similarity Search) index to efficiently retrieve relevant questions from a large repository of user-generated and system-generated content. The FAISS index is structured to support high-dimensional vector searches using quantization and partitioning techniques, enabling rapid retrieval of semantically similar questions. By combining this retrieval capability with generated questions, the system ensures that the practice test covers relevant topics, further enhancing its educational value.

After generating and retrieving questions, the system applies a multi-step post-processing mechanism. The post-processing includes re-ranking questions using a re-ranking algorithm, such as LambdaMART or a cross-encoder ranking model, to prioritize the most relevant and high-quality questions, as well as associate accurate answers based on their semantic similarity, contextual alignment, and determined difficulty level. Additionally, the post-processing includes using word embeddings and attention-based models to detect and remove duplicate questions and ensure that only accurate answers are retained. This filtering process ensures that the final practice test is cohesive, free from redundancies, and maintains a high standard of quality.

Furthermore, the system is configured to utilize cloud-based storage solutions, such as BigQuery and Cloud Spanner, to store and manage the generated and retrieved question banks. This allows for scalable management of large datasets and seamless integration with other educational platforms. The system's architecture supports asynchronous processing and caching, which improves the response time and reduces computational overhead, making it suitable for large-scale deployments.

The automated creation of practice tests not only saves educators and students substantial time but also ensures that the content is tailored to individual learning needs and goals. By automating the question generation and validation process using state-of-the-art NLP techniques and machine learning models, the system provides a practical solution that enhances learning efficiency and content relevance. The technical aspects, such as the use of multi-level embeddings, attention-based similarity measurement, and scalable cloud infrastructure, present clear technical improvements over manual or less sophisticated methods, demonstrating a concrete application of the technology.

1. A computer-implemented method comprising: using a server computer, obtaining a data input from a client computer; identifying a language and a subject associated with the data input; determining that the data input is an eligible test input based on the identified language being part of a predefined set of languages and the identified subject being part of a predefined set of subjects; determining that the eligible test input is a practice-test-type input using a classification model; executing a first inference stage of the classification model over a first selection from the practice-test-type input to generate a first prediction indicating that the first selection comprises a candidate question; transmitting to a generative model a first prompt comprising the first selection to output a suggested question; transmitting, to the generative model, a second prompt comprising the suggested question to output a suggested answer; selecting the suggested question as a practice test question; selecting the suggested answer as a practice test answer; and generating a practice test comprising a digital card, the digital card having a first graphical user element configured to display the practice test question and having a second graphical user element configured to display the practice test answer, wherein each of the classification model and the generative model comprising a trained machine-learning model. 2. The computer-implemented method of clause 1, further comprising: executing an embedding model over the candidate question to output a first word embedding corresponding to the candidate question; executing the embedding model over the suggested question to output a second word embedding corresponding to the suggested question; measuring a similarity between the first word embedding and the second word embedding; and assessing a suitability of the suggested question as the practice test question based on a similarity score being above a first selected threshold. 3. The computer-implemented method of clause 2, further comprising: executing a second inference stage of the classification model over a second selection from the practice-test-type input to generate a second prediction indicating that the second selection comprises a candidate answer; executing the embedding model over the candidate answer to output a third word embedding corresponding to the candidate answer; executing the embedding model over the suggested answer to output a fourth word embedding corresponding to the suggested answer; measuring a similarity between the third word embedding and the fourth word embedding; and assessing a suitability of the suggested answer as the practice test answer based on a similarity score being above a second selected threshold. 4. The computer-implemented method of clause 3, wherein the first selection comprises a plurality of candidate questions. 5. The computer-implemented method of clause 3, wherein the second selection comprises a plurality of candidate answers. 6. The computer-implemented method of clause 3, wherein the first prompt, the second prompt, or a combination thereof further comprises the second selection. 7. The computer-implemented method of clause 1, further comprising segmenting the data input into one or more sections using a segmentation model and determining the eligibility for each section from the one or more sections based on identified language and subject for each section. 8. The computer-implemented method of clause 7, wherein the segmentation model is a rule-based model configured to extract text data from the data input and identify the one or more sections by detecting specific patterns within the text data, wherein the specific patterns include at least one of: beginning and end of paragraphs, beginning and end of lists, text separated by spaces, changes in indentation levels, presence of bullet points or numbered sequences, heading or subheading styles, line breaks or page breaks, quotation marks or block quote indicators, punctuation patterns such as colons or ellipses, variations in sentence length or structure, phrases, or enumerated answer options; and wherein the patterns are used to delimit boundaries between the one or more sections. 9. The computer-implemented method of clause 8, wherein the segmentation model is configured to utilize syntactic and semantic feature extraction techniques to identify logical boundaries between the one or more sections. 10. The computer-implemented method of clause 7, wherein the eligibility model is a large language model configured to determine the language and the subject of each section from the one or more sections. 11. The computer-implemented method of clause 1, wherein the classification model is a large language model trained on a labeled training dataset, the labeled training dataset comprising training text labeled with question-specific labels when corresponding to questions and labeled with answer-specific labels when corresponding to answers. 12. The computer-implemented method of clause 1, wherein the classification model is an Information Maximizing Generative Adversarial Network (Info-GAN) trained on a generated synthetic training dataset. 13. The computer-implemented method of clause 12, wherein the classification model is a foundation model pre-trained on a large-scale language corpus and further fine-tuned on a labeled training dataset, the labeled training dataset selected based on the identified subject of the practice-test-type input. 14. The computer-implemented method of clause 1, wherein the classification model is further configured to handle ambiguous sections for identifying questions and answers of the practice-test-type input by using uncertainty estimation techniques to quantify a confidence level of text classification model's predictions, wherein the uncertainty estimation techniques include at least one of Monte Carlo Dropout or Bayesian Neural Networks, and wherein the ambiguous sections with low confidence scores are flagged. 15. The computer-implemented method of clause 1, further comprising using the classification model to detect contextual shifts between sections for identifying questions and answers, wherein the contextual shifts are detected by analyzing changes in one or more linguistic features. 16. The computer-implemented method of clause 3, wherein the first word embedding comprises a first set of multi-level embeddings and the second word embedding comprises a second set of multi-level embeddings, each set including word-level embeddings for capturing an individual word meaning, sentence-level embeddings for capturing contextual meaning within a sentence, and paragraph-level embeddings for capturing contextual meaning across multiple sentences, and wherein the first and second sets of multi-level embeddings are used to compute the similarity by comparing embeddings at different levels of granularity to identify semantic relationships between the candidate question and the suggested answer. 17. The computer-implemented method of clause 16, wherein the similarity is determined using a cosine similarity. 18. The computer-implemented method of clause 3, further comprising using an attention model to weigh an importance of individual words or phrases within the candidate question and the suggested question, wherein the attention model is configured to assign higher weights to the individual words or the phrases that are contextually significant, and wherein the similarity between the candidate question and the suggested question is calculated based on their respective attention-weighted the first and second word embeddings. 19. The computer-implemented method of clause 3, wherein generating the suggested question comprises: accessing a question-answer database containing a plurality of records corresponding to questions having associated answers, wherein each question in the question-answer database includes an associated question word embedding; measuring a question similarity between the first word embedding and the question word embedding for each question; determining a matching question in the question-answer database for the question, based on the measured question similarity being above a third selected threshold; and selecting a corresponding question to the matching question as the suggested question. 20. The computer-implemented method of clause 3, wherein generating the suggested question comprises: accessing a question-answer database containing a plurality of records corresponding to questions having associated answers, wherein each answer in the question-answer database includes an associated answer word embedding; measuring an answer similarity between the third word embedding and the answer word embedding for each question; determining a matching answer in the question-answer database, based on the measured answer similarity being above a third selected threshold; and selecting a corresponding question to the matching answer as the suggested question. The disclosure encompasses the subject matter of the following numbered clauses:

1 FIG. 100 110 130 120 130 160 illustrates a system environmentdepicts interconnected components such as a user deviceinteracting with application server computervia a network. Further, application server computermay be communicatively coupled to database.

110 130 110 110 110 110 110 User devicecan be any suitable device capable of communicating with application server computervia a web interface or other communication methods. Additionally, user devicemay refer to any device configured to process data and interact with users. Examples of user deviceinclude, but are not limited to, a personal computer, desktop computer, workstation, server, laptop, tablet, mobile phone (e.g., smartphone), smart glasses, virtual reality (VR) headset, augmented reality device, internet-of-things (IoT) device, or any other type of suitable computing device. User devicemay feature a user interface such as a display, microphone, camera, keypad, or other input/output equipment that facilitates user interaction. Furthermore, user devicemay include a hardware processor, memory, and/or circuitry configured to perform any functions or actions as described in this disclosure. For convenience, user devicemay be interchangeably referred to as a computing device within the present description.

110 110 100 120 110 130 110 User deviceincludes a processor in signal communication with a network interface and memory. The memory stores software instructions that, when executed by the processor, enable the device to perform one or more operations as described herein. User deviceis configured to communicate with other devices and components within system environmentvia network. A user may employ user deviceto transmit data, such as test-related data, to application server computer. Additionally, user devicecan utilize various types of interfaces to upload study materials, request the generation of practice tests, and receive the resulting practice test content. The interfaces may include web-based applications accessed through a browser, mobile applications, or standalone programs, allowing users to seamlessly interact with the system and manage their practice test content.

110 130 110 130 The communication between user deviceand application server computercan occur through a web-based communication interface, such as a standard web browser (e.g., Chrome, Safari, Firefox) or a dedicated web application. The communication interface between user deviceand application server computermay support various functions, including document uploads, selecting test generation parameters, and viewing or interacting with the generated practice test content. For smartphones and tablets, a mobile application can provide a streamlined and improved interface that supports the same functionalities as the web-based version. Similarly, a standalone desktop program may be used to access the system without needing a browser, offering additional features such as offline access or integration with other software.

112 The communication interface provided on a webpage or application can dynamically adapt based on the input datareceived from the user. For example, when a user uploads study materials recognized as text documents, the interface may present options for selecting specific chapters or topics for generating questions. If the uploaded material is in a different format, such as a list of practice questions, the interface can switch to displaying options for verifying and editing the content of those questions or for generating new answers based on them. Similarly, if a user inputs metadata, such as subject or difficulty level, the interface can provide a different set of options, such as selecting the type of questions to generate or configuring the layout of the practice test.

Additionally, the interface can modify its layout and functionalities depending on the type of uploaded content, such as lecture notes, a textbook chapter, or a set of previous exam questions. For example, when lecture notes are uploaded, the interface might enable segmentation options for selecting specific sections to generate targeted questions. In contrast, if a whole textbook chapter is uploaded, the interface could offer topic-based options for breaking down the chapter into manageable units to generate comprehensive practice tests. The system could use distinct interface layouts for different types of questions, such as multiple-choice questions, short-answer questions, or descriptive questions, based on the content and user preferences.

The web-based interface can also incorporate context-sensitive changes in response to real-time user inputs or the results of automated content analysis performed by the server. For example, if the user uploads content that the server determines is unsuitable for generating practice questions (e.g., images or unsupported languages), the interface can provide feedback or suggest alternative options, such as requesting additional text input or uploading compatible file types.

130 130 The flexibility in interface design disclosed herein enables application server computerto deliver a customized user experience, making it adaptable for a range of devices, user preferences, and varying data inputs. Whether accessed through a mobile application, a standalone program, or a dynamic webpage, the user interface is configured to integrate with the application server computer.

112 112 130 112 Input datacan be any data in the form of educational or instructional content provided by the user to the system for the purpose of generating practice tests or other learning materials. It may include various types of documents, files, or multimedia content, such as text, images, or formatted data, and can be presented in multiple formats. Examples of input datainclude text documents such as study notes, lecture transcripts, textbook excerpts, chapters, articles, and essays, which can be in formats like .txt, .docx, .pdf, or .rtf, containing written content that application server computercan process to extract information and generate practice questions. Input datamay also include pre-existing practice tests or lists of questions that the user wants to incorporate into a new practice test or further refine, such as question-and-answer pairs, multiple-choice questions, or fill-in-the-blank questions. In addition, educational content files, such as PowerPoint presentations, spreadsheets, or structured learning materials outlining specific concepts, topics, or learning objectives, can serve as input data.

112 112 110 Moreover, input datacan include metadata, such as subject category (e.g., Math, Physics), topic (e.g., Algebra, Calculus), difficulty level, or intended audience (e.g., high school students, college students), which provides additional context for the input content. The input data can also incorporate images and diagrams, such as labeled diagrams, charts, or infographics that support learning. While these visual elements may not be directly used for question generation, they can be referenced or analyzed to enhance the practice test content. Multimedia content, such as audio files, video recordings, or other multimedia elements that contain spoken explanations or visual demonstrations of concepts, may also be included. Structured data formats, such as . csv or .json files that outline specific information, like a list of questions with corresponding answers or metadata fields for classification, can also be provided as input data. Furthermore, input datacan include URLs or links to online articles, research papers, or external content that the system can access, parse, and use to generate related practice test content. The input data may be submitted by the user through various interfaces, such as web browsers, mobile applications, or standalone software, depending on the capabilities of user device.

110 130 120 120 120 120 120 120 In various embodiments, user deviceis configured to communicate with application server computervia network. Networkmay be any suitable type of wireless and/or wired network. Networkmay be connected to the Internet or a public network. Networkmay include all or a portion of an Intranet, a peer-to-peer network, a switched telephone network, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), a wireless PAN (WPAN), an overlay network, a software-defined network (SDN), a virtual private network (VPN), a mobile telephone network (e.g., cellular networks, such as 4G or 5G), a plain old telephone (POT) network, a wireless data network (e.g., Wi-Fi, WiGig, WiMAX, etc.), a long-term evolution (LTE) network, a universal mobile telecommunications system (UMTS) network, a peer-to-peer (P2P) network, a Bluetooth network, a near-field communication (NFC) network, and/or any other suitable network. Networkmay include fiber optics, optical fibers, and the like to implement quantum communication channels. Networkmay be configured to support TCP/IP, UDP, or functionally equivalent communication protocols.

130 170 170 The application server computergenerally comprises a suitable hardware system designed for processing received data and generating a practice test. The practice testmay consist of a set of questions that can be displayed to the user. These questions may have corresponding answers, which may or may not be shown to the user, depending on the implementation. In some cases, the questions can include various formats, such as fill-in-the-blank, multiple-choice, true/false, matching, or short-answer questions, among others. The answers are designed to correspond accurately to their respective questions.

130 130 130 In certain embodiments, application server computermay be implemented by a cluster of computing devices, such as virtual machines. For example, application server computermay be implemented by a plurality of computing devices using distributed computing and/or cloud computing systems in a network. In certain embodiments, application server computermay be configured to provide services and resources (e.g., data and/or hardware resources as described herein, etc.) to other components and devices.

130 130 Application server computermay comprise one or more processors operably coupled with a network interface and a memory. The one or more processors comprise electronic circuitry, including, but not limited to, state machines, one or more CPU chips, logic units, cores (e.g., a multi-core processor), FPGAs, ASICs, or DSPs. For example, one or more processors may be implemented in cloud devices, servers, virtual machines, and the like. The one or more processors may be a programmable logic device, a microcontroller, a microprocessor, or any suitable number and combination of the preceding. The one or more processors are configured to process data and may be implemented in hardware or software. The one or more processors may include an ALU for performing arithmetic and logic operations. The one or more processors may register the supply operands to the ALU and store the results of ALU operations. The one or more processors may further include a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers, and other components. The one or more processors are configured to implement various software instructions to perform the operations of application server computerdescribed herein. In this way, the one or more processors may be a special-purpose computer designed to implement the functions disclosed herein. The one or more processors may be configured to perform one or more operations of any of the methods described in this disclosure.

130 130 The network interface of application server computeris configured to enable wired and/or wireless communications. The network interface may be configured to communicate data between application server computerand other devices, systems, or domains. For example, the network interface may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-Wave interface, a RFID interface, a Wi-Fi interface, a LAN interface, a WAN interface, a MAN interface, a PAN interface, a WPAN interface, a modem, a switch, and/or a router. The one or more processors may be configured to send and receive data using the network interface. The network interface may be configured to use any suitable type of communication protocol.

130 131 160 130 130 132 133 134 136 138 140 142 144 Application server computerincludes multiple subcomponents responsible for different stages of practice test generation. These subcomponents include the practice test service, which manages the workflow and coordinates interactions between the models and a database, and various machine learning models of application server computer, each performing a specific task. In some implementations, application server computerincludes an optional segmentation model, an eligibility model, a classification model, an extraction model, a generative model, an optional embedding model, an optional attention model, and a ranking model.

130 132 112 112 112 132 In some cases, application server computerincludes a segmentation modelconfigured to partition input datainto one or more sections, enabling the system to apply further processing steps such as eligibility checks and content classification on a more granular level. For example, if provided input dataincludes some sections that are eligible for practice test generations and other sections that are not eligible for practice test generation, the input data can be partitioned into different sections, and only sections that are eligible for practice test generation can be used. Additionally, or alternatively, if input dataincludes a first section listing various concepts, and a second section listing questions and answers, such sections may be identified by segmentation model.

132 Segmentation modelcan be implemented as a rule-based model that detects specific patterns within the text data to identify logical boundaries between sections. These patterns include, but are not limited to, the beginning and end of paragraphs, beginning and end of lists, changes in indentation levels, text separated by spaces, the presence of bullet points or numbered sequences, heading or subheading styles, line breaks or page breaks, quotation marks or block quote indicators, punctuation patterns such as colons or ellipses, variations in sentence length or structure, and key phrases or keywords that signal the start or end of a section. The model can also recognize enumerated answer options, enabling the identification of questions and corresponding answers within the input data.

132 132 In some cases, segmentation modelcan also be configured to use syntactic and semantic feature extraction techniques, including part-of-speech tagging and sentence boundary detection. These techniques enable segmentation modelto analyze the grammatical structure and the semantic content of the text, improving the identification of logical boundaries between sections, especially in complex or unstructured data.

132 To implement segmentation model, a combination of text-processing libraries and machine-learning models (MLMs) can be employed. The rule-based approach could be developed using text processing tools such as Python's Natural Language Toolkit (NLTK) or spaCy, which offer built-in capabilities for detecting paragraph breaks, sentence boundaries, and punctuation patterns. Regular expressions can also be utilized to detect specific markers like indentation changes, lists, and bullet points. For syntactic and semantic feature extraction, models like BERT (Bidirectional Encoder Representations from Transformers) or similar transformer-based architectures can be leveraged to perform part-of-speech tagging and analyze the contextual meaning of text segments. BERT's ability to generate contextual embeddings can be particularly useful for identifying sentence boundaries and understanding the relationships between sentences, which aids in segmenting text into coherent sections. Additionally, the segmentation model can incorporate custom-trained machine learning models that classify different types of segments (e.g., questions, answers, headings) based on labeled training data, which further improves the segmentation accuracy for various document types.

132 132 132 In some cases, segmentation modelcan include an unsupervised clustering algorithm such as K-means or a hierarchical clustering algorithm. Such algorithms can be used in conjunction with text embeddings to group similar sentences or paragraphs, allowing segmentation modelto dynamically identify and segment cohesive blocks of text even when explicit patterns are not present. Segmentation modelcan also be made adaptive by employing reinforcement learning techniques, where the model learns segmentation strategies based on user feedback or predefined success metrics.

130 133 133 112 133 133 112 133 133 Further, in various embodiments, application server computerincludes an eligibility model. Eligibility modelis configured to determine whether a given input data, or its segmented sections, are suitable for generating practice test content. In various embodiments, eligibility modelserves as a gatekeeper by filtering out ineligible inputs. Eligibility modelevaluates the input databy identifying a language and a subject associated with each section or the entire input. Eligibility modelthen determines if the identified language is part of a predefined set of supported languages and if the identified subject is part of a predefined set of recognized subjects. This eligibility modelensures that only relevant content is processed further, filtering out content that may not be appropriate for practice test generation, such as data in unsupported languages or subject areas.

The eligibility model can be implemented using a large language model that is capable of performing language detection and subject classification. For example, transformer-based models such as BERT or GPT, which have been pre-trained on a large-scale corpus of multilingual and multi-domain text, can be fine-tuned for this specific task. The language detection component of the model can analyze the input text and classify it into one of several supported languages, ensuring that only texts in recognized languages are considered eligible. This detection can be performed using a multilingual model like XLM-R (Cross-lingual Language Model-RoBERTa) that supports various languages and provides robust language classification.

133 133 For subject classification, eligibility modelcan be further fine-tuned on labeled training data that categorizes text inputs into specific subjects, such as Mathematics, Biology, History, Physics, and the like. During training, eligibility modelis configured to learn to recognize specific keywords, phrases, and contextual indicators that are characteristic of each subject area. For instance, if a text input contains terms like “derivative,” “integral,” and “calculus,” the model could classify the content as being related to Mathematics. The subject classification component can be developed using a hierarchical classification approach, where broader categories (e.g., Science, Humanities) are identified first, followed by more granular subcategories (e.g., Chemistry, Physics, and History).

133 133 In addition to using pre-trained language models, eligibility modelcan also use rule-based techniques for initial filtering, such as detecting the presence of specific metadata or tags provided by the user. For example, if the input data includes metadata indicating the subject and language, the model can use these attributes to perform a quick eligibility check before conducting a more detailed analysis using machine learning. This multi-layered approach of combining rule-based techniques with machine learning models can result in eligibility modelbeing efficient and accurate in assessing whether an input should proceed to the next stages of processing.

133 133 133 Additionally, a confidence score for each eligibility determination can be generated by eligibility model. For example, eligibility modelcan generate a confidence score for both the language detection and subject classification components, and if the confidence score falls below a certain threshold, the input can be flagged for manual review or additional processing. Eligibility modelcan also employ probabilistic methods, such as Bayesian Networks or Gaussian Processes, to refine its confidence estimates and make more informed decisions about the eligibility of complex or noisy data.

130 134 134 112 134 112 In various embodiments, application server computerincludes a classification model. In various embodiments, classification modelcan be a machine learning model configured to analyze segmented sections of input dataand determine whether each section represents a question, an answer, or another type of text content. One of the functions of classification modelis to distinguish between different types of content and label each segment of input dataaccordingly, enabling the system to identify candidate questions and answers within the input data.

134 134 134 In various embodiments, classification modelis configured to identify candidate questions and answers from the practice-test-type input. For example, classification modelcan classify a first selection from the input as a candidate question and a second selection as a candidate answer. Classification modelperforms this function by analyzing the content and structure of the text segments and applying learned patterns to determine whether a segment is more likely to be a question or an answer.

134 134 134 In various embodiments classification modelcan be a large language model trained on a labeled training dataset, where training text is labeled with question-specific labels for segments corresponding to questions and answer-specific labels for segments corresponding to answers. This implies that classification modelhas been trained to recognize the linguistic and semantic features typical of questions and answers, such as interrogative structures (e.g., “What is . . . ?”, “How does . . . ?”) for questions and declarative or explanatory statements for answers. By using a labeled training dataset, classification modellearns to differentiate between questions and answers with high accuracy, even when the language and structure vary across different input types.

134 134 134 134 Classification modelcan be implemented using transformer-based architectures, such as BERT or GPT, which have been fine-tuned on the labeled training data to perform this specific classification task. Classification modelanalyzes each segment in context, considering both the local (word-level) and global (sentence- or paragraph-level) features to determine its classification. For example, if a segment begins with a keyword like “Explain” or “Describe,” classification modelmay classify it as a question, while a segment containing phrases like “The answer is” or “It can be defined as” might be classified as an answer. Additionally, classification modelcan employ attention mechanisms to focus on words and phrases that are indicative of questions or answers, further refining its predictions.

134 134 134 In some cases, classification modelcan be implemented as an Information Maximizing Generative Adversarial Network (Info-GAN) trained on a synthetic training dataset. Such Info-GAN can be adapted to help classify questions and answers by learning distinct and interpretable representations of these two content types. In this specific context, classification modelcan be set up with two main parts: a generator and a discriminator, along with an additional auxiliary network. The generator creates synthetic text segments based on random inputs combined with a special code that represents whether the text should look like a question or an answer. The discriminator's role is to differentiate between real and generated segments, while the auxiliary network tries to predict the code used to generate the text segment (i.e., whether it was meant to be a question or an answer). In this context, a generator of classification modelcan be trained to produce text segments that are realistic and accurately represent questions and answers. Further, a discriminator can be trained to accurately classify generated segments as questions or answers.

134 134 134 134 134 3 5 In some cases, classification modelcan be implemented as a foundation large language model (LLM) that has been pre-trained on a broad, large-scale language corpus. Additionally, classification modelcan undergo further training using a labeled training dataset, with the training data selected specifically based on the identified subject of the practice-test-type input. As a foundation model, classification modelbenefits from having a comprehensive understanding of the language before being fine-tuned for the specialized task of classifying questions and answers. This two-step training process allows classification modelto generalize effectively across various subjects and adapt to domain-specific content with improved accuracy. In some instances, classification modelcan be a variant of a GPT model, such as ChatGPT..

134 134 134 134 Further, classification modelcan be configured to handle ambiguous sections in the input data using uncertainty estimation techniques, such as Monte Carlo Dropout or Bayesian Neural Networks. These techniques quantify the confidence level of classification model's predictions, enabling classification modelto flag segments that are difficult to classify or have low confidence scores. For example, if a segment is neither clearly a question nor an answer, for example, due to vague language or insufficient context, classification modelcan assign a lower confidence score to its prediction and flag the segment for further review or reprocessing.

134 134 Additionally, or alternatively, classification modelcan use contextual shifts between sections to identify questions and answers. These shifts are detected by analyzing changes in linguistic features, such as interrogative structures, sentence modality, or semantic cues characteristic of questions, and declarative structures or explanatory context characteristic of answers. By tracking these shifts, classification modelcan more accurately determine where questions end and answers begin, especially in complex or unstructured inputs.

112 134 134 In certain scenarios, when the input dataincludes one or more sections that cannot be clearly identified as either a question or an answer, the classification modelmay classify such sections as neither. Alternatively, the classification modelmay categorize these sections as a concept section, introduction section, summary section, or similar types.

130 136 136 134 134 136 160 136 160 Further, in various embodiments, application server computerincludes an extraction model. Extraction modelis designed to extract questions and answers identified by classification model. When classification modeldetermines that a specific section of the input data is a question, extraction modelextracts that section and stores it separately as a question in an appropriate database, such as database. Similarly, extraction modelcan extract the corresponding answer and store it in database, as further detailed below.

136 134 134 136 Extraction modelcan be implemented as a code that extracts sentences or sections based on labels such as “question,” “answer,” “neither question nor answer,” “concept,” “introduction,” or similar, as assigned by classification model. For example, when classification modelassigns a “question” label to a section, extraction modelidentifies and processes it accordingly.

134 112 136 160 160 If classification modelidentifies a section of input dataas neither a question nor an answer, extraction modelcan store such sections in databasewith an indication that they are neither. In some cases, these sections may also be linked to one or more related questions and/or answers stored in database, as further described below.

130 138 138 Additionally, in various embodiments, application server computerincludes a generative model. The generative modelis configured to generate answers and, in certain cases, questions, as further detailed below.

138 134 112 112 134 112 138 134 138 In various embodiments, generative modelis configured to generate a suggested answer based on a candidate question extracted from a practice-test-type input. The candidate question may be identified by classification model, which analyzes input datato determine which parts of input datacorrespond to questions. Once classification modelclassifies a segment of input dataas a candidate question, generative modelcan be used to produce a corresponding suggested answer. In some cases, if classification modelidentifies a candidate answer, this answer can be represented as a first word embedding, serving as a vectorized representation of the extracted content. Similarly, the suggested answer generated by generative modelcan be represented using a second word embedding for comparison.

138 Generative modelcan leverage its broad language understanding, developed through pre-training and fine-tuning, to generate coherent and contextually relevant answers to the identified questions. Depending on the scenario, the suggested answer may either be a newly generated response created by the model or an existing answer retrieved from a suitable database, as further detailed below.

138 In various embodiments, generative modelcan be implemented using a large language model (LLM) architecture such as a Transformer-based model (e.g., GPT, BERT, or Text-To-Text Transfer Transformer), which has been pre-trained on a massive corpus of text data and further fine-tuned for the specific task of generating answers and questions for practice test-type inputs. The implementation involves two main phases: pre-training and fine-tuning, followed by a process to evaluate its performance and suitability for practice test generation.

138 In one implementation, the initial phase of training generative modelinvolves training it as a foundational LLM on a diverse and extensive corpus of text data. The pre-training process typically leverages the transformer architecture, which consists of multiple layers of self-attention mechanisms and feed-forward networks. The self-attention layers allow the model to capture relationships between words, phrases, and sentences within the context of large-scale text data. During this pre-training phase, the model learns general language structures, semantics, and contextual dependencies without any specific focus on practice tests.

The training dataset used in this phase can include various sources such as books, articles, websites, and research papers to ensure the model gains a broad understanding of grammar, syntax, and general world knowledge. The pre-training objective is usually a language modeling task, such as predicting the next word in a sentence (as in GPT models) or filling in missing words (as in BERT). This objective helps the model learn how language is typically structured and how to generate fluent and coherent text.

138 Once generative modelis pre-trained, it undergoes a fine-tuning phase using a specialized training dataset that is more relevant to the specific application of generating practice test questions and answers. This fine-tuning dataset can consist of labeled examples of practice test questions, answers, and related instructional content. The dataset is annotated to include domain-specific information based on subjects such as Mathematics, Science, or History, which aligns with the context of the input data provided by users.

138 During fine-tuning, generative modelis exposed to input data labeled with specific attributes, such as question labels or answer labels, and is trained to generate text segments that match these labels. The training objective during fine-tuning could involve minimizing the cross-entropy loss between the generated sequences and the ground truth sequences (e.g., correct answers to questions), ensuring that the model learns to produce accurate and contextually relevant content. This fine-tuning step allows the model to focus on the generation of practice test-type content and increases its ability to handle specific domain knowledge.

138 138 After training, generative modelcan be evaluated on a separate validation set to assess its performance. Metrics such as BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and perplexity can be used to measure the relevance of the generated content. If generative modelis integrated with similarity measurements, additional evaluation criteria may include similarity scores between generated and expected answers to ensure that the content is contextually accurate.

138 134 138 134 138 134 112 134 138 134 138 In some cases, generative modelmay be a more advanced large language model (LLM) compared to classification model. For instance, generative modelmay be implemented using GPT-4.0, whereas classification modelmay be based on GPT-3.5. Although generative modelmay be configured to perform the same tasks as classification model, including identifying questions and answers in input data, classification modelmay be generally preferred for this role because it is more efficient and requires less computational power and training resources compared to generative model. This makes classification modela more suitable choice for initial processing and classification tasks, while generative modelcan be reserved for generating content when needed.

138 134 134 112 112 138 112 138 Besides generating answers, the generative modelmay also generate questions based on concepts identified by the classification model. For instance, if classification modeldetermines that input dataincludes sections unrelated to a practice test (or if the entirety of input datadoes not correspond to a practice test), the generative modelcan generate questions derived from the identified concepts. For example, if the input datacontains the concept “Newton's Laws of Motion,” the generative modelmight generate questions like: What are Newton's three laws of motion? How does Newton's second law apply to a situation where a car accelerates on a flat road?

138 112 138 Further, the generative modelcan generate suitable answers based on the concepts discussed in input data. For instance, in response to the question “What are Newton's three laws of motion?” the generative modelmight generate the following answer: Newton's first law states that an object at rest will remain at rest, and an object in motion will remain in motion unless acted upon by an external force. Newton's second law explains that force equals mass times acceleration (F=ma). Newton's third law states that for every action, there is an equal and opposite reaction.

112 160 138 138 In various cases, the generated questions and answers, along with the relevant sections of input datathat resulted in their creation, can be stored relationally in a database. Furthermore, based on specific input to the generative model, questions and answers can be generated in particular styles. For example, if the input specifies generating multiple-choice questions, the generative modelcan produce such questions.

Which of the following best describes Newton's first law of motion? A. The acceleration of an object is directly proportional to the net force acting on it. B. For every action, there is an equal and opposite reaction. C. An object at rest stays at rest, and an object in motion stays in motion unless acted upon by an external force. D. The total energy in a closed system remains constant. For instance, for the concept “Newton's Laws of Motion,” a multiple-choice question might be:

138 Other styles of questions may include fill-in-the-blank or free-response questions. For example, for the same concept, the generative modelmight produce:

Fill-in-the-blank question: “Newton's second law of motion is represented by the equation______=mass×acceleration.” Answer: Force.

Free-response question: “Explain how Newton's third law applies when a person jumps off a small boat onto a dock.” Answer: When the person pushes off the boat to jump, they apply a force to the boat. In turn, the boat applies an equal and opposite force, causing it to move backward.

112 160 Consequently, the various questions and answers generated for a concept identified in the input datacan be stored relationally in database, enabling efficient retrieval and use in various contexts.

130 140 140 112 140 134 160 134 138 140 Additionally, in various embodiments, application server computerincludes an embedding model. In various embodiments, embedding modelis configured to transform the textual data from input datainto high-dimensional vector representations that encapsulate the semantic and contextual meaning of the text (also referred to herein as “input text”). These vectorized representations, referred to as word embeddings, are used for various tasks, such as similarity measurement, classification, and generation of practice test content. Embedding modelcan be used to represent the extracted candidate answer (i.e., the answer as identified by classification model, or the answer that can be retrieved from a databasecorresponding to the question identified by classification model) as a first word embedding and the suggested answer generated by generative modelas a second word embedding. These embeddings are then compared to measure similarity, allowing the system to determine the suitability of the candidate answer for inclusion in the practice test. In some cases, embedding modelis configured to generate multi-level embeddings, including word-level, sentence-level, and paragraph-level embeddings. This approach allows the model to capture different granularities of information: word-level embeddings reflect individual word meanings, sentence-level embeddings capture contextual relationships within a sentence, and paragraph-level embeddings consider the broader context across multiple sentences or paragraphs. These multi-level embeddings enable a comprehensive similarity comparison by aligning the embeddings at different levels of granularity, ensuring that the system can identify semantic relationships between the candidate answer and the suggested answer even when the content structure varies.

140 In various embodiments embedding modelis capable of implementing advanced neural network architectures, such as Transformer-based models like BERT, GPT, or T5, to generate contextual embeddings that capture word meanings based on their usage within the surrounding text. Alternatively, Word2Vec, GloVe, or FastText can be used.

140 In various embodiments, when implemented as a transformer-based model, embedding modelcan be pre-trained on large-scale language corpora using training objectives such as masked language modeling or next-sentence prediction to learn general language patterns. Subsequently, the model can be fine-tuned on domain-specific data relevant to practice-test-type inputs to refine its ability to capture subject-specific terminology and structures.

130 142 142 Further, in various embodiments, application server computerincludes an attention model. In various embodiments, attention modelis configured to enhance the accuracy of similarity calculations and content extraction by assigning different importance weights to individual words or phrases within the input data. The similarity calculations can be performed for determining similarity between a first text and a second text. The first text and the second text can be any suitable text representing questions, answers, concepts, and the like.

142 142 In some embodiments, attention modelcan be configured to weigh the significance of words or phrases within the candidate answer and suggested answer or a candidate question and suggested question. This attention mechanism assigns higher weights to words or phrases that are deemed contextually significant, which allows the system to capture the most relevant parts of the text when comparing candidate and suggested answers or candidate and suggested questions. For instance, if a candidate answer includes key phrases such as “primary cause” or “main effect,” attention modelwould assign these phrases higher weights during the embedding process, ensuring that they contribute more to the final similarity score between the candidate and suggested answers. This weighted comparison can be particularly useful when measuring semantic similarity between the two word embeddings, as it helps the system focus on the most meaningful components of the content rather than treating all words equally.

130 144 144 144 112 144 In various embodiments, application server computeralso includes a ranking model. In various embodiments, ranking modelis configured to prioritize and order questions and answers generated or retrieved by the system based on their relevance, quality, or other predefined criteria (e.g., difficulty). In various embodiments, ranking modelis configured to assign a ranking score to each question and/or answer based on how well it matches concepts described in input data. Ranking modelis particularly useful when multiple candidate questions or answers are generated or retrieved, allowing the system to select and present the most relevant content for practice test generation.

144 138 134 138 144 144 In some cases, ranking modelcan be employed to refine and improve the output of other models, such as generative modelor classification model. For example, when multiple suggested answers are generated by generative modelfor a single candidate question, ranking modelcan be used to evaluate these answers and prioritize them based on factors such as semantic relevance, coherence, and clarity. Similarly, if multiple question-answer pairs are retrieved from a question-answer database, ranking modelcan rank these pairs to identify the ones that are most closely aligned with the input question or context.

144 144 144 144 Ranking modelcan be implemented using various machine learning techniques, including pointwise, pairwise, or listwise ranking approaches. In a pointwise ranking approach, ranking modelevaluates each item independently and assigns a score based on its features. In a pairwise ranking approach, ranking modelconsiders pairs of items and learns to predict which one is better, improving the ranking based on relative comparisons. In a listwise ranking approach, ranking modeldirectly improves the entire ranking of a list of items to ensure that the top-ranked items are as relevant as possible.

144 144 To support these ranking strategies, ranking modelcan utilize advanced neural architectures such as transformer-based models, including cross-encoder models or bi-encoder models. A cross-encoder ranking model compares the input text and candidate content jointly, using attention mechanisms to capture complex interactions between them. This approach allows ranking modelto produce highly accurate ranking scores by considering the full context of both the input and the candidate content simultaneously. A bi-encoder model, on the other hand, generates separate embeddings for the input and candidate content and then measures their similarity using metrics such as cosine similarity or dot product. This approach is computationally efficient and well-suited for large-scale ranking tasks.

144 In some embodiments, ranking modelcan also incorporate LambdaMART, a tree-based ranking algorithm that is effective for improving ranking tasks with specific constraints, such as maximizing precision or minimizing errors in the top-ranked items. LambdaMART uses gradient boosting to learn a series of decision trees, with each tree focusing on correcting the ranking errors of the previous trees. This iterative refinement allows LambdaMART to produce a robust ranking model that can handle complex relationships between features.

144 To evaluate the performance of ranking model, metrics such as Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), or Precision at K (P@K) can be used. These metrics measure how well the top-ranked items align with the expected output, ensuring that the system consistently produces high-quality content for practice test generation.

130 160 160 160 130 160 130 In various embodiments, application server computeris communicatively coupled to database. In some cases, databasemay serve as a question-answer database that stores a collection of questions and their corresponding answers. Databasemay have an associated server that is configured to receive search requests from application server computer, perform searches within database, and return the relevant data back to application server computerbased on the search results.

160 160 In various embodiments, question-answer databaseis configured as a specialized storage system designed to store and manage a large collection of question-answer pairs that can be accessed and utilized during the practice test generation process. This database serves as a repository for existing educational content, which can be queried and retrieved based on semantic similarity to the input data. To facilitate efficient searching and retrieval of question-answer pairs, question-answer databaseemploys indexing and embedding techniques, enabling quick access to the most relevant content based on the context of the input query.

160 140 Question-answer databaseis structured to include multiple layers of indexing, such as a question embedding index and a question metadata lookup index. The question embedding index stores vectorized representations (embeddings) of the questions in the database, allowing for rapid similarity-based searches. Each question is converted into a high-dimensional vector using embedding model, capturing the semantic and contextual meaning of the question. These embeddings are stored in the question embedding index, making it possible to perform vector-based searches to find questions that are semantically similar to a given input.

160 160 160 To support efficient and scalable similarity searches, the question embedding index can use vector search algorithms, such as FAISS (Facebook AI Similarity Search), which uses quantization and partitioning techniques to reduce the dimensionality of the embeddings and enable fast approximate nearest neighbor searches. The FAISS index allows the server of databaseto identify the top-k closest questions in databasebased on their vector representations, even when the number of stored questions is very large. This capability is useful for finding relevant question-answer pairs that match the input text, ensuring that the server of databasecan retrieve the most suitable content for practice test generation.

160 In addition to the question embedding index, the question metadata lookup index is used to store and retrieve metadata associated with each question-answer pair. Metadata may include attributes such as subject, difficulty level, topic, and other tags that describe the content of the question and its corresponding answer. This metadata index allows performing filtered searches based on specific criteria, such as retrieving questions only from a particular subject (e.g., mathematics) or filtering results based on difficulty level (e.g., beginner, intermediate, advanced). By using this metadata index, the server of databasecan refine its searches and provide more targeted question-answer pairs that align with the user's requirements or the context of the practice test input.

160 160 The combination of the question embedding index and the question metadata lookup index enables a multi-faceted search approach. When a query is made, the server of databasemay perform a vector search for a question (or an answer) within database. When performing a vector search, vector similarity search using the question embedding index can be used to identify the most semantically relevant question. In some cases, the question metadata lookup index can also be used to narrow down the search space based on high-level attributes such as subject or topic.

144 Once the relevant question-answer pairs are retrieved, they can be ranked and further processed using other components of the system, such as ranking model, to ensure that the final practice test content is of the highest quality.

2 FIG. 1 FIG. 200 212 212 112 200 130 shows an example flowchart describing a methodfor generating or extracting practice test questions based on input data, which can be a document or other content related to a practice test. In various embodiments, input datamay be similar or the same as input data, as shown in. In various embodiments, operations of methodare performed by application server computer.

220 130 133 212 212 220 200 220 200 230 212 1 FIG. At operation, the application server computerconducts an eligibility check, utilizing the eligibility model, as previously described in connection with. This eligibility check involves verifying whether the input datais in English or another suitable language determined to be eligible for test preparation. Additionally, it evaluates whether the subject matter within the input datacorresponds to eligible categories, such as Arts and Humanities, Science, Social Science, Math, Language, or other suitable subject matters. In certain scenarios, categories like Math and Language Learning may be excluded from eligibility, while in other cases, different subject matters might be deemed ineligible. If the input data is deemed not to be eligible (operation, No), methodmay be completed. Alternatively, If the data is determined to be eligible (operation, Yes), methodproceeds to operationto determine if input datais a practice test.

230 212 134 212 212 230 200 232 212 212 136 At operation, whether input datais a practice test or just a list of concepts that can be used for formulating a practice test can be made based on a content analysis using classification model, which can identify questions within input data. If questions are identified in the input data(operation, Yes), methodproceeds to operationat which questions are extracted from input data(or identified within input data). Such extraction of questions can be achieved by extraction model, as described above.

212 230 230 200 234 212 230 134 212 234 212 If no identifiable questions are found in the input data, as indicated by the decision operation(operation, No), methodproceeds to operation, where concepts and definitions are extracted from input data. In various cases, at operation, classification modelcan identify questions, answers, concepts, definitions, formulas, and the like, and label each portion of input datawith a corresponding label, such as “question,” “concept,” “definition,” “answer,” and the like. Subsequently, at operation, all segments of input datacorresponding, for example, to definitions and concepts, can be extracted.

232 138 236 232 212 138 160 138 138 138 138 Following the completion of operation, generative modelcan be used at operationto generate answers to the questions extracted during operation. For a given question extracted from input data, the generative modelmay reference the question-answer database. If a matching question-answer pair is identified in the database, the model selects the most relevant pair to provide an answer closely aligned with the extracted question. Alternatively, if no suitable match is found, the generative modelgenerates an answer directly based on the extracted question and the knowledge it possesses. The knowledge within generative modelis obtained through pre-training on extensive datasets that includes diverse topics, including scientific literature, encyclopedic data, and domain-specific information. Additionally, generative modelmay be fine-tuned with specific data relevant to its application, such as curated educational materials, industry documents, or proprietary databases. This allows generative modelto synthesize responses based on patterns and context derived from its training and fine-tuning processes.

232 236 200 238 160 230 212 230 234 238 After questions are extracted during operationand answers are generated at operation, methodadvances to operation, where the questions and answers are stored in database. Alternatively, these questions and answers may be saved in a dedicated question bank, a memory storage space specifically designed for organizing content for a particular practice test. Additionally, if it was determined during operationthat the input datadoes not contain a pre-existing practice test (operation, No), but instead includes concepts and definitions extracted at operation, these concepts and definitions can also be stored in the question bank at operation. This enables their later use in the creation of a new practice test.

238 200 240 242 240 160 242 After completing operation, methodproceeds to operationsandat which new questions and answers are generated at operationand new questions and answers are retrieved from question-answer databasevia vector search at operation.

212 212 138 138 212 138 212 130 2 FIG. In some cases, new questions can be generated based on the concepts identified within input data. For instance, if the input dataconsists of a textbook chapter or lecture notes without any pre-existing questions, generative modelcan analyze the text to identify key concepts and topics discussed within the content. Using these identified concepts, generative modelconstructs relevant questions. For example, if input datadescribes the process of photosynthesis, generative modelmight generate questions such as “What is the role of chlorophyll in photosynthesis?” or “How does sunlight contribute to the photosynthesis process?” These generated questions are then passed on to the next operations, such as re-ranking and validation, as further described in the flowchart of. By generating new questions based on the core ideas present in the input text of input data, application server computerensures that even content without explicit questions can be transformed into a structured practice test, providing users with meaningful test material.

242 234 160 160 At operation, the concepts and definitions obtained during operationcan be converted into vector embeddings. These embeddings are then used to perform a vector search within databaseto identify the most closely related questions based on the generated embeddings. Once identified, the corresponding questions can be retrieved and incorporated into the practice test. In cases where question-answer pairs are available in database, both the questions and their corresponding answers can be retrieved for use in the practice test.

234 240 160 242 244 Once the questions and answers are extracted (as in operation), generated (as in operation), and/or retrieved from databasethrough vector search (as in operation), relevant new questions are selected for inclusion in the practice test at operation. In some cases, the selected questions may represent a subset of the generated, extracted, or retrieved questions, chosen based on specific relevance criteria. Alternatively, all the questions obtained in the preceding operations may be included in the practice test, depending on the desired scope and purpose of the test.

248 144 212 Further, at operation, ranking modelcan be employed to prioritize and order the questions based on their relevance, quality, and alignment with the input data. The ranking model uses metrics such as semantic similarity, coherence, and relevance scores to select the top questions. The ranking process can use a cross-encoder model such as Cohere Reranking API, where a particular text (e.g., a particular concept or original extracted question from input data) is treated as a query, and generated questions are treated as documents for reranking. To enhance accuracy and ensure relevance, the query includes the name of the generated question bank from a prior step, ensuring that the top-ranked questions align with the same subject matter. This process selects a specific number of questions (e.g., two, three, four, five, six, or a similar range) for each original question or concept.

248 200 250 After ranking operation, methodproceeds to operation, where the re-ranked questions undergo a comprehensive validation process. This validation operation can involve the use of both advanced machine learning models and predefined rule-based systems to review each question for factual accuracy, grammatical correctness, and alignment with the original input content.

The use of machine learning models in the validation process allows for automated and intelligent evaluation of various aspects of the questions. For example, a fact-checking model can be employed to verify the correctness of the information presented in the questions and answers by cross-referencing with a database of trusted sources. For instance, if a generated question states, “What is the capital of Canada?” and the suggested answer is “Toronto,” the fact-checking model would flag this as incorrect and suggest “Ottawa” as the correct answer. Similarly, a grammar-checking model, such as T5 or BERT, can be used to detect and correct grammatical errors. If a question is phrased awkwardly, such as “What is photosynthesis's role?” the grammar-checking model would identify this as an error and suggest the corrected form, “What is the role of photosynthesis?” Furthermore, a semantic consistency model may be utilized to ensure that the question and answer are logically coherent and contextually related. If a question asks, “What is the function of mitochondria?” and the answer states, “It is responsible for the process of photosynthesis,” the semantic consistency model would detect the mismatch and prompt for a correction, such as “Mitochondria generate energy for the cell.”

In addition to using machine learning models, rule-based systems are applied to enforce specific logical and structural standards for the questions and answers. These rules provide a structured framework that the questions can follow to be considered valid. For example, format rules can ensure that each question begins with an interrogative word (e.g., “What,” “How,” “Why”) and ends with a question mark. If a question does not follow this rule, such as “Describe the process of photosynthesis,” the rule-based system would suggest rephrasing it to “What is the process of photosynthesis?” to align with the expected format. Similarly, answer length rules specify that responses should not exceed a certain word count or that short-answer questions can have concise answers. If a generated answer for a short-answer question is too detailed or verbose, the rule-based system can flag it as needing simplification.

The validation operation also leverages rules that prohibit the use of ambiguous or unclear phrases. For instance, if a question includes vague terms such as “kind of” or “sort of,” the rule-based system can flag these terms and suggest removing or replacing them with clearer language. This ensures that the questions are straightforward and unambiguous, enhancing their quality and clarity. By utilizing both machine learning models and rule-based systems, the validation process ensures that the generated or extracted questions are factually accurate, grammatically sound, and contextually aligned with the original input content. This multi-faceted approach can be used to reduce errors.

200 252 Methodconcludes with operation, where the final practice test is compiled by listing the validated questions and corresponding answers. The compiled practice test can be formatted and presented to the user in various ways, depending on the desired interface and user preferences. For example, one common method of presentation is through digital flashcards, where each question is displayed on one side of a digital card, and the answer is revealed when the user chooses to flip the card. This interactive format is particularly effective for self-paced study, enabling users to review each question and answer individually and promoting active recall during the learning process.

Alternatively, the compiled practice test can be displayed as a conventional list of questions and answers, where all questions and their corresponding answers are shown in a sequential order. This format is suitable for users who prefer to read through all the material at once or for printing purposes, allowing the user to have an offline version of the test for review. The list format can also include additional elements such as the categorization of questions by topic or difficulty level, enabling users to focus on specific areas where they may need further practice.

Furthermore, the compiled practice test can be presented using other interactive formats such as multiple-choice quizzes, where the generated questions are shown with several answer options, and users can select the correct one. This format simulates traditional exam conditions and can be used to assess the user's knowledge and understanding in a more structured manner. Similarly, short-answer or fill-in-the-blank formats can be used, where users are prompted to provide answers based on the generated questions, and the system automatically evaluates the responses for correctness.

The compiled test can also be displayed through a dynamic graphical user interface (GUI) that adapts based on user input. For instance, the system may present the first question and allow the user to interactively navigate through subsequent questions. This interface may also include options for reviewing specific topics, flagging questions for later review, or providing hints for challenging questions. In addition, the compiled test can be integrated with analytics tools that track user performance, such as recording the number of correct answers, measuring response time for each question, and providing feedback on areas that need improvement.

In some embodiments, the compiled practice test may also include supplementary content such as explanatory notes or additional reference materials for each question. For example, if a question asks about a specific scientific concept, the compiled test could include a short paragraph or a hyperlink to a resource that further explains the concept, thereby providing users with immediate access to more in-depth information.

244 Overall, the compiled practice test generated in operationis flexible and can be tailored to suit various user needs and preferences.

3 FIG. 1 FIG. 300 130 132 138 illustrates a flowchart of method, which describes the workflow for generating, retrieving, and processing practice test questions using a combination of machine learning models and database operations. This process is executed as a practice test service operated by application server computer, which is configured to initiate and manage the entire practice test generation workflow. The practice test service is responsible for receiving input data, such as study materials or user requests, and performing initial verification and preprocessing before handing it off to subsequent operations. Additionally, the practice test service provides the processed input data to the ML Practice Test Service, which performs further data processing using various models, including models-as illustrated in. Moreover, the practice test service integrates with external interfaces, such as web or mobile applications, to facilitate user interactions and ensure seamless communication throughout the process.

314 300 312 312 212 112 312 132 138 2 FIG. 1 FIG. At operation, methodreceives input data, which contains the content used for generating the practice test. In various embodiments, input datacan be similar to or the same as input dataor input data, as depicted in respectiveand. In some instances, input datamay include additional metadata provided via the practice test service to the ML Practice Test Service. This metadata may include parameters for configuring models-, descriptions of the input data, guidance information for the models during data processing, details about the user's device, and instructions on how the practice test data should be presented to the user.

300 Metadata associated with input data in methodincludes a variety of attributes and parameters that guide the practice test generation process. Examples of such metadata include subject information, difficulty level, question format, user preferences, contextual information, user interaction data, model configuration parameters, and the like. For instance, subject information can specify the subject or topic of the input data, such as “Biology,” “Physics,” or “History,” enabling the models to focus on generating questions relevant to the specified field. Difficulty level metadata can indicate the desired complexity of the questions, such as “Beginner,” “Intermediate,” or “Advanced,” allowing the system to tailor the questions to match the user's proficiency level. Metadata related to question format can define the type of questions to be generated, such as multiple-choice, short answer, true/false, or descriptive questions, thereby ensuring that the generated content aligns with the desired format.

User preferences may include specific topics or subtopics the user wants to concentrate on, such as “Photosynthesis” within Biology, to generate targeted questions. Contextual information can provide additional background, such as summaries or descriptions of the input data, which can help the models better understand the content and produce more relevant questions and answers. Presentation instructions within metadata can define how the final practice test should be displayed, such as through digital flashcards, list formats, or interactive quizzes, and may include specifications for visual styles, fonts, or layout preferences. Additionally, metadata can include user interaction data, such as frequently missed questions, areas where the user struggled, or specific sections flagged for review. This historical information can guide the system to generate practice tests that focus on areas needing improvement.

Model configuration parameters within the metadata may describe specific settings for configuring the models, such as the number of questions to generate, thresholds for similarity scores when matching questions, or parameters for adjusting the granularity of content segmentation. These configuration parameters provide further control and customization over the practice test generation process, ensuring that the generated content meets the user's needs and expectations.

314 300 314 300 320 212 314 300 316 312 133 312 316 300 312 316 300 318 1 FIG. At operation, methoddetermines whether there is any cached data available for the practice test. If the data is cached (operation, Yes), methodretrieves the cached questions from a first question bankand provides them for the practice test. Specifically, input datais hashed to generate a unique key, which is then used to cache the mapping between the hash value and the ID of the question bank created for that specific input. For new requests, if the hash of the input matches an existing cached entry, the corresponding question bank ID is used to retrieve the pre-existing question bank, which is then provided for the practice test. If no cached data is found (operation, No), methodmoves to operation, where the eligibility of the input datais evaluated for use in generating a practice test. Eligibility is determined using eligibility model, as described in relation to. If input datais found to be ineligible (operation, No), methodterminates. However, if input data, or any segment thereof, is determined to be eligible (operation, Yes), methodproceeds to operation.

318 134 312 134 312 134 312 At operation, classification modelis employed to classify input data. Classification modelfirst identifies the type of input data. For example, classification modeldetermines whether input datais a practice-test-type input or a non-practice-test-type input. A practice-test-type input typically includes questions and answers, while a non-practice-test-type input may consist of educational materials that can be used to generate practice tests, such as research papers, textbook pages, lecture notes, presentation slides, and similar content.

312 318 136 318 136 312 318 138 318 332 160 330 If input datais classified as a practice-test-type input, at operationthe identified questions are extracted. The question extraction process can be performed, for example, by extraction model. Additionally, in some cases, at operation, extraction modelcan also extract answers if they are present within input data. Furthermore, at operation, after extracting questions, answers to these questions can be generated using the generative model, as previously described. Upon completion of operation, question-answer pairs become available. In some instances, these pairs may be used in subsequent operationto generate similar new questions, or similar questions may be retrieved from the question-answer databaseduring operation, as detailed below.

318 300 330 332 330 312 312 160 312 360 363 361 363 362 312 318 330 312 160 Following operation, methodadvances to operationsand, which can be executed in parallel. At operation, if the input datais identified as non-practice-test-type input, concepts and definitions from the input dataare extracted. Vector embeddings are then generated based on these concepts and definitions, enabling a vector search within the question-answer databaseto retrieve questions and answers that match the concepts and definitions found in input data. This search leverages a question reservoircontaining a question index, a question ID lookup(which maps the question indexto the question identification (ID) used in the question metadata table), and a question metadata table. The metadata table is designed to store text corresponding to various questions and answers, along with associated attributes such as subject, grade level, and difficulty level. Furthermore, even if the input datais identified as practice-test-type input at operation, additional questions and answers can still be retrieved at operationbased on the questions and/or answers extracted from the input data. This process may involve generating vectorized representations of the extracted questions and answers and performing a vector search within the question-answer databaseto identify relevant additional questions and associated answers.

362 Question metadata tablerefers to information and attributes associated with each question in a database, providing context and enabling efficient organization, retrieval, and filtering. Metadata helps models and algorithms make more informed decisions about which questions to include, how to rank them, and how to present them in a way that best suits the user's requirements. Examples of question metadata include topic or subject, which specifies the subject matter of the question, such as “Biology,” “Physics,” “Chemistry,” or more granular topics like “Photosynthesis,” “Newton's Laws,” or “Chemical Reactions.” Metadata may also include difficulty level, indicating the complexity of the question as “Beginner,” “Intermediate,” “Advanced,” or a numerical difficulty score, enabling the system to tailor practice tests according to user proficiency. Question type is another metadata attribute that defines the format of the question, such as “Multiple Choice,” “Short Answer,” “True/False,” “Fill-in-the-Blank,” or “Essay,” allowing the system to filter questions based on the preferred format or assessment style. Metadata can also include keywords or tags, which are descriptive keywords such as “energy,” “forces,” or “ecosystem” that help in searching for questions related to specific concepts and assist in semantic analysis for retrieving similar questions.

Additionally, concepts covered within the question metadata list the concepts or learning objectives addressed, such as “Newton's First Law of Motion” or “Energy Transfer in Ecosystems,” enabling mapping to curriculum standards or specific learning goals. The metadata can also specify answer options and the correct answer, particularly for multiple-choice questions, where it includes the list of answer choices and the correct one, facilitating structural understanding and evaluation. Metadata may include the question's origin or source, describing the textbook, lecture, or source from which the question was derived, or indicating whether it was generated by a model or retrieved from a previous test. Historical performance data can be part of the metadata, showing how users previously performed on the question, such as average response time, percentage of correct responses, or frequency of selection. This data helps determine the difficulty level or identify questions that need revision. Associated explanations or hints can also be included, providing additional context or support for users who may struggle with the question.

318 312 300 332 138 312 312 332 312 If it is determined at operation, that input datais a non-practice-test-type input, methodcan proceed to operation, where generative modelis used to generate questions based on the various concepts identified within input data. Furthermore, even when input datais determined to be a practice-test-type input, in some cases, at operationadditional questions can be generated based on the questions and/or answers extracted from input data.

330 332 334 144 300 336 340 300 Once questions are either retrieved or generated at operationsor, respectively, they are processed at operation, where the questions and answers are ranked using ranking model. This ranking operation prioritizes and orders the questions based on various criteria, such as relevance, quality, and semantic alignment with the original content, as previously explained. Following the ranking, methodproceeds to operationfor additional post-processing tasks, such as formatting and validation, removing duplicate questions, eliminating inaccurate answers, and applying moderation filters, such as adjusting language or tone. After post-processing, the validated and ranked questions are stored in Second Q Bank at operationfor subsequent use in the practice test. Methodthen completes, ensuring that the final set of questions is accurately aligned with the input data and ready for presentation to the user.

4 FIG. 1 FIG. 400 400 130 133 138 130 410 400 130 130 133 133 illustrates an example of methodfor generating a practice test. In various embodiments, operations of methodare performed by application server computer, and more particularly by various models-of application server computer. Some operations can be optional, as indicated by boxes with dashed lines. At operation, methodbegins by determining the eligibility of data input, which is obtained from a client computer using a server computer, such as application server computerdepicted in. To assess eligibility, application server computerutilizes eligibility model, which identifies the language and subject associated with the data input. Eligibility modelthen checks whether the identified language and subject match predefined sets of acceptable languages and subjects, thereby establishing whether the input data qualifies as eligible test input.

400 412 130 134 400 414 If the data input is determined to be eligible, methodadvances to operation, where application server computerclassifies the input as a practice-test-type input using a data classification model, such as classification model. A practice-test-type input typically includes educational material formatted with questions and answers or is otherwise suitable for practice test generation. After determining the input type, methodproceeds to operation.

414 130 134 136 416 134 418 At operation, application server computer, using classification model, classifies a first selection from the practice-test-type input as a candidate question. After classifying a particular section of the practice-test-type input as a question, such a question can then be extracted, for example, by extraction model. Subsequently, at an optional operation, classification modelcan be used to identify a second selection from the practice-test-type input as a candidate answer. The candidate question is then can be represented using a first word embedding at operation, which encodes the semantic and contextual meaning of the question text into a high-dimensional vector for further processing.

416 136 416 416 416 Additionally, during operation, the identified answer can be extracted using, for example, extraction model. In some cases, operationmay not be performed. For example, the extraction part of operationmay not be performed if practice-test-type input does not contain answers. Alternatively, even if practice-test-type input does contain such answers, such answers may not be extracted. Furthermore, operationmay be skipped entirely.

420 130 138 422 Next, at operation, application server computeremploys a generative model, such as generative model, to generate a suggested question based on the candidate question and, in some cases, a candidate answer. This suggested question is represented using a second word embedding at operation.

424 130 At an optional operation, application server computermeasures the similarity between the first word embedding (representing the candidate question) and the second word embedding (representing the suggested question).

424 400 425 425 130 426 425 400 420 At operation, methodcalculates a similarity score based on the comparison of these embeddings. This similarity score is then evaluated at operationto determine whether the suggested question is suitable as a practice test question. If the similarity score is above a predefined threshold (operation, Yes), application server computerselects the suggested question as the practice test question at operation. If the similarity score does not meet the threshold (operation, No), methodmay proceed to operationat which a new suggested question is generated.

426 400 428 138 138 After completion of operation, methodproceeds to operation, at which a suggested answer is generated for the suggested question. In some cases, for generating suggested answer, generating modelcan be used. In some cases, input for generating modelmay include a suggested question, a candidate question, and a candidate answer to generate a suitable suggested answer.

428 400 430 After completion of operation, methodproceeds to operationat which the practice test is generated. The practice test can be represented as one or more digital cards, where each digital card is configured to include a first graphical user element displaying the practice test question and a second graphical user element displaying the practice test answer. This format provides a user-friendly and organized way to present the practice test to the user, making it accessible and easy to navigate.

138 138 In various embodiments, the displayed practice test questions may not be the original questions extracted from the practice-test-type input. Instead, they can be newly generated by the generative modelbased on the extracted questions, ensuring they are sufficiently distinct from the originals. Similarly, when answers are extracted from practice-test-type input, the displayed practice test answers can also be generated by the generative modeland associated with the newly generated questions. This approach is employed, for example, to create a practice test that is distinct from the one provided in the practice-test-type input.

5 FIG. 500 500 130 133 138 130 500 510 410 400 133 illustrates method, which is an example of generating practice tests based on non-practice-test-type input data. In various embodiments, the operations of methodare executed by application server computer, utilizing different models (e.g., models-) within the application server computer. Methodbegins at operation, where the server determines the eligibility of the data input. This operation may be similar to operationof method, where the eligibility modelidentifies the language and subject of the data input and verifies that it matches predefined sets of acceptable languages and subjects, qualifying the input as an eligible test input.

500 512 130 134 If the data input is determined to be eligible, methodmoves to operation, where application server computerdetermines that the eligible input is a non-practice-test-type input using classification model. A non-practice-test-type input refers to content that does not already include structured question-and-answer formats but can still be leveraged for practice test generation. Examples of such input include lecture notes, textbook pages, academic papers, or other educational materials.

514 130 134 136 516 At operation, application server computerutilizes classification modelto identify one or more concepts from the non-practice-test-type input. After identifying such concepts, these concepts can then be extracted using, for example, extraction model. This extraction identifies concepts or topics within the data that are relevant for generating questions. At operation, each extracted concept is represented using a concept word embedding, which transforms the semantic meaning of the concepts into high-dimensional vectors for efficient comparison and further processing.

518 130 160 520 1 FIG. Subsequently, at operation, application server computeraccesses a question-answer database (e.g., database, as depicted in) that contains numerous records of questions and their corresponding answers. Each question in the database is represented by a corresponding question word embedding, which enables the system to measure the similarity between the concept word embeddings and the question word embeddings. At operation, the system measures the similarity between the concept word embeddings and the question word embeddings in the database to identify any overlap. To enable efficient searching, the database may employ a FAISS index for the questions and answers, using quantization and partitioning to support vector search. This allows for fast and scalable retrieval of the most relevant questions based on the similarity scores.

500 522 523 500 524 523 500 526 138 138 528 138 Methodthen proceeds to operation, where the system evaluates whether there is a match between the extracted concepts and the questions in the database based on the similarity scores. If the similarity score for any question is above a selected threshold (operation, Yes), methodadvances to operation, where the matched questions and their corresponding answers are selected for use in the practice test. If no matching questions are found (operation, No), methodtransitions to operation, where a generative model (e.g., generative model) is used to generate new questions based on the extracted concepts. In some cases, generative modelmay also generate corresponding answers for these newly created questions at operation. Alternatively, the system can search the question-answer database again after generating questions using generative modelto identify matching records in the database.

530 500 At operation, methodcompiles and presents the generated practice test, organizing the selected or generated questions and answers into a coherent test format. The method then completes, having successfully generated a practice test that is contextually relevant to the original non-practice-test-type input data.

The computer-implemented methods described herein provide a framework for generating and validating questions and answers from practice-test-type input, as well as input containing concepts and definitions. A classification model can be used to identify candidate questions and answers through multiple inference stages. For instance, a second inference stage of the classification model can process a second selection from the input to identify candidate answers. These candidate answers are then evaluated using an embedding model that generates word embeddings for both the candidate and suggested answers. The similarity between these embeddings can be calculated, and a suggested answer is deemed suitable if its similarity score exceeds a predefined threshold. This ensures that generated answers are contextually relevant and aligned with the practice test's content.

The methods discussed herein also support processing multiple candidate questions and answers. The first selection may include a plurality of candidate questions, while the second selection may include multiple candidate answers. These selections can be incorporated into prompts for downstream processing, enabling a more comprehensive evaluation and refinement of the generated content.

In some cases, to prepare the input data for these operations, the method can employ a segmentation model that divides the input into logical sections. This segmentation process uses rule-based techniques to detect structural patterns, such as paragraph breaks, bullet points, or changes in indentation. Syntactic and semantic feature extraction techniques further enhance the segmentation by identifying logical boundaries between sections. Once segmented, an eligibility model determines the language and subject of each section, ensuring that only relevant portions of the input are analyzed further.

The classification model can be designed to handle diverse and ambiguous inputs. It may be implemented as a large language model trained on labeled datasets, an Information Maximizing Generative Adversarial Network (Info-GAN) trained on synthetic datasets, or a foundation model pre-trained on large-scale language corpora and fine-tuned for specific subjects. To manage ambiguous sections, the system employs uncertainty estimation techniques such as Monte Carlo Dropout or Bayesian Neural Networks, which flag sections with low confidence scores for additional review. Furthermore, the model is configured to detect contextual shifts between sections by analyzing linguistic features, improving the accuracy of question and answer identification.

The methods discussed herein use an embedding model to evaluate the relationships between candidate and suggested content, such as candidate and suggested questions or candidate and suggested answers. The embedding model generates multi-level embeddings that capture semantic relationships at the word, sentence, and paragraph levels. These embeddings are compared using similarity measures, such as cosine similarity, to assess their similarity. An attention model can enhance this process by assigning higher weights to contextually significant words or phrases, ensuring that semantic elements are prioritized in the similarity calculations.

Furthermore, to generate suggested questions and answers, the methods discussed herein can, in some embodiments, access a question-answer database containing precomputed word embeddings for both. For suggested questions, the embeddings of candidate questions are compared to those in the database, with matches determined based on similarity scores exceeding a selected threshold. Similarly, candidate answers are represented as embeddings and compared to answer embeddings in the database to retrieve corresponding questions. This process ensures that the generated questions and answers are sufficiently novel but contextually related to the input data.

As described above, both segmentation and eligibility processes can be used to prepare a practice test. The segmentation model can be configured to identify logical sections using a variety of patterns, such as paragraph breaks, bullet points, or sentence structure variations, while the eligibility model evaluates the sections'language and subject. Together, these steps prepare the input for classification and embedding operations.

By integrating segmentation, classification, embedding, and attention-based similarity techniques, the methods provide a solution for generating practice test content. The methods described herein ensure that the generated questions and answers are not only relevant and novel but can also be generated for diverse input scenarios, including those containing ambiguous input text.

According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and/or other computing devices that are coupled using a network, such as a packet data network. The computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is persistently programmed to perform the techniques, or may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the described techniques. The computing devices may be server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body mounted or implantable devices, smartphones, smart appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or aerial vehicles, any other electronic device that incorporates hard-wired and/or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and/or a network of server computers and/or personal computers.

6 FIG. 6 FIG. 600 is a block diagram that illustrates an example computer system with which an embodiment may be implemented. In the example of, a computer systemand instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

600 602 600 602 Computer systemincludes an input/output (I/O) subsystemwhich may include a bus and/or other communication mechanism(s) for communicating information and/or instructions between the components of the computer systemover electronic signal paths. The I/O subsystemmay include an I/O controller, a memory controller and at least one I/O port. The electronic signal paths are represented schematically in the drawings, for example as lines, unidirectional arrows, or bidirectional arrows.

604 602 604 604 At least one hardware processoris coupled to I/O subsystemfor processing information and instructions. Hardware processormay include, for example, a general-purpose microprocessor or microcontroller and/or a special-purpose microprocessor such as an embedded system or a graphics processing unit (GPU) or a digital signal processor or ARM processor. Processormay comprise an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

600 606 602 604 606 606 604 604 600 Computer systemincludes one or more units of memory, such as a main memory, which is coupled to I/O subsystemfor electronically digitally storing data and instructions to be executed by processor. Memorymay include volatile memory such as various forms of random-access memory (RAM) or other dynamic storage device. Memoryalso may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor. Such instructions, when stored in non-transitory computer-readable storage media accessible to processor, can render computer systeminto a special-purpose machine that is customized to perform the operations specified in the instructions.

600 608 602 604 608 610 602 610 604 Computer systemfurther includes non-volatile memory such as read only memory (ROM)or other static storage device coupled to I/O subsystemfor storing information and instructions for processor. The ROMmay include various forms of programmable ROM (PROM), such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). A unit of persistent storagemay include various forms of non-volatile RAM (NVRAM), such as FLASH memory, or solid-state storage, magnetic disk or optical disk such as CD-ROM or DVD-ROM and may be coupled to I/O subsystemfor storing information and instructions. Storageis an example of a non-transitory computer-readable medium that may be used to store instructions and data which when executed by the processorcause performing computer-implemented methods to execute the techniques herein.

606 608 610 The instructions in memory, ROMor storagemay comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server or web client. The instructions may be organized as a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or no SQL, an object store, a graph database, a flat file system or other data storage.

600 602 612 612 600 612 612 Computer systemmay be coupled via I/O subsystemto at least one output device. In one embodiment, output deviceis a digital computer display. Examples of a display that may be used in various embodiments include a touch screen display or a light-emitting diode (LED) display or a liquid crystal display (LCD) or an e-paper display. Computer systemmay include other type(s) of output devices, alternatively or in addition to a display device. Examples of other output devicesinclude printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators or servos.

614 602 604 614 At least one input deviceis coupled to I/O subsystemfor communicating signals, data, command selections or gestures to processor. Examples of input devicesinclude touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and/or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and/or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

616 616 604 612 614 Another type of input device is a control device, which may perform cursor control or other automated control functions such as navigation in a graphical interface on a display screen, alternatively or in addition to input functions. Control devicemay be a touchpad, a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processorand for controlling cursor movement on an output devicesuch as a display. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gearshift mechanism or other type of control device. An input devicemay include a combination of multiple different input devices, such as a video camera and a depth sensor.

600 612 614 616 614 612 In another embodiment, computer systemmay comprise an internet of things (IoT) device in which one or more of the output device, input device, and control deviceare omitted. Or, in such an embodiment, the input devicemay comprise one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measurement devices or encoders and the output devicemay comprise a special-purpose display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

600 614 600 612 600 624 630 When computer systemis a mobile computing device, input devicemay comprise a global positioning system (GPS) receiver coupled to a GPS module that is capable of triangulating to a plurality of GPS satellites, determining and generating geo-location or position data such as latitude-longitude values for a geophysical location of the computer system. Output devicemay include hardware, software, firmware and interfaces for generating position reporting packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that specify a position of the computer system, alone or in combination with other application-specific data, directed toward host computeror server computer.

600 600 604 606 606 610 606 604 Computer systemmay implement the techniques described herein using customized hard-wired logic, at least one ASIC or FPGA, firmware and/or program instructions or logic which when loaded and used or executed in combination with the computer system causes or programs the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer systemin response to processorexecuting at least one sequence of at least one instruction contained in main memory. Such instructions may be read into main memoryfrom another storage medium, such as storage. Execution of the sequences of instructions contained in main memorycauses processorto perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

610 606 The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage. Volatile media includes dynamic memory, such as memory. Common forms of storage media include, for example, a hard disk, solid state drive, flash drive, magnetic data storage medium, any optical or physical data storage medium, memory chip, or the like.

602 Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a bus of I/O subsystem. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.

604 600 600 602 602 606 604 606 610 604 Various forms of media may be involved in carrying at least one sequence of at least one instruction to processorfor execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a communication link such as a fiber optic or coaxial cable or telephone line using a modem. A modem or router local to computer systemcan receive the data on the communication link and convert the data to a format that can be read by computer system. For instance, a receiver such as a radio frequency antenna or an infrared detector can receive the data carried in a wireless or optical signal and appropriate circuitry can provide the data to I/O subsystemsuch as placing the data on a bus. I/O subsystemcarries the data to memory, from which processorretrieves and executes the instructions. The instructions received by memorymay optionally be stored on storageeither before or after execution by processor.

600 618 602 618 620 622 618 622 618 618 Computer systemalso includes a communication interfacecoupled to I/O subsystem. Communication interfaceprovides a two-way data communication coupling to network link(s)that are directly or indirectly connected to at least one communication networks, such as a networkor a public or private cloud on the Internet. For example, communication interfacemay be an Ethernet networking interface, integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of communications line, for example an Ethernet cable or a metal cable of any kind or a fiber-optic line or a telephone line. Networkbroadly represents a local area network (LAN), wide-area network (WAN), campus network, internetwork or any combination thereof. Communication interfacemay comprise a LAN card to provide a data communication connection to a compatible LAN, or a cellular radiotelephone interface that is wired to send or receive cellular data according to cellular radiotelephone wireless networking standards, or a satellite radio interface that is wired to send or receive digital data according to satellite wireless networking standards. In any such implementation, communication interfacesends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

620 620 622 624 Network linktypically provides electrical, electromagnetic, or optical data communication directly or through at least one network to other data devices, using, for example, satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network linkmay provide a connection through networkto a host computer.

620 622 626 626 628 630 628 630 630 600 630 630 630 Furthermore, network linkmay provide a connection through networkor to other computing devices via internetworking devices and/or computers that are operated by an Internet Service Provider (ISP). ISPprovides data communication services through a worldwide packet data communication network represented as internet. A server computermay be coupled to internet. Server computerbroadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer executing a containerized program system such as DOCKER or KUBERNETES. Server computermay represent an electronic digital service that is implemented using more than one computer or instance and that is accessed and used by transmitting web services requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app services calls, or other service calls. Computer systemand server computermay form elements of a distributed computing system that includes other computers, a processing cluster, server farm or other organization of computers that cooperate to perform tasks or execute applications or services. Server computermay comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. Server computermay comprise a web application server that hosts a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system or other data storage.

600 620 618 630 628 626 622 618 604 610 Computer systemcan send messages and receive data and instructions, including program code, through the network(s), network linkand communication interface. In the Internet example, a server computermight transmit a requested code for an application program through Internet, ISP, local networkand communication interface. The received code may be executed by processoras it is received, and/or stored in storage, or other non-volatile storage for later execution.

604 604 600 The execution of instructions as described in this section may implement a process in the form of an instance of a computer program that is being executed, and consisting of program code and its current activity. Depending on the operating system (OS), a process may be made up of multiple threads of execution that execute instructions concurrently. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Several processes may be associated with the same program; for example, opening several instances of the same program often means more than one process is being executed. Multitasking may be implemented to allow multiple processes to share processor. While each processoror core of the processor executes a single task at a time, computer systemmay be programmed to implement multitasking to allow each processor to switch between tasks that are being executed without having to wait for each task to finish. In an embodiment, switches may be performed when tasks perform input/output operations, when a task indicates that it can be switched, or on hardware interrupts. Time-sharing may be implemented to allow fast response for interactive user applications by rapidly performing context switches to provide the appearance of concurrent execution of multiple processes simultaneously. In an embodiment, for security and reliability, an operating system may prevent direct communication between independent processes, providing strictly mediated and controlled inter-process communication functionality.

In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.