Method and System for Streaming Transmission

This application claims the priority and benefit of Taiwan application serial no. 114111183, filed on Mar. 25, 2025, the disclosure of which is hereby incorporated in its entirety by reference herein.

This disclosure relates to a stream transmission method and system for data transmission using a language model to complete tasks.

With the flourishing development of artificial intelligence (AI) and deep learning technology, Large Language Models (LLM) have become core tools in the field of natural language processing, with their application scope covering voice assistants, language translation, automatic summary generation, and many other scenarios. In some applications, AI agent plays an intermediary role in calling large language models, responsible for receiving requests from the client, and utilizing the computing power of large language models to complete complex tasks, such as text analysis, answering questions, or generating content. Subsequently, the client will get the execution results of these tasks through the way of query.

The client typically adopts polling to query whether the agent has completed the task and obtained the result. The working principle of this polling mechanism is that the client continuously sends query requests to the agent at certain time intervals to detect the status of task execution. On one hand, the agent needs to process frequent queries from the client, leading to excessive consumption of system resources; on the other hand, when the task execution time is longer, too frequent polling may increase unnecessary load, while too sparse polling may increase the latency of query results.

The present disclosure provides a method and system that effectively reduces the resource consumption of the agent while ensuring that the client can get task execution results in real time. The present disclosure proposes a stream transmission method and system.

This disclosure proposes a stream transmission method performed by a computer system. This method includes: receiving a request from a client, and establishing a task according to this request; when executing the task, calling a language model to continuously generate multiple data fragments, and storing these data fragments in a stream pool; receiving a query corresponding to the task from the client, and determining whether the task is completed; in responding to the task being completed, establishing a one-time connection with the client to send all of the stored data fragments to the client; and in responding to the task not yet being completed, establishing a stream connection with the client to send the existing stored data fragments to the client, continuously receiving a subsequent data fragment from the language model, and sending the subsequent data fragment to the client through the stream connection.

From another perspective, the embodiment of this disclosure proposes a stream transmission system, including a client and a server. The server is communicatively connected to the client. The server receives a request from the client, and establishes a task according to the request. When executing the task, the server calls a language model to continuously generate multiple data fragments, and stores the data fragments in the stream pool. The server receives a query corresponding to the task from the client, and determines whether the task is completed. In response to the task being completed, the server establishes a one-time connection with the client to send all stored data fragments to the client. In response to the task not yet completed, the server establishes a stream connection with the client to send the stored data fragments to the client, continuously receive a subsequent data fragment from the language model, and sends the subsequent data fragments to the client through the stream connection.

Some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, when the same element symbols appear in different drawings, they will be regarded as the same or similar elements. These embodiments are only a part of the disclosure and do not disclose all possible implementations of the disclosure. More precisely, these embodiments are only examples of the systems and methods in the scope of patent claims of the present disclosure.

Regarding the terms “first”, “second”, etc. used in this document, they do not specifically refer to order or sequence, but are merely used to distinguish elements or operations described with the same technical terminology.

1 FIG. 1 FIG. 100 110 120 110 120 110 110 110 120 110 120 is a schematic diagram illustrating a stream transmission system according to an embodiment. Referring to, a stream transmission systemincludes a clientand a server, wherein the clientis communicatively connected to the server. The clientmay be a user device or another server. For example, the clientmay be a personal computer, smartphone, laptop, tablet computer, etc. An application, such as a chatbot, artificial intelligence agent, customer service machine, etc., is executed on the clientand this application will send a request to the server. In some embodiments, the clientis another server that sends requests to the serverafter interacting with other devices.

120 122 123 120 121 124 125 121 124 121 124 122 123 121 122 123 The serverruns multiple worker applications-, and the serveralso establishes a task queue, a stream pool, and a database. The task queueand the stream poolcontain stream data structures. For example, the task queueand the stream poolmay be implemented using Redis, but the present disclosure is not limited to this. In other embodiments, Kafka may be used for implementation. The worker applications-retrieve a task from the task queueand execute this task. For example, the worker applications-may be implemented as workers under the Celery framework.

120 110 120 120 121 122 123 121 120 122 123 The serverreceives a request from the clientand establishes a task based on this request. This request may include one or more texts, images, symbols, and other data. The purpose of this request may be to ask questions, generate images, generate code, generate summaries, process images, etc., but the present disclosure is not limited to these. The serverwill establish an identifier, status, etc. for the task, and this status at least indicates whether the task has been completed. The task established by the serveris added to the task queue. Each worker application-retrieves a task from the task queueand executes the task when there are sufficient resources. In other words, the execution of tasks is asynchronous. In some embodiments, the servermay adopt KEDA (Kubernetes Event-driven Autoscaling) to manage the worker applications-.

122 123 132 132 120 132 132 120 132 122 123 110 132 132 124 When the worker applications-execute tasks, they call a language model. In some embodiments, the language modelis provided by other servers, and the servermay call the language modelthrough an application interface (API). In other embodiments, the language modelis deployed on the server. This language modelmay be from the GPT (Generative Pre-trained Transformer) series, Gemini, BERT (Bidirectional Encoder Representations from Transformers) series, LLaMA (Large Language Model Meta AI), etc., but the present disclosure is not limited to these. The worker applications-may input questions from the clientto the language model, and the language modelwill continuously generate data fragments, which are stored in the stream pool. Each data fragment may include one or more chunks, tokens, pixels, etc., but the present disclosure is not limited to these.

120 110 120 120 110 110 120 110 110 132 120 110 120 When the serverreceives a query about a task from the client, the serverwill first determine whether this task has been completed. If the task has been completed, the serverwill establish a one-time connection with the clientto send the corresponding data fragments to the client. If the task has not been completed, the serverwill establish a stream connection with the clientto send the corresponding data fragments to the client. After that, when the language modelgenerates subsequent data fragments, the serverwill send these subsequent data fragments to the clientthrough the stream connection. However, in this embodiment, the servermay establish different connections according to whether the task has been completed, and when the task has not been completed, data is sent in a streaming way, which can avoid unnecessary polling.

110 120 110 132 110 110 In some embodiments, the requests sent by the client, the one-time connection, and the stream connection all comply with HyperText Transfer Protocol (HTTP). For example, the above-mentioned requests include GET or POST commands, and the stream connection is a Server-Sent Event (SSE) connection. When establishing a one-time connection, the servercloses the connection after responding (i.e., sending data fragments). If the clientwants to continue querying the task, it must re-establish the connection. In contrast, when establishing a stream connection, the connection is not closed after transmitting data fragments, so there is no need to re-establish the connection when sending subsequent data fragments. In this way, if the task has not been completed, once the language modelgenerates new data fragments, these data fragments will be sent to the clientthrough the stream connection, which can reduce the waiting time for the clientto receive data fragments.

2 FIG. 2 FIG. 201 110 120 202 120 121 203 120 110 122 204 122 121 122 124 205 206 122 125 207 110 120 208 120 125 125 125 209 120 125 210 120 110 110 is a flowchart illustrating the transmission process when a task is completed according to an embodiment. Referring to, first in step, the clientsends a request to the server. In step, the serverestablishes a task corresponding to this request and adds this task to the task queue. In step, the serversends the identifier of this task to the client. Taking the worker applicationas an example, in step, the worker applicationretrieves the task from the task queueand calls the language model to execute this task. The worker applicationcontinuously receives multiple data fragments from the language model and stores these data fragments in the stream poolin step. When the task is completed, in step, the worker applicationstores the task result formed by all data fragments in the database. Next, in step, the clientsends a query for this task to the server, which includes the identifier of the task. In step, the serverqueries whether there is a task result for this task in the database, thereby determining whether the task is completed. If the databasedoes not include the task result, it is determined that the task has not been completed; if the databaseincludes the task result, it is determined that the task has been completed. In this example, since the task has been completed, in step, the serverretrieves the task result from the database. In step, the serverestablishes a one-time connection with the clientand sends the task result (including all of the stored data fragments) to the clientthrough the one-time connection.

3 FIG. 3 FIG. 2 FIG. 201 205 301 110 120 302 120 125 303 125 120 304 120 124 124 305 120 110 124 110 124 110 306 122 125 307 120 110 is a flowchart illustrating the transmission process when a task is not yet completed according to an embodiment. Referring to, steps-are the same as the process in, and thus the description will not be repeated here. In step, the clientsends a query for the task to the server. In step, the serverqueries whether there is a task result for this task in the database, thereby determining whether the task is completed. In this example, since the task is not yet completed, in step, the databasereturns a query result indicating that the task is not yet completed (i.e., no task result is available) to the server. In step, the serversends a request to the stream pool, requesting the stream poolto transmit the stored data fragments. Next, in step, the serverestablishes a stream connection with the clientand continuously sends the existing data fragments already stored in the stream poolto the clientthrough the stream connection. When the language model continuously generates subsequent data fragments, these subsequent data fragments are also added to the stream pooland sent to the clientthrough the stream connection. In step, the task is completed, and the worker applicationstores the task result in the database. The task result includes the aforementioned stored data fragments and subsequent data fragments. In step, the serversends a message to the clientto indicate that the task has been completed.

110 132 120 120 110 120 124 110 110 120 120 124 110 In this embodiment, the clientdoes not directly call the language modelbut through the server, in other words, the serveracts as an intermediary between the clientand the language model, one of the benefits of doing this is that the servermay temporarily store data fragments. In this embodiment, data fragments are temporarily stored in the stream pool. If the clientloses data fragments, for example, when events such as application crashes, power outages, system crashes occur, the clientmay send a message to the server, and the serverwill retrieve the data fragments from the stream pool, and send these data fragments to the clientagain.

4 FIG. 4 FIG. 400 410 420 120 420 110 410 410 420 410 420 110 420 420 410 120 120 132 120 420 410 410 420 is a schematic diagram illustrating a stream transmission system according to an embodiment. In the embodiment of, a stream transmission systemincludes an application frontend, an application backend, and the server, where the application backendis also referred to as the client. In this example, the application frontendis used to provide a user interface, where the user may input questions to be answered by a chatbot. For example, the application frontendruns on user devices such as mobile phones, personal computers, etc. The application backendserves as a server for the application frontend, used to receive user input (for example, a question). The application backendalso acts as the clientmentioned above, in other words, the application backendplays the role of triggering tasks and getting task results. The application backendreceives user input from the application frontend, and generates a request to the serverbased on this user input, where the serverestablishes a task and calls the language modelin an asynchronous way to complete this task. The serverwill send the data fragments output by the language model to the application backend, which then sends these data fragments to the application frontendto be displayed on the user interface (i.e., the chatbot's answer). In some embodiments, the application frontendand application backendmay connect with any protocol, and the present disclosure is not limited in this regard.

410 In this embodiment, the application frontendis related to a chatbot, but in other embodiments, it may also be an image generation tool, code generation tool, summarization tool, etc., and the present disclosure is not limited in this regard.

132 110 420 410 Through the technical means disclosed in this disclosure, the data fragments output by the language modelare temporarily stored in the stream pool, and when the clientqueries about the task, the data fragments may be sent to the application backendin real-time, and subsequently presented on the application frontend, thus reducing the waiting time for users to see the answer.

120 110 132 132 120 131 120 120 122 501 132 131 122 132 5 FIG. 5 FIG. As mentioned above, the serverserves as an intermediary between the clientand the language model. In addition to calling the language model, the servermay also use any cloud service, or the serveritself may execute functions to assist in completing tasks. In some embodiments, the servermay implement function calls.is a flowchart illustrating the implementation of function calls according to an embodiment. Please refer to. The worker applicationis used as an example for explanation here. User input is converted into requests and tasks. At step, a first prompt and function definition corresponding to this request are obtained and sent to the language model. The first prompt may include the user's input, and the function definition may be extracted from a default database or provided by the cloud service. In some embodiments, these functions belong to REST (Representational State Transfer) application programming interfaces. For example, if the user input is “Please translate the file at the following link to English, https://file.to.trans/”, the function definition includes definitions for a file translator and an image recognizer. In this embodiment, the worker applicationsends multiple function definitions, and the language modeldetermines which function to call.

502 132 503 122 132 511 513 122 511 513 132 After receiving the first prompt and function definition, at step, the language modelperforms inference to determine the function to be called (referred to as a called function) and parameters for this called function. There is no limitation on the number of called functions and parameters here. At step, the worker applicationreceives the called function from the language modeland at least one parameter corresponding to this called function. Here, it is assumed that there are a total of three called functions-, and then the worker applicationexecutes the corresponding called functions-according to the parameters to obtain a function output. Continuing with the above example, the output of the language modelis “Call function: file translator. Parameters: {file location: https://file.to.trans/; translation language: English}”. The function output is “https://file.to.download/”.

504 122 132 505 132 122 132 506 122 2 FIG. 3 FIG. At step, the worker applicationsends the function output and a second prompt to the language model. For example, after combining the function output and the second prompt, it becomes “Execute file translator with parameters, get response from file translator: {translated file location: https://file.to.download/}”. At step, the language modelperforms inference and sends the response to the worker application. The response from the language modelmay be, for example, “File translation completed, here is the download link https://file.to.download/”. At step, the worker applicationadds this response to the stream pool. Subsequent steps are as shown inor, and the description will not be repeated here.

410 420 120 120 120 120 5 FIG. Since the output of the language model is mostly text, other functions are needed if other features (such as website search) are added. When calling functions, tasks such as handling parameter formats and selecting appropriate functions would increase the development complexity of the application frontendor application backend. However, according to the process of, function calls are completed by the server. In addition, the servercan also work with other application frontends and application backends, meaning that multiple applications can share the server. Having the serverto do the function calls can reduce the cost of duplicate development.

6 FIG. 6 FIG. 6 FIG. 6 FIG. 6 FIG. 120 601 602 603 604 605 606 is a flowchart illustrating a stream transmission method according to an embodiment, which may be executed by a computer system, such as the servermentioned above. At step, a request is received from a client, and a task is established based on this request. At step, when executing the task, a language model is called to continuously generate multiple data fragments, and these data fragments are stored in a stream pool. At step, a query corresponding to the task is received from the client. At step, it is determined whether the corresponding task is completed. In response to the task being completed, at step, a one-time connection is established with the client to send all of the stored data fragments to the client. In response to the task not yet being completed, at step, a stream connection is established with the client to send the existing stored data fragments to the client, continuously receive a subsequent data fragment from the language model, and send the subsequent data fragment to the client through the stream connection. Each step inhas been explained in detail as above, so the description will not be repeated here. It is worth noting that each step inmay be implemented as multiple codes or circuits, and the disclosure is not limited in this regard. In addition, the method ofmay be used in conjunction with the above embodiments or used independently; in other words, other steps may also be added between the steps of.

In the disclosure, when the server receives a query about a task from the client, the server will first determine whether this task has been completed. If the task has been completed, the server will establish a one-time connection, a Server-Sent Event (SSE) connection, with the client to send the corresponding data fragments to the client. If the task has not been completed, the server will establish a stream connection with the client to send the corresponding data fragments to the client. After that, when the language model generates subsequent data fragments, the server will send these subsequent data fragments to the client through the stream connection. Accordingly, the client does not have to continuously send query requests to the server at certain time intervals to detect the status of task execution. Still, the language model will continuously generate data fragments, which are stored in the stream pool. Even the server suddenly stops to send the data fragments to the client, the data fragments will not be lost.

In the disclosure, the server establishes different connections according to whether the task has been completed, and when the task has not been completed, data is sent in a streaming way, which can avoid unnecessary polling. In addition, when establishing a stream connection, the connection is not closed after transmitting data fragments, such that there is no need to re-establish the connection when sending subsequent data fragments. According to the way, if the task has not been completed, once the language model generates new data fragments, these data fragments will be sent to the client through the stream connection, which can reduce the waiting time for the client to receive data fragments. Accordingly, the disclosure effectively reduces the resource consumption of the server while ensuring that the client can get task execution results in real time.

Although the disclosure has been disclosed as above with embodiments, it is not used to limit the disclosure. Any person with ordinary knowledge in the relevant technical field may make some modifications and refinements without departing from the spirit and scope of the disclosure. Therefore, the scope of protection of the disclosure should be based on the appended claims.