Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method for sorting in a column organized relational database, the method comprising: sorting only key columns in the column organized relational database to reduce memory and CPU usage, wherein the sort may be performed using an in memory sort; after sorting the key columns, ordering a payload column according to an order indicated by the key columns, wherein the payload column is associated with the key columns in the column organized relational database, comprising: prior to ordering a plurality of payload columns, refraining from automatically concatenating the plurality of payload columns; and refraining from automatically de-concatenating any concatenated payload columns after the ordering the plurality of payload columns; assessing memory capacity prior to ordering the payload column.
This invention relates to optimizing sorting operations in column-organized relational databases, addressing inefficiencies in memory and CPU usage during sorting tasks. Traditional sorting methods often process all columns, including large payload data, which consumes excessive resources. The invention improves this by sorting only the key columns first, reducing memory and CPU overhead. The sort may be performed in-memory for efficiency. After sorting the key columns, the payload columns are ordered based on the sorted key columns. The method avoids automatically concatenating or de-concatenating payload columns during this process, preserving data integrity and reducing unnecessary operations. Before ordering the payload columns, the system assesses available memory capacity to ensure efficient resource utilization. This approach minimizes computational overhead while maintaining data consistency, particularly beneficial for databases with large payload columns. The invention enhances performance by focusing sorting operations on essential key columns and dynamically managing payload data based on memory availability.
2. The method of claim 1 wherein assessing memory capacity prior to ordering the payload column comprises: determining an amount of required memory necessary to order the payload column; and detecting an amount of available memory accessible to order the payload column.
This invention relates to optimizing memory usage in data processing systems, particularly for ordering payload columns in memory. The problem addressed is inefficient memory allocation when sorting or rearranging data columns, which can lead to performance bottlenecks or system failures due to insufficient memory resources. The method involves assessing memory capacity before ordering a payload column to ensure sufficient resources are available. This assessment includes determining the amount of memory required to perform the ordering operation and detecting the available memory that can be accessed for this purpose. By comparing the required memory against the available memory, the system can avoid operations that would exceed memory limits, thereby preventing crashes or excessive resource consumption. The invention also includes a preliminary step of identifying the payload column to be ordered, which may involve analyzing data structures or user inputs to determine the specific column that needs sorting or rearrangement. The method ensures that memory constraints are evaluated before initiating the ordering process, allowing for dynamic adjustments or alternative approaches if insufficient memory is detected. This approach is particularly useful in large-scale data processing environments where memory management is critical for maintaining system stability and performance. By proactively assessing memory availability, the system can efficiently handle data operations without risking resource exhaustion.
3. The method of claim 2 comprising: determining the plurality of payload columns may be ordered concurrently based the required memory and the available memory.
This invention relates to optimizing data processing in systems with limited memory resources. The problem addressed is efficiently managing payload data in a database or data processing system where memory constraints may hinder performance. The solution involves dynamically ordering payload columns based on memory requirements to ensure optimal use of available memory during concurrent operations. The method includes analyzing the memory requirements of multiple payload columns and comparing them to the available memory in the system. Based on this analysis, the columns are ordered in a way that maximizes memory utilization while preventing bottlenecks. This ordering allows for concurrent processing of the columns, improving overall system efficiency. The approach ensures that high-priority or memory-intensive columns are processed first, while lower-priority columns are handled in a way that does not overwhelm the system's memory capacity. The invention also involves determining the optimal sequence for processing these columns, taking into account both the required memory for each column and the available memory at any given time. This dynamic adjustment helps maintain smooth operation even when memory resources are constrained. The method may be applied in database management systems, data warehousing, or any system where large datasets must be processed efficiently under memory limitations. By intelligently ordering payload columns, the system avoids unnecessary delays and ensures that data processing remains efficient and reliable.
4. The method of claim 3 comprising: detecting a plurality of agents each available to perform at least one of the sorting and the ordering; and ordering the plurality of payload columns concurrently wherein each of the plurality of payload columns is ordered individually by at least one of the plurality of agents.
This invention relates to data processing systems that sort and order payload columns in a database. The problem addressed is the inefficiency of traditional single-agent sorting methods, which can be slow and resource-intensive when handling large datasets with multiple columns. The solution involves a distributed approach where multiple agents work concurrently to sort and order different payload columns independently. Each agent is assigned to one or more columns, allowing parallel processing that improves performance and reduces processing time. The system first identifies available agents capable of performing sorting or ordering tasks. Then, the payload columns are distributed among these agents, with each agent handling its assigned columns in parallel. This concurrent processing ensures that multiple columns are sorted simultaneously, leveraging the computational power of multiple agents to enhance efficiency. The method optimizes resource utilization by dynamically assigning tasks to available agents, ensuring that the sorting and ordering operations are completed faster than traditional sequential methods. This approach is particularly useful in large-scale data environments where quick processing of multiple columns is critical.
5. The method of claim 4 further comprising: determining a maximum parallelism for ordering the plurality of payload columns based on a column width of each of the plurality of payload columns, and a relative cost of performing the ordering.
This invention relates to optimizing data processing in systems that handle structured data, particularly for improving the efficiency of ordering operations on payload columns. The problem addressed is the computational overhead and resource consumption when sorting or reordering multiple columns of data, especially in large datasets where column widths and data types vary significantly. The method involves determining an optimal level of parallelism for ordering a set of payload columns. This is achieved by analyzing the width of each column and the relative computational cost associated with ordering operations. The column width is a key factor because wider columns (e.g., those containing large data types like strings or binary data) require more memory and processing time compared to narrower columns (e.g., integers or booleans). The relative cost is assessed based on the specific ordering algorithm used, such as comparison-based sorts or hash-based methods, and the hardware capabilities of the system. By dynamically adjusting the parallelism level, the method ensures that ordering operations are distributed efficiently across available processing resources, minimizing bottlenecks and reducing overall execution time. This approach is particularly useful in database systems, data warehousing, and real-time analytics platforms where performance and scalability are critical. The solution balances the trade-off between parallel processing overhead and the benefits of concurrent execution, leading to more efficient data manipulation.
6. The method of claim 3 comprising: assessing an amount of anticipated memory that will be accessible at a time during which the plurality of payload columns are ordered; determining the plurality of payload columns that may be ordered concurrently based on the available memory; and determining a sequence in which the plurality of payload columns may be ordered based on the anticipated memory.
This invention relates to optimizing the ordering of payload columns in a data processing system, particularly in environments with limited memory resources. The problem addressed is efficiently managing memory usage when sorting or reordering multiple payload columns to minimize delays and resource contention. The method involves first assessing the amount of memory that will be available during the ordering process. This assessment considers the system's current memory state and anticipated memory demands from other processes. Based on this evaluation, the method determines which payload columns can be ordered concurrently without exceeding available memory. The system then analyzes dependencies between columns and memory constraints to establish an optimal sequence for ordering them. This sequence ensures that columns are processed in a way that maximizes throughput while staying within memory limits. The approach may also include dynamically adjusting the ordering sequence if memory conditions change during execution. By preemptively evaluating memory availability and structuring the ordering process accordingly, the method reduces the risk of system slowdowns or failures due to insufficient memory. This is particularly useful in large-scale data processing or real-time systems where memory resources are constrained.
7. The method of claim 2 comprising: determining the amount of required memory is greater than the amount of available memory; and spilling to disk the ordering of the payload column using a spill sort to sort the key columns.
This invention relates to database systems and specifically addresses memory management during sorting operations. The problem being solved is the inefficiency and potential failure of sorting large datasets when available memory is insufficient to hold all required data. Traditional sorting methods may either fail or degrade performance significantly when memory constraints are encountered. The method involves a two-step process. First, it determines whether the amount of required memory for a sorting operation exceeds the available memory. If this condition is met, the system initiates a spill sort process. During the spill sort, the payload column (the data being sorted) is temporarily stored on disk rather than in memory. The key columns (the columns used to determine the sorting order) are sorted in memory, while the corresponding payload data is managed on disk. This approach reduces memory usage by offloading the bulk of the data to disk while still performing efficient in-memory sorting on the key columns. The spill sort ensures that the sorting operation can proceed even when memory is limited, maintaining performance and reliability. This technique is particularly useful in large-scale data processing environments where datasets frequently exceed available memory capacity, such as in big data analytics or database management systems. By dynamically adjusting the sorting process based on memory availability, the method ensures efficient resource utilization and prevents system failures due to memory constraints.
8. The method of claim 1 wherein ordering a payload column according to an order indicated by the key columns comprises: determining a consumption order in which a user will consume a plurality of payload columns; and performing an ordering of each of the plurality of payload columns according to the consumption order.
This invention relates to data processing systems, specifically methods for optimizing the ordering of payload columns in a database or data structure based on user consumption patterns. The problem addressed is inefficient data retrieval, where payload columns are stored in an order that does not align with how users access or process the data, leading to unnecessary computational overhead and slower performance. The method involves dynamically reordering payload columns according to a consumption order determined by analyzing how a user interacts with the data. First, the system identifies the consumption order by tracking which payload columns are accessed most frequently or in what sequence they are typically used. This may involve statistical analysis of past queries, user behavior, or predefined access patterns. Once the consumption order is established, the system reorders the payload columns in the database or data structure to match this sequence. This ensures that frequently accessed or sequentially dependent columns are stored in a way that minimizes retrieval time and computational effort. The method may also involve adjusting the ordering dynamically as consumption patterns change over time, ensuring continuous optimization. This approach is particularly useful in large-scale databases or systems where performance bottlenecks arise from inefficient data access patterns. By aligning storage order with usage patterns, the system reduces latency and improves overall efficiency.
Unknown
February 25, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.