Scaler: Efficient and Effective Cross Flow Analysis

In current software systems, programmers are not required to write all programs from scratch. Instead, they could employ external libraries to quickly develop the software system by invoking “Application Programming Interface (API)”. APIs define the methods and data formats that applications can use to request and exchange information with external components or libraries. There are two common mechanisms for invoking external APIs: dynamic linking and dynamic loading, as detailed in the following.

As discussed above, XFA monitors interactions between different components in the whole system stack, where each component (either the application or a library) is treated as an island. We observe that API invocation is the way for a component to interact with the external world. Therefore, we propose to intercept all API invocations in order to collect the data for performance analysis. More specifically, XFA collects the accurate runtime and number of invocations for each API. In the end, XFA provides two views to help the user diagnose the performance issue. One is component view, which provides the following information: how much time one component spends on itself and other components. Another is API view that presents the runtime distribution information of all APIs in a library.

Scaler is a detailed implementation/profiler of XFA. Scaler aims to satisfy the following requirements:

• •

  Least Manual Effort: To use Scaler, users do not need to recompile or change any code or use custom OS or hardware.

• •

  Maximum Generality: Scaler should support different types of API invocations, such as dynamic linking and loading, but does not rely on customized linker or the availability of symbol information or the source code.

• •

  Low Recording Overhead: Scaler should not introduce high performance and memory overhead that can prevent its usage in the production environment, even given a significant number of API invocations.

In order to achieve the least manual effort, we could not employ the compilation-based approach as that requires the source code, build scripts, and the proper system environment, which is not always accessible. We also cannot utilize the preloading technique, as that will require Scaler to know all function signatures beforehand. To satisfy maximum generality, we should not rely on a particular version of dynamic linker/loader or hardware performance counters. Further, existing hardware performance counters typically utilize sampling as the basic method, which cannot provide the full tracing functionality. After excluding all profiling methods, only binary instrumentation (like Pin (Luk et al., 2005) or DynamoRIO (Bruening et al., 2003)) and software-based process tracing (e.g., ptrace (Wikipedia, 2023)) exist in existing work. However, both general binary instrumentation (like Pin (Luk et al., 2005) or DynamoRIO (Bruening et al., 2003)) and process tracing are notoriously known for their high performance overhead, easily over $20\times$, which is even prohibitively high for development phases.

To meet all of these requirements, Scaler employs a Selective Binary Instrumentation strategy, that only instruments the locations related to API invocations. By using selective binary instrumentation, there is no need to recompile the code, rely on the custom hardware, and couple with a specific version of the linker or loader. By limiting the scope of instrumentation, Scaler reduces the instrumentation overhead compared to general binary instrumentation that instruments or checks the whole binary. Further, Scaler’s selective binary instrumentation supports different modes of dynamic linking and dynamic loading. For APIs defined by .rela.plt, Scaler instruments the binary of the corresponding .plt entries. For APIs defined by .rela.dyn, the address resolution occurs before the program’s execution begins. For these scenarios, Scaler performs the binary instrumentation by changing the corresponding .got/.plt.got entry directly. Scaler instruments within the program’s startup code, just before the main() function, so that the address resolution of these APIs has been completed. For dynamic loading, since dlsym() typically returns the resolved address to the program, Scaler re-direct the return addresses to the custom common interceptor by instrumenting and intercepting the dlsym invocations.

Note that selective binary instrumentation alone is not sufficient to guarantee the low runtime overhead. In addition to the instrumentation, Scaler requires handling each API differently, such as storing the events of each API invocation, and then returning back to the invocation site. Unfortunately, it is not easy to complete these tasks efficiently. For instance, existing library interposition approach, like DITool (Serra et al., 2000), redirects .got.plt entries to custom functions directly. DITool requires the user to redefine each custom function with the same signatures so that they can handle the above-mentioned tasks, which is clearly not generalizable enough. In the development of Scaler, we have tried to utilize the hash table to locate the location of storing events, but this method imposed a large overhead when multiple entries are mapped to the same bucket.

To overcome the generalization and performance issue, Scaler proposes Universal Shadow Table that maps each API of different types to one shadow entry in the Universal Shadow Table, as shown in Figure 2. The shadow entry encloses all necessary information for each specific API, such as the storing location, the jumping target, and the returning target after the interception. That is, Scaler only needs to rely on the shadow entry to parse the callee (API) information in constant time. Since the size of each shadow entry is much larger than the size of a normal entry (e.g., 16 bytes for a .plt entry), it can include more information inside, overcoming the size limit of the original entry. Universal Shadow Table also makes it possible to utilize a common interceptor (as shown in Figure 2) to handle different types of APIs. Overall, the Universal Shadow Table achieves the maximum generality and low performance overhead.

Another potential issue for the profiling is the memory or storage overhead, as existing work typically appends the recorded events one after the other, causing a proportional increase in storage/memory volume over time. Scaler proposes the Relation-Aware Data Folding to reduce its memory/storage overhead while maintaining its accuracy. This design is based on the observation of Scaler’s major purpose: Scaler requires the understanding of the time (and percentage) one component spent on other related components (component view), and the time (and percentage) spent on a specific API inside a library (API view). Therefore, Scaler could summarize the events of each API together, instead of appending the events into the log file and performing the analysis offline. By summarizing the events of each API together, Scaler’s storage overhead does not increase proportionally over the total runtime. As mentioned above, Scaler’s recording maintains the relative relationship between APIs and libraries. In cases where the same API is invoked by different libraries, the summarization will only group invocations originating from the same library together. That is the reason why such a method is called ‘Relation-Aware Data Folding”.

As mentioned above, Scaler intercepts different types of API invocations caused by dynamic linking and dynamic loading. The purpose of the interceptor is to redirect different types of API invocation to the Universal Shadow Table.

The Universal Shadow Table holds one shadow entry for each API defined in .rela.plt, .rela.dyn and dynamically loaded via dlsym(), as shown in Figure 2. The UST is critical in Scaler’s runtime.

Each shadow entry includes a set of assembly instructions (134 bytes in total) that implements the following functionalities, before jumping to the Tracer.

• •

  Reading per-thread context in TLS ($20$ bytes): Scaler checks the per-thread context at first. If the context is not initialized, Scaler skips the tracing and invokes the corresponding API directly. Note that Scaler keeps the per-thread data for all API invocations. If the per-thread context is not initialized, then it is infeasible to update the tracing data.

• •

  Increment the number of API invocations and record timestamp ($45$ bytes): Scaler increments the number of invocations for the current API. By default, Scaler always records the number of invocations. At the end of this code section, Scaler checks whether it is necessary to perform the timing, if yes then UST will invoke Tracer to record the timestamp before API execution. For extensibility, Scaler allows users to configure the frequency of collecting the runtime.

• •

  Invoking the real API ($31$ bytes): Scaler only invokes the real API directly when the per-thread context is not initialized or timing is not required based on the setting. Scaler handles differently for different types of API invocations. For dynamic loading (via dlsym) or after address resolution in dynamic linking, the execution will jump to the API address directly. For API invocation in dynamic linking (before address resolution), the UST will simulate the behavior of the original .plt entries and invoke ld-linux.so to resolve the address. The return address of the real API is recorded and replaced with the current UST entry so that the real API will return to the UST after execution. Note that before invoking the real API, the UST needs to save necessary registers (context) beforehand, and then recover them before returning to the caller.

• •

  Record the duration and return ($38$ bytes): UST will invoke Tracer again to record the timestamp after API invocation finishes. UST passes information required by the Tracer by pushing them onto the call stack. After Tracer finishes, UST will jump back to the real return address.

The Tracer component is responsible for tracing and collecting the information of each API invocation by the Universal Shadow Table.

Scaler keeps separate data for each thread. In this way, there is no need to utilize mutex locks for the update, as different threads are not updating the same tracing data simultaneously. Further, such a design reduces cache misses caused by true/false sharing (Liu and Berger, 2011), but at the cost of more memory/storage consumption. The per-thread data will be stored in separate files in the end, and the Offline Visualizer will integrate all data from different threads together in the end. Note that Scaler employs initial-exec TLS model (PeterDing, [n. d.]) to store per-thread variables, which only require a single mov instruction to access. In contrast, the default model (dynamic TLS) requires an extra function invocation called __tls_get_addr, which is very inefficient.

To collect the execution time, Scaler employs the light-weight rdtsc instruction that can read CPU clock cycles in user space efficiently, avoiding expensive system calls. It collects the timestamp before and after each API execution, and the difference between these two timestamps is the execution time of the current API invocation. The Tracer will invoke the Online Data Folder to record the collected information.

Scaler designs a data structure that folds the traced data online. The data folding will follow two principles: (1) it reduces the memory/storage as much as possible, which also helps reduce the time of offline analysis. (2) It preserves the accuracy of traced data. In the end, The data folder outputs the traced data to the disk files at the end of the execution or upon receiving the signal from users. Each thread outputs one copy of data.

Scaler’s offline visualizer includes Python scripts used to generate component views and API views based on the log files generated by Scaler runtime. As discussed above, the offline visualizer integrates the per-thread data together. Since Scaler already attributes the runtime of invocations for multithreaded programs as discussed in Section 3.4, it simply summarizes the data from different threads together in its offline analysis.

The component view shows the execution time for itself ( “Self”) and other components. For instance, the component view for the application shows the percentage of runtime spent in the application and other libraries that are invoked directly by the application, while the component view for a library also includes the runtime spent in the library (“Self”) and other libraries called by this library. In particular, the runtime of “Self” equals the total runtime of this component minus the runtime spent on all APIs invoked by this component. For the runtime of a specific library, Scaler simply summarizes the runtime of all APIs belonging to this library, which can be collected by analyzing the corresponding elf file.

For the API view, Scaler simply aggregates per-thread data together for all APIs. As mentioned in Section 3.3, each thread has a copy of the data that records API invocations by this thread. Since different threads employ the same array-based data structure for tracking invocations, Scaler only needs to summarize the data with the same index together (indicating the same API) in different per-thread arrays.

Note that Scaler’s component view displays the waiting time (shown as “Wait”) separately instead of counting them as normal API invocations of pthreads library. The API invocations related to the waiting (e.g., condition/barrier waits) indicate that programs are not actively doing useful work, leading to serious performance issues. Separating the waiting time into a distinct category helps identify such issues as mentioned in Section 4.2. Further, Scaler summarizes the waiting time from different types of threads together, reporting load imbalance issues when different types of threads have significantly different amounts of waiting time. Scaler learns such a mechanism from SyncPerf (Alam et al., 2017). Note that Scaler is able to achieve this due to its capability of the full trace.

We evaluated all the above-mentioned applications to confirm Scaler’s effectiveness and compared the result with perf. Such comparison shows the necessity of our proposed lightweight full-trace method. We only show the comparison with perf due to the following reasons: first, perf does not require recompilation of the application and libraries, which is as convenient as Scaler. Second, other full trace based profilers have significantly higher runtime overhead compared to Scaler, making it unnecessary for the comparison. Instead, perf, a sampling based profiler, has a similar performance overhead as Scaler. Other sampling based tools have both lower collection frequency and higher runtime overhead compared to perf.

Overall, Scaler detects $6$ bugs in highly-optimized benchmarks, as shown in Table 2, while perf can only detect one bug. Note that $2$ out of these $6$ bugs are reported for the first time. After fixing these bugs, the performance improves between $25.5\%$ and $76.7\%$ (as shown in the Column “Speedup”). Among these bugs, $4$ bugs are related to the design and implementation of the main application, indicating that the abnormal behavior of APIs can be utilized to infer the inappropriate design or configuration of the application. The other two performance bugs (dedup-3 and swaptions) are related to performance issues caused by external libraries. The results show the effectiveness of Scaler. Note that Scaler may miss bugs caused by non-API functions or hardware, as discussed in Section 5.1.

In the remainder of this section, we will study performance issues listed in Table 2. Since we already discussed the canneal bug in Section 1 and Figure 1, this example will be skipped. Further, since the root cause of dedup-2 (Alam et al., 2017) and ferret is similar, we only describe the ferret example here.

In this section, we further evaluate the performance overhead of Scaler, and compare it with existing work. For perf, we use sampling rate $4000$ and the “-g” option (in order to collect the callstack). For vtune, we use the default sampling interval – 10ms and select the default “User Mode Sampling (ums)” option. For bpftrace, we use Scaler to collect the list of all invoked APIs, and then write a bpftrace script to attach uprobe and uretprobes for all of these APIs. For ltrace, we use “–no-signals -o /dev/null” flag to minimize the impact of signal and output.

We also evaluated the memory overhead of Scaler, perf (sandar Milenk ovic, 2012), vtune-ums (corporation, 2022) and bpftrace (ioviser, 2013). The results can be seen in Table 5. Overall, Scaler introduces orders of magnitude lower memory overhead compared to other existing work. Specifically, Scaler only imposes $15.5\%$ memory overhead. In comparison, perf’s memory overhead is $6.7\times$, vtune-ums’s overhead is around $18.3\times$, and bpftrace’s overhead is $3.1\times$. We observe that perf and vtune-ums has relatively high memory overhead for application with small memory footprints, while Scaler still provides low memory overhead for these applications. The major reason for Scaler’s low overhead is due to its Relations-Aware Data Folding. The major memory overhead for Scaler is proportional to the number of APIs. For each API, Scaler allocates one universal shadow table ($134$ bytes) entry and one struct that records API-specific information ($112$ bytes). All runtime data are dynamically folded at runtime and do not need extra space for recording.

We further evaluate the impact of perf sampling rate on the profiling result and the runtime overhead. In theory, setting a higher sampling rate will improve profiling accuracy at the cost of higher runtime overhead. However, in practice perf’s sampling rate has limitations and will not strictly follow the frequency specified by the user. In Table 6, we increased the sampling rate from 4000Hz used in Section 4 to 8000Hz and profiled programs under two different sampling rates. We also adjusted the kernel flag “perf_event_max_sample_rate” accordingly to increase the system sampling rate limit. We then used the official “perf diff” tool to calculate the time difference for every function recorded under different sampling rates. This tool normalizes the difference value to $0\%$-$100\%$ and output a percentage value in the report(manual page, 2022). Finally, we parsed the report and find the function with the maximum difference value and reportd in column “Max output difference”. From Table 6, we can see that using a $2\times$ higher sampling rate on average only changed the maximum output difference by $0.57\%$. And the average performance overhead remains the same. This result indicates that using sampling rate 4000Hz in Section 4 is appropriate because it already reached the maximum feasible sampling rate in our experiment environment.

In the remainder of this section, we will analyze several corner cases to help facilitate the understanding of how Scaler components work together to overcome these challenges.

Scaler cannot detect hardware-related performance bugs, e.g., cache issues, especially those ones caused by application code. Such performance bugs can be detected better using perf or CachePerf (Zhou et al., 2022). Currently, Scaler does not intercept internal functions of applications or APIs of statically linked libraries, because these functions or APIs do not use the linkage table. Since Scaler works inside user space, it does not intercept kernel functions.

Scaler helps identify the number and percentage of API invocations, which share the same purpose as perf or gprof. However, all three tools still require human effort to actually find out the root cause of performance issues.

Supporting static linking libraries is a future research direction, but the proposed techniques such as Universal Shadow Table and Relation-Aware Data Folding can still be applied to profile statically linked libraries. The major technical challenge is to identify statically linked APIs inside compiled programs. But this problem can be bypassed if the symbol table is provided to Scaler. Once the address of a static linking API is known, Scaler needs to replace the first two instructions in the function to make the program jump to the universal shadow table. The instructions used for replacement are similar to what Scaler currently uses to replace the .plt. Similar to simulating instructions inside the .plt, Scaler needs to simulate the behavior of the overridden instructions. This simulation can be achieved by copying the replaced instructions to other memory regions and modifying jump targets and relative addressing instructions accordingly. Since Scaler only overrides two instructions, the simulation should not significantly impact the performance overhead.

Scaler is easy to extend, and can be connected with different performance analysis tools. For example, it could collect all synchronizations (and parameters) for synchronization analysis (Alam et al., 2017; Zhou et al., 2018); the whole sequence of API invocations will provide insights for tail performance analysis (Dean and Barroso, 2013). Despite these possibilities, Scaler is not designed as the replacement for sampling-based tools (e.g., perf) but as a complement to these existing profilers.