When Queues Become Vulnerabilities: Reverse Engineering GCD, XPC Races, and macOS Detection

0:11 Awesome. So, hi everyone. My name is Olivia Gallucci and I work at Datadog. Today, I will be discussing how to detect race conditions on macOS focusing on how GCD and its misuse can lead to failures in privileged system services. So, yeah. Before we dive into the talk, I want to cover processes, concurrency, interleavings, and parallelism. Starting with a quote from Jonathan Levin, the concept of a process is inherent to all modern operating systems. A process corresponds to an instance of a running executable.

0:44 Processes are used by the system as containers for resources like virtual memory and descriptors and for maintaining execution statistics. It's important to note that processes are not runnable entities. A process corresponds to an instance of a running executable, but it's not the actual executing code. The run runnable entities in the executable are the threads. The process itself is technically only a container for one or more threads and provides virtual memory images and the descriptors and ports shared by all the threads.

1:17 Thus, when one refers to a process as executing, the correct terminology is at least one of the threads of the process PID is executing. While we won't go into this level of detail for most of the talk, it's important to understand what's actually happening. As a recent graduate, I interpreted phrases like the process is executing literally when they in fact refer to more abstract concepts. Another concept, concurrency, it is the system's ability to make progress on multiple tasks by overlapping or interleaving their execution. And you can think about using a single core or limited resources like or also like two lines of people to use one vending machine.

1:59 Next, interleavings are the different possible execution orders in which operations from multiple threads or tasks can overlap when accessing shared resources, often leading to non-deterministic or unpredictable behavior. Then there is parallelism, which refers to actually executing multiple tasks simultaneously on multiple cores. Overall, concurrency improves responsiveness and resource utilization even when tasks aren't running in true parallel. This talk is about both concurrency and parallelism. Concurrency displays the logic and timing bugs, while parallelism is a supporting concept in how GCD scales execution and how that scaling can go wrong. For example, thread overcommitment and interleaving of privileged and unprivileged operations.

2:44 So, now we can start by discussing Grand Central Dispatch or GCD. So, GCD is Apple's concurrency framework and system library that manages task queues and execution without explicit thread management by developers. In less technical terms, GCD is a queue-based API that abstracts thread management away from the developer. These task queues hold and schedule blocks of work to be done and to be executed later, while threads are again those system resources that actually run and perform those tasks.

3:15 GCD is widely used in system daemons and across cross-process communication services. It manages dispatch queues to organize work into tasks, and those dispatch queues come in two types. The first is serial queues, which ensures that like tasks run one at a time in the order they're received. The second is concurrent queues, which allows multiple tasks to execute simultaneously and it's managed by the system scheduler. Most importantly, GCD uses quality of service or QoS classes to prioritize tasks and which task gets system resources and when they run.

3:51 They are the signal that the system uses to allocate resources under contention. Thus, QoS classes directly influence scheduling priority. How? Well, the kernel scheduler uses QoS to prioritize CPU allocation affecting responsiveness and interactivity. Under the hood, their public API contracts defined by Apple and implemented end-to-end across libdispatch, pthreads, and the kernel scheduler. Using these APIs, developers can create custom dispatch queues or use global ones with these specific QoS levels.

4:23 Since these queue choices affect that scheduler availability and all of that, these queue choices become security relevant once you factor in QoS. Since both the developer and other programs can modify scheduling priority. In other words, correct QoS ensures that a high priority operation isn't delayed behind a a lower priority one, which could otherwise create exploitable timing windows. Bugs that can arise from this are things like race conditions, TOCTTOU, or in other concurrency issues. For reference, a race condition is when two or more executions access a shared state without proper synchronization, such that the scheduling interleavings can change outcomes.

5:03 If you want to learn more about this on macOS, I actually have a blog post specifically on TOCTTOU attacks and how they occur. And as well as a a podcast episode with Hackers on the Rocks essentially on that same blog post. But yeah, anyways, GCD does not automatically prevent these. Correct usage of queues and serializing access is required to avoid races. So, when dispatch queue target hierarchies and QoS propagation are misconfigured, this can create temporal vulnerabilities such as race conditions, deadlocks, priority inversions, and even sandbox escapes in the correct context.

5:37 Now, let's take a closer look at some of these vulnerabilities and starting with priority inversion. Priority inversion occurs when a high priority task ends up waiting on a low priority task. So, you can imagine there's a CEO of high priority who urgently needs a locked conference room currently occupied by an intern of low priority. But, the intern can't finish their work and leave the conference room because a middle manager of medium priority keeps interrupting them with mundane questions. The CEO is left in the hallway while the middle manager talks, effectively flipping the organization's chain of command.

6:15 Now, we can look at this in practice. On Darwin, the scheduler can indefinitely preempt, aka pause, low cost threads in favor of higher cost threads, which is unusual compared to other OS's. In practice, this means a low priority thread holding a resource like a lock might never get scheduled to release it if a high cost thread keeps running, leading to priority inversion deadlocks. This is especially acute with spin lock, which is when a thread, specifically the one that can't acquire the lock, does not sleep. Instead, it busy waits in a tight loop, aka spins, repeatedly checking until the lock becomes free.

6:51 Using plain spin locks, aka like a a busy wait lock with the scheduler help across threads of different costs, can freeze progress on Darwin OS's since it's possible for a higher cost thread to preempt a lower cost thread indefinitely. In such a case, a high priority task could be blocked behind a lower priority one. Apple's kernel actually implements priority inheritance in certain synchronization primitives to combat this, but only if you use the correct ones. And now, I'd like to cover some of the methods that are and are not possible with this.

7:22 So, first, a mutex is a traditional lock that provides mutual exclusion and supports priority inheritance. There's also unfair locks. These are lightweight, low-level locks that allow the system to avoid the overhead of fairness guarantees while still enabling priority boosting mechanisms. For For with both of these, if a high cost thread waits on a lock held by a low-cost thread, the kernel will temporarily boost the low-cost thread priority to match the high-cost thread until a lock is released. In other words, using locks like Pthread mutex or OS unfair lock, it allows that system to raise that low priority thread's cost so it doesn't stall that higher priority thread.

8:01 Unfortunately, this cost protection isn't guaranteed across all concurrency tools. Thus, many people prefer to stick to primitives like OS unfair lock instead of reader or writer locks, semaphores, or custom locking implementations and mechanisms. These more complex locks don't support cost inheritance, meaning that you risk priority or like like risks like like performance priority inheritance and then other sort of like thread priority issues. And again, this is because the scheduler has no way to know which thread should be boosted.

8:34 However, if you do use synchronization mechanisms that don't support cost, you can actually expose these priority inversions and the patterns within them. So, this becomes apparent when we look at dispatch semaphores. For reference, these are GCD counter-based synchronization primitives that let threads wait until other threads signal them without tracking which specific thread owns that resource. Both dispatch semaphores and spin locks don't carry ownership information. So, in this case, like a high priority thread waiting on a semaphore stuck in a low priority task, will simply just wait with no boost for the low priority task.

9:10 Apparently, many priority inversion issues on macOS have come from misuse of semaphores and spin locks, and online I've seen this referred to commonly as the semaphore anti-pattern. And from what I can tell, it's called the semaphore anti-pattern because it's a very common pattern that looks like a convenient way to bridge asynchronous work into a sync control flow. But, it breaks the scheduling and progress guarantees that Apple's concurrency stack is built around. The net effect is that a high cost work item sync is synchronously waiting on a low cost task and will just stall unexpectedly.

9:45 Now, there's a few ways to observe or demo this on your Mac around the systems and around apps. But the main way is to use Xcode's thread performance checker, which is a runtime tool. Essentially, it detects priority inversions at runtime. You can enable it in your scheme settings and you can run your app to get warnings. And when you run the app with this enabled, Xcode will log a warning if it catches a high cost thread waiting on a lower cost thread. For example, you might see a warning similar to what is displayed on screen.

10:18 And this indicates a potential priority inversion. So, using the tool is a great way to kind of watch for cost mismatches in real time, especially if you're learning about things like vulnerability research on macOS or or like actual developer and care about your app and stuff. So, yeah. The checker will flag both issues of priority inversion and even non-UI work running on the main thread, helping you catch these problems early. And what I've learned from this is that during code reviews, we should probably flag any scenario where a high priority queue or thread is blocking work on a scheduled item to a lower priority queue.

10:53 Cuz that cost mismatch is a real red flag for like potential priority inversion. For example, this would be something like a UI, in this case user interactive, a task that blocks on an API that runs on like a utility or background queue. And that's because these utility and background queues are lower than that user interactive queue. These patterns might not crash, but they create a window where a high priority task is needlessly impeded. Again, the kernel might band-aid these issues by boosting the thread's priority temporarily, and so reproducibility can be shaky, but that won't fix the underlying logic issue of mispriority must prioritize work.

11:34 So, yeah. Next, outside of priority version and coast mismatches, we also have a dispatch sync deadlocks, aka dispatch sync deadlocks on serial queues. Dispatch sync is a synchronous submission to a queue. Specifically, the function pauses your current thread until a specific task finishes on a target queue. However, this creates a major risk of deadlocks. If you use dispatch sync on the exact same serial queue you are already running on, such as calling the main queue from the main thread, the app instantly freeze.

12:06 If you use like any sort of system where you're blocking on yourself, it usually creates a situation where the thread ends up waiting on itself. one of the most common scenarios is if a service services listener queue tries to call back into itself synchronously. The same pitfall applies to the main dispatch queue, which is also a serial queue. Calling dispatch sync from the main thread will hang the app, and Apple's documentation actually warns explicitly against this scenario, but even without referring to Apple's documentation, there's a lot of different like blogs on this exact situation.

12:43 In one case, I saw a developer accidentally created a startup deadlock by queuing work to background threads that each did dispatch sync on the main queue. The initial workload exhausted the GCD worker thread pool. Subsequently, the main thread executed a synchronous dispatch call via an underlying system API. This forced the main thread to block while waiting for an available worker thread. Because all the worker threads were already occupied on the main thread, the system entered an inescapable circular wait.

13:16 So, the app froze instantly. The lesson from this was never call dispatch sync on a queue from within that queue or any scenario where you can end up waiting on yourself. And for reference, if you're curious, the correct approach is usually to use dispatch async for cross queue calls or to restructure the code so that you don't need synchronous callbacks to the same serial executor. Thus, scanning for any usage of dispatch sync calls is probably wise.

13:41 If you spot dispatch sync targeting a main or serial listener queue from within a callback of that same queue, you've likely found a bug. In reviews, I also recommend flagging flagging patterns like this because it almost guarantees to deadlock service under some situation. It's usually an issue of like, are you probing it correctly? And then lastly, we have resource starvation via worker thread busyness. Flooding a concurrent queue with too many unfinished tasks can exhaust the system's available threads serving other parts of your app or the OS from the resources that they need to run. And this is often known as like thread pool starvation or saturation.

14:18 This can appear in telemetry as high thread turn or CPU spikes. And if we remember from the beginning of this talk, GCD uses a thread pool under the hood to run concurrent queue tasks. And these tasks are always dequeued in a first-in-first-out order regardless of how they're executed. Thus, misusing GCD can lead to resource starvation. Funny enough, early GCD documentation even promised that the system would smartly limit thread creation, but time showed that it's easy to hit pathological cases.

14:50 One developer I saw said that after adopting libdispatch heavily, they ran into thread explosion which was surp- surprising because they expected the number of cores to more or less match the number of threads. And or sorry, the number of yeah, the number of cores would more or less match the number of threads. And then Apple's response to this was to remove synchronization entirely, but that's another story. Anyways, in other words, they discovered that their app was spawning dozens of threads far beyond core count due to tasks just blocking each other.

15:20 In extreme cases, if all GCD threads are busy, especially if they're blocked waiting on something, the system may create even more threads to try to break that stalemate. In fact, the libdispatch thread pool will spawn additional threads if existing ones are blocked to avoid deadlock, which can lead to thread explosion in addition to the thread priority issues we've already discussed. This means that your CPU could suddenly have tens or hundreds of threads just thrashing the CPU.

15:46 Such thread churn not only hurts performance, but also can starve other system components of CPU time since the scheduler is busy juggling all these other threads. Here are symptoms in telemetry might look like high thread count or rapid thread creation and teardown and sustained CPU spikes without an apparent increase in useful work done. So, Apple's actually learned pretty hard from these types of experiences. one in particular is a now abandoned API, security transforms in Mac OS 10.7. It inadvertently created a new queue and thread per task, causing severe Many Mac OS daemons in iOS 12 were later rewritten to be single-threaded to improve performance, reflecting on that realization that unconstrained concurrency can backfire.

16:32 In summary, to avoid resource starvation, you should, you know, generally limit the number of concurrent dispatches and especially avoid blocking on those calls and threads. So, yeah. if you have a situation where lots of worker blocks are stuck, for example, waiting on like locks or semaphores or synchronous calls, you risk both priority inversion and thread pool exhaustion. So, now that we've covered how GCD's queue semantics and scheduling choices can create priority inversions, deadlocks, and starvation, we want to ground this in an incident where this caused a big -oh, right?

17:07 In practice, dangerous bugs show up when privileged services assume serialization but the actual execution model is concurrent due to queue configuration mistakes. In 2018, you might have heard of this bug, it's now very famous, but a guy named Brandon Azad found a CVE in the GSS Cred XPC service that was exactly the situation. A dispatch queue targeting error that turns unexpected interleavings into an exploitable race condition and ultimately arbitrary code execution within that root context. So, what is com.apple.gsscred?

17:39 com.apple.gsscred is a macOS identifier written in Apple's reverse DNS naming scheme. The com.apple prefix is Apple's name space and the reverse domain can name convention that makes the name globally unique and in this case actually owned by Apple. macOS uses these bundle identifiers as stable IDs to label and route apps across system services across the OS. So, they'll show up in places like logs, entitlements, launch services, and permission decisions like the transparency consent and control database.

18:10 The GSS Cred portion is the specific component referring to a built-in system service involved in managing generic security services, aka GSS, credentials, most commonly Kerberos or enterprise SSO tickets. From a detection engineering perspective, seeing com.apple.gsscred in auth, keychain, or IPC and XPC telemetry is often normal SSO behavior, but it's also a useful pivot. You can correlate it with the calling process, the timing, and the volume to spot suspicious impersonation, unusual ticket operations, or unexpected processes trying to trigger credential flows.

18:46 Going back to the vulnerability, Azad found a high-impact race condition in this com.apple.gsscred XPC service. It allowed an unprivileged process to trigger a memory corruption condition in a privileged root service reachable via XPC leading to arbitrary code execution within that context. The vulnerability originated from a missing target queue configuration. While the service instantiated a serial dispatch queue for events, it never bound the queue to incoming client connections. As a result, message handlers defaulted to a concurrent queue, destroying any expected results of like serialization.

19:26 This oversight created a use after free race condition. Interleaving requests allowed one thread to deallocate a credential while another thread was still actively using it. Asahi weaponized this precise timing to corrupt memory and then inject his code. Ultimately, this incident demonstrated that implicit threading assumptions are inherently unsafe. Race windows can thus be exploited without kernel compromise, and XPC services must therefore explicitly enforce serialization to prevent user space race conditions from escalating into system-level breaches.

20:01 This exploit, despite the CVE being patched, is a really great illustration for why our work today focuses on GCD and XPC interleavings. A single queue targeting mistake turned a should-be-serialized privileged service into a concurrent handler, which created a race window that an unprivileged client could reliably hit for memory corruption and root context code execution. When auditing telemetry, I I've been monitoring for things like privileged XPC services that incorrectly assume a single-threaded execution model. This architectural flaw typically manifests through things like unmanaged concurrent paths such as omitted target queues, unsynchronized mutable states across handlers, or cross-queue blocking waits.

20:48 Furthermore, I'll investigate anomalous runtime patterns like rapid-fire XPC messaging. These bursts usually warrant scrutiny, especially if they correlate with process crashes, daemon restarts, or irregular GSS cred activity. And one thing to note is that this bug wasn't special to GSS cred. It was dangerous because a sandbox unprivileged client could reach a privileged XPC surface and exploit a concurrent concurrency mistake to cross that privilege boundary.

21:21 Moving from the CVE to sandboxing and XPC kind of frames this broader detection problem. We want to identify which daemons sit on those boundaries like root or high entitlement services reachable via XPC and focus reviews and telemetry on handler level races and trust assumptions that can turn that normal IPC into things like sandbox escapes. The macOS sandbox confines apps, but system daemons within elevated privileges often expose these XPC interfaces to those sandbox clients.

21:54 Race conditions and handlers can even escalate privileges or bypass these isolation boundaries. if you want to learn a little bit more about like how this is done and what like the setup actually looks like, XPC encapsulates Mac IPC with language bindings and Apple's docs highlight IPC setup and message delivery and then Jonathan Levin also has like I think a couple chapters on this type of thing as well. And why this matters for security is that a vulnerability in an XPC service that runs as root or with entitlements can act as a sandbox escape vector.

22:23 This elevates unprivileged code into privileged context undermining the security of the endpoint itself. Now that we've established why these bugs matter, which is sandboxed client can reach privileged services and subtle GCD or queuing mistakes can turn that boundary into either an exploit or reliability failure in daemons. And although I've discussed examples of this already, the next step is turning that understanding into something that we can actually action on for to like general protection engineering or vulnerability research. So, I want to start with some static review patterns.

22:57 The first one is XPC connection queuing. When a daemon accepts a new XPC connection, we want to verify that XPC connections at target queue is always called. If that doesn't happen, message handlers may execute on an unintended queue with concurrency characteristics we didn't expect. This is a strong signal because it's easy to codify and maps directly to race exposure in privileged services. The second pattern is serial versus concurrent execution assumptions.

23:27 A lot of unsafe code looks correct if you assume that requests arrive one at a time. In practice, XPC clients can create parallel pressure very easily. So, we should obviously look for, you know, logic that implicitly depends on serial handling, especially around authorization state, object life cycle, or shared caches. If the implementation relies on ordering guarantees that are not explicitly enforced, usually can be a high-value finding, maybe some moolah, you know. And the third signal is synchronous calls, especially dispatch sync, which I've which we've discussed before.

24:05 Any use of dispatch sync in a privileged daemon, I feel like should be scrutinized at this point or treated as suspicious and maybe should require explicit documentation because I've just seen this in like every blog post I've read on this subject. Not every synchronous dispatch, at least like I've looked into, is wrong, but it often indicates blocking behavior, lock inversion risk, or a path towards deadlock under load. And this is the kind of construct that like might behave fine and happy path testing, but will still fail under adversarial timing.

24:34 The last pattern is shared mutable state without synchronization. We want to identify global singleton state or shared objects that are accessed from multiple handlers without locks, atomics, or a dedicated serial queue funnel. This is one of the most like common root causes behind timing-dependent flaws. And from like a business standpoint, this is also where secure coding guidance can probably have the biggest return because the same review rule prevents both reliability defects and exploit primitives. So, static review gets us these candidate weaknesses. Telemetry will help us see where those weaknesses are becoming an operational risk. So, let's get into what to look for in telemetry.

25:15 The first telemetry signal is crash patterns. If we see repeated crashes, assertions, or guard failures inside a privileged daemon, especially at timing-sensitive code paths, major indicator. A crash in isolation, of course, like usually just looks like a stability bug. a repeated crash pattern though, especially around state transitions, cleanup paths, or on request handling boundaries can indicate a race window trying to be hit. And I think this is at least for me has been a strong a signal for detection and prioritization.

25:45 The second signal is thread turn spikes. If a daemon suddenly starts creating an unusual number of threads, or if queue drain behavior changes sharply, that can mean that the service is under concurrency stress it wasn't designed for. And in detection terms, this is valuable because attackers probing race windows often generate this exact kind of telemetry, or at least that's what I did when I tried. And even when it's not malicious, these signals still points us to code with weak concurrency controls.

26:15 The third signal is queuing backlogs. Long wait times on dispatch queues, heavy synchronous weights, or evidence that work is piling up faster than it drains are all useful indicators. These controls often show up before a visible crash, and they can suggest deadlocks, priority inversion, or lock contention. And for us, that means we might be able to detect exploitation attempts, or really just unsafe code paths that we might want to look into earlier for VR earlier in this failure chain.

26:45 The final layer is behavioral detection. First, we should flag processes with frequent thread creation spikes relative to baseline. The idea is like relative deviation, not absolute volume. And this is because some demons are naturally very noisy. What we care about is when a process starts behaving different from its normal profile. This makes the detection more robust and reduces false positives when you're like trying to figure out what you want to look into next or what concerns you want to try to fix first.

27:14 Second, we should look at rate limiting XPC invocations. I recognize that's kind of like a controversial subject and it doesn't always make sense in all contexts, but if a client is issuing many parallel requests into a privileged service, it might be trying to probe rate race windows. And this is especially interesting when like request volume is high, concurrency is high, and the target daemon normally expects like low to moderate pal- parallelism in the context that it's in.

27:41 Even if we don't immediately block on the activity, we should at least log it, score it, and then correlate it with crashes or queuing delays. Third, we should detect queue drain time anomalies. Elevated synchronization wait times can be a strong sign of deadlocks, lock contention, or scheduling inversion. This is a good example where like performance telemetry becomes security telemetry. The same measurement that helps like SRE or platform engineering can help threat detection engineering identify adversarial timing behaviors or a way for us to probe into a vulnerability.

28:14 So, the overall model in my opinion is pretty straightforward. Static reviews tells us where race conditions are likely to exist. Telemetry tells us when those weak points are being stressed. Behavioral detections tell us when that stress looks abnormal or adversarial. So, now we know what the problems are and the detectable opportunities are. Let's go over what we covered today. So, we learned that race conditions on macOS are not just reliability bugs in the right service boundary, they become security problems. We looked at how GCD works, where common concurrency hazards come from, and why queue design close and synchronization choices matter in privileged code. We also covered how priority inversion and dispatch syncs misuse can create these deadlocks, the starving and also timing windows that are difficult to catch in normal testing. Most importantly, we grounded this in that very famous GSS credit case where a queuing misconfiguration broke serializing assumptions and turned concurrent interleavings into root context code execution.

29:15 From a detection engineering perspective once again, we should review for unsafe queuing patterns, watch telemetry for thread turn crashes or queue backlogs, and treat architectural assumptions about serialization as security relevant. I think if we do that well, we could probably catch these issues a lot earlier and probably reduce exploit opportunities in privileged macOS services. So, I have more in terms of break and go through here. The slides are uploaded online. I will say there's a ton of like YouTube tutorials and blog posts. I've linked all of the ones that I've used throughout this resource in here because they also helped me set up my environment for my research and all of that. one of them was the iCode Guy. That was the one animation that didn't fit in the color scheme with this. His is amazing. He has like 2 hours long of lectures and animated lectures on GCD and how it works and all the security primitives and he teaches you how to set up your Xcode environment. That was personally the one I used. So, if you want your setup to look like mine that I used his.

30:12 and yeah, there's even excellent documentation from Apple in this case because a lot of developers care about performance. Outside of all of this, I released a newsletter called read to read. It's on Apple security. I did this last year after I graduated. I post approximately once a month on my research or whatever I'm working on. And yeah, thank you so much for providing me the opportunity to present. Hope you all enjoyed. If you have questions, feel free to find me outside because we're actually in the other speakers walk window now. So, bye.

30:43 >>