Building a CPU Topology on MacOS X

LMAX Exchange

Within a project I’ve been working on I’ve had the need to simulate
the capabilities of Linux’s /proc/cpuinfo on Mac OS. Specifically I
needed to build a topology of the CPUs on a given system. I.e. I need
to map the operating system’s processors to hardware threads, then build
a picture of which cores and sockets those threads reside. For example
my Mac looks something like:

CPU0 (Thread 0) ---+
|---> Core 0 ---+
CPU1 (Thread 1) ---+ |
| ----> Socket 0
CPU2 (Thread 2) ---+ |
|---> Core 1 ---+
CPU3 (Thread 3) ---+

this sounds very simple, it’s actually fraught with a number of little
niggles. Not only did it require getting down and dirty with a bit of
C++ and X86 Assembly, it also required writing a MacOS kernel extension.

The first step was to understand what information was available from the CPU. Intel exposes an instruction called CPUID.
The is the primary mechanism for getting information about the CPU.
There is a raft of information available from listing of the CPU
features available (e.g. hyperthreading) to sizes of the various levels
of cache and the associated cache lines. To access the CPUID
instruction we need a little bit of inline assembler.

The code shows how to get the vendor string from the CPU. On my Mac I get the following:

// Output:
Vendor String: GenuineIntel

those unfamiliar with Intel inline assembly, the Intel CPU defines a
number of registers. The ones used for the CPUID instruction are EAX,
EBX, ECX, and EDX (referenced as RAX, RBX, etc if using 64 bit
instructions via the REX extension). These used for both input and
output. An inline asm segment consists of 3 parts. The first part is
the instruction to be executed. In this case the "cpuid" instruction. The second line defines the output parameters. The snippet "=a" (data[0]) means store the result in the EAX register in the variable data[0]. The "=a"
refers to the 2nd letter of the register designation. The 3rd and
final section are the input parameters. The CPUID instruction takes 2
parameters, one in EAX and one in ECX.

The particular CPUID reference that provides information needed to building the topology is 0xB (11) – the extended topology enumeration leaf. The data returned from this instruction is:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
EAX | Shift | Reserved |
EBX | No. Process at this level | Reserved |
ECX | Level No. | Level type | Reserved |

extended topology enumeration leaf is one of the CPUID indexes that
also makes use of ECX as an input parameter. This indicates the level
of the CPU that you wish to work at. I started at level 0 and worked my
way up. The ‘Level type’ describes whether the current level is a CPU
thread (1) or a physical core (2) or invalid (0). All values greater
than 2 are reserved, presumably for later use. The 2 other useful
values are the x2APIC ID and Shift. The x2APIC ID is a unique identifier for the smallest logic unit in the CPU. The Shift value is used to group x2APIC ID values into units at the next logical level. This is done by shifting the x2APIC ID right by the value specified by Shift. For example on my using the following code on my workstation (2 sockets, 8 cores, 16 threads):

Outputs the following:

Shift: 1, Count: 2, Level Id: 0, Level Type: 1, x2APIC: 33, Group: 16
Shift: 5, Count: 8, Level Id: 1, Level Type: 2, x2APIC: 33, Group: 1

indicates that the hardware thread indicated by APIC 33 has the core id
of 16 and socket id of 1. The socket and core ids aren’t really
references, but values that indicate threads have have the same id share
that unit. This gets me most of the way there, I can group all of my
threads into a hierarchy of cores and sockets. However there is one
small niggle remaining. I need this code to run all of the CPUs on my
system. On Linux this is reasonably simple, you count the number of
CPUs then iterate through all of them and use the pthread_setaffinity_np(...) to specify specify which CPU to run the CPUID instructions on.

However, on Mac OS X actual thread affinity is not supported;
just a mechanism logically grouping or separating threads, powerful in
its own right, but not what I’m after here. This is where the Kernel
module comes in. The XNU Kernel defines a funky little method called mp_rendevous(...).
This method takes in a number function pointers. The key one
(action_func), is run once on each of the CPUs. So to get the topology
information for all of the CPUs we can use it like so:

Because the method mp_rendevous() is defined in the kernel the code above needs to be packaged as a kernel extension.
Even then, getting access to the method is interesting. It’s not
defined in a header that can be easily included, however it is available
at link time when compiling a kernel module. Therefore in order
compile correctly, it’s necessary to provide your own forward
declaration of the method. The same is true of the cpu_number(). Calling into the kernel method from user space requires use of the IOKit framework, but I’ll leave the details of that as an exercise for the reader.

Any opinions, news, research, analyses, prices or other information ("information") contained on this Blog, constitutes marketing communication and it has not been prepared in accordance with legal requirements designed to promote the independence of investment research. Further, the information contained within this Blog does not contain (and should not be construed as containing) investment advice or an investment recommendation, or an offer of, or solicitation for, a transaction in any financial instrument. LMAX Group has not verified the accuracy or basis-in-fact of any claim or statement made by any third parties as comments for every Blog entry.

LMAX Group will not accept liability for any loss or damage, including without limitation to, any loss of profit, which may arise directly or indirectly from use of or reliance on such information. No representation or warranty is given as to the accuracy or completeness of the above information. While the produced information was obtained from sources deemed to be reliable, LMAX Group does not provide any guarantees about the reliability of such sources. Consequently any person acting on it does so entirely at his or her own risk. It is not a place to slander, use unacceptable language or to promote LMAX Group or any other FX and CFD provider and any such postings, excessive or unjust comments and attacks will not be allowed and will be removed from the site immediately.