{"id":2752,"date":"2013-04-26T06:27:59","date_gmt":"2013-04-26T11:27:59","guid":{"rendered":"http:\/\/appcrawler.com\/wordpress\/?p=2752"},"modified":"2013-04-26T06:29:40","modified_gmt":"2013-04-26T11:29:40","slug":"linux-threads-and-cpu-affinity","status":"publish","type":"post","link":"http:\/\/appcrawler.com\/wordpress\/2013\/04\/26\/linux-threads-and-cpu-affinity\/","title":{"rendered":"Linux threads and CPU affinity"},"content":{"rendered":"

Linux provides native threading capabilities. This allows a program to run with multiple processes all accessing the same shared data, without the headache of Interprocess Communication (IPC) calls. Essentially, a single process makes calls to the system pthread library to create threads, which can be thought of as \u201clightweight processes\u201d (this is actually the Sun System V term given to them).<\/p>\n

We first show the C language code we will use to test how Linux assigns thread operations to CPU\u2019s for execution.<\/p>\n

\r\n#include\r\n#include\r\n#include\r\n#include\r\n#include\r\n\r\npthread_t thisThread[4];\r\n\r\nvoid* runNativeThread(void *arg) {\r\n\r\n  unsigned long i = 0;\r\n  for(i=0; i<1000000000000L;i++);\r\n  return NULL;\r\n}\r\n\r\nint main(void) {\r\n  int i = 0;\r\n  int returnCode;\r\n\r\n  while(i < 4) {\r\n    returnCode = pthread_create(&(thisThread[i]), NULL, &runNativeThread, NULL);\r\n    if (returnCode != 0) {\r\n      printf(\"\\ncan't create thread :[%s]\", strerror(returnCode));\r\n    }\r\n    i++;\r\n  }\r\n\r\n  for (i = 0; i < 4; i++) {\r\n    pthread_join(thisThread[i], NULL);\r\n  }\r\n  return 0;\r\n}\r\n<\/pre>\n
We then compile our program.  Note this must be linked against the pthread library.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>gcc -lpthread -o threads threads.c\r\n<\/pre>\n
We then run our program.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>.\/threads &\r\n[1] 16830\r\n<\/pre>\n
Next, we use the taskset command to show the processor affinity for our parent process as well as each thread.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>for p in $(ps -L -p 16830 | grep -v LWP | awk '{print $2}'); do taskset -p $p; done\r\npid 16830's current affinity mask: ff\r\npid 16831's current affinity mask: ff\r\npid 16832's current affinity mask: ff\r\npid 16833's current affinity mask: ff\r\npid 16834's current affinity mask: ff\r\n<\/pre>\n
The hexadecimal value of 0xff is a binary bitmask that shows each processor can be used by the process (or any given thread in the process).  Please note we have eight cores on our test server.<\/p>\n
In other words:<\/p>\n
\u2022\t00000011 = binary 3, or 0x3 in hexadecimal.  This would indicate two processors could be used
\n\u2022\t00001111 = binary 15, or 0xF in hexadecimal.  This would indicate four processors could be used
\n\u2022\t11111111 = binary 255, or 0xFF in hexadecimal.  This would indicate eight processors could be used<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     51.75      0.00      1.81      0.05      0.00     46.39\r\nAverage:          0     64.71      0.00      2.64      0.00      0.00     32.66\r\nAverage:          1     99.60      0.00      0.40      0.00      0.00      0.00\r\nAverage:          2     82.20      0.00      0.60      0.00      0.00     17.20\r\nAverage:          3     10.48      0.00      1.61      0.40      0.00     87.50\r\nAverage:          4      7.29      0.00      3.64      0.00      0.00     89.07\r\nAverage:          5     22.42      0.00      2.02      0.00      0.00     75.56\r\nAverage:          6     25.81      0.00      4.03      0.00      0.00     70.16\r\nAverage:          7    100.00      0.00      0.00      0.00      0.00      0.00\r\n<\/pre>\n
Notice almost all CPU\u2019s are being used.  We then run our command again to determine CPU distribution, and we see that the distribution has changed, although it is still fairly well balanced.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     50.98      0.00      0.53      0.03      0.00     48.46\r\nAverage:          0     29.80      0.00      2.04      0.20      0.00     67.96\r\nAverage:          1     44.56      0.00      0.81      0.00      0.00     54.64\r\nAverage:          2     88.78      0.00      0.40      0.00      0.00     10.82\r\nAverage:          3     55.73      0.00      0.00      0.00      0.00     44.27\r\nAverage:          4      7.83      0.00      0.60      0.20      0.00     91.37\r\nAverage:          5     86.20      0.00      0.20      0.00      0.00     13.60\r\nAverage:          6     15.49      0.00      0.60      0.20      0.00     83.70\r\nAverage:          7     78.24      0.00      0.20      0.00      0.00     21.56\r\n<\/pre>\n
Next, we change our CPU affinity for each thread using the taskset command.  For the sake of completeness, we show the system call to sched_setaffinity() when we run this.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>for p in $(ps -L -p 16830 | grep -v LWP | awk '{print $2}'); do strace -f -etrace=sched_setaffinity taskset -p f $p 2>&1; done\r\npid 16830's current affinity mask: ff\r\nsched_setaffinity(16830, 256,  { f, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 0\r\npid 16830's new affinity mask: f\r\npid 16831's current affinity mask: ff\r\nsched_setaffinity(16831, 256,  { f, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 0\r\npid 16831's new affinity mask: f\r\npid 16832's current affinity mask: ff\r\nsched_setaffinity(16832, 256,  { f, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 0\r\npid 16832's new affinity mask: f\r\npid 16833's current affinity mask: ff\r\nsched_setaffinity(16833, 256,  { f, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 0\r\npid 16833's new affinity mask: f\r\npid 16834's current affinity mask: ff\r\nsched_setaffinity(16834, 256,  { f, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }) = 0\r\npid 16834's new affinity mask: f\r\n<\/pre>\n
Lastly, we show our threads are now tied to the first four processors.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome\/>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     50.80      0.00      0.93      0.13      0.00     48.14\r\nAverage:          0     99.20      0.00      0.80      0.00      0.00      0.00\r\nAverage:          1     99.40      0.00      0.60      0.00      0.00      0.00\r\nAverage:          2     99.60      0.00      0.40      0.00      0.00      0.00\r\nAverage:          3    100.00      0.00      0.00      0.00      0.00      0.00\r\nAverage:          4      1.01      0.00      0.61      0.00      0.00     98.38\r\nAverage:          5      1.20      0.00      1.41      0.00      0.00     97.39\r\nAverage:          6      1.62      0.00      2.43      0.00      0.00     95.94\r\nAverage:          7      2.63      0.00      1.01      0.81      0.00     95.55\r\nqa04:oracle:qa4:\/u01\/orahome\/>\r\n<\/pre>\n
Our next test will be a java program that again creates four threads.<\/p>\n
\r\npublic class mythread implements Runnable {\r\n  public static void main (String args[]) {\r\n    for (int i = 1; i <= 4; i++) {\r\n      mythread m = new mythread();\r\n    }\r\n  }\r\n  public mythread() {\r\n    Thread t = new Thread(this);\r\n    t.start();\r\n  }\r\n  public void run() {\r\n    for (long j = 1; j <= 1000000000000L; j++) {\r\n    }\r\n  }\r\n}\r\n<\/pre>\n
We then compile our class and run it.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome>javac mythread.java\r\nqa04:oracle:qa4:\/u01\/orahome>java mythread &\r\n[1] 16483\r\n<\/pre>\n
We see that each thread created has CPU affinity of 255.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome>for p in $(ps -L -p 16483 | grep -v LWP | awk '{print $2}'); do taskset -p $p; done\r\npid 16483's current affinity mask: ff\r\npid 16484's current affinity mask: ff\r\npid 16485's current affinity mask: ff\r\npid 16486's current affinity mask: ff\r\npid 16487's current affinity mask: ff\r\npid 16488's current affinity mask: ff\r\npid 16489's current affinity mask: ff\r\npid 16490's current affinity mask: ff\r\npid 16491's current affinity mask: ff\r\npid 16492's current affinity mask: ff\r\npid 16493's current affinity mask: ff\r\npid 16494's current affinity mask: ff\r\npid 16495's current affinity mask: ff\r\npid 16496's current affinity mask: ff\r\npid 16497's current affinity mask: ff\r\npid 16498's current affinity mask: ff\r\npid 16499's current affinity mask: ff\r\npid 16500's current affinity mask: ff\r\npid 16501's current affinity mask: ff\r\npid 16502's current affinity mask: ff\r\npid 16503's current affinity mask: ff\r\npid 16504's current affinity mask: ff\r\n<\/pre>\n
We first see that our CPU\u2019s are being fairly equally used.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     50.62      0.00      0.75      0.05      0.00     48.58\r\nAverage:          0     61.69      0.00      0.20      0.00      0.00     38.10\r\nAverage:          1     68.47      0.00      0.20      0.00      0.00     31.33\r\nAverage:          2     12.73      0.00      1.21      0.20      0.00     85.86\r\nAverage:          3     54.22      0.00      1.41      0.00      0.00     44.38\r\nAverage:          4     37.68      0.00      2.40      0.00      0.00     59.92\r\nAverage:          5    100.00      0.00      0.00      0.00      0.00      0.00\r\nAverage:          6     33.80      0.00      0.20      0.00      0.00     66.00\r\nAverage:          7     35.94      0.00      0.40      0.00      0.00     63.65\r\nqa04:oracle:qa4:\/u01\/orahome>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     52.42      0.00      2.31      0.08      0.00     45.19\r\nAverage:          0     45.55      0.00      2.23      0.00      0.00     52.23\r\nAverage:          1     64.33      0.00      5.21      0.20      0.00     30.26\r\nAverage:          2     45.25      0.00      1.82      0.40      0.00     52.53\r\nAverage:          3     46.79      0.00      1.20      0.20      0.00     51.81\r\nAverage:          4     45.38      0.00      4.82      0.00      0.00     49.80\r\nAverage:          5     91.80      0.00      0.20      0.00      0.00      8.00\r\nAverage:          6     32.33      0.00      2.61      0.00      0.00     65.06\r\nAverage:          7     47.70      0.00      0.20      0.00      0.00     52.10\r\n<\/pre>\n
We then change the CPU affinity for each thread to 15, and again show our system call.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome>for p in $(ps -L -p 16483 | grep -v LWP | awk '{print $2}'); do strace -f -etrace=sched_setaffinity taskset -p f $p 2>&1 | awk '{print $1,$2,$3,$4,$5,$6}'; done\r\nsched_setaffinity(16483, 256, { f, 0, 0,\r\npid 16483's current affinity mask: f\r\npid 16483's new affinity mask: f\r\nsched_setaffinity(16484, 256, { f, 0, 0,\r\npid 16484's current affinity mask: f\r\npid 16484's new affinity mask: f\r\nsched_setaffinity(16485, 256, { f, 0, 0,\r\npid 16485's current affinity mask: f\r\npid 16485's new affinity mask: f\r\nsched_setaffinity(16486, 256, { f, 0, 0,\r\npid 16486's current affinity mask: f\r\npid 16486's new affinity mask: f\r\nsched_setaffinity(16487, 256, { f, 0, 0,\r\npid 16487's current affinity mask: f\r\npid 16487's new affinity mask: f\r\nsched_setaffinity(16488, 256, { f, 0, 0,\r\npid 16488's current affinity mask: f\r\npid 16488's new affinity mask: f\r\nsched_setaffinity(16489, 256, { f, 0, 0,\r\npid 16489's current affinity mask: f\r\npid 16489's new affinity mask: f\r\nsched_setaffinity(16490, 256, { f, 0, 0,\r\npid 16490's current affinity mask: f\r\npid 16490's new affinity mask: f\r\nsched_setaffinity(16491, 256, { f, 0, 0,\r\npid 16491's current affinity mask: f\r\npid 16491's new affinity mask: f\r\nsched_setaffinity(16492, 256, { f, 0, 0,\r\npid 16492's current affinity mask: f\r\npid 16492's new affinity mask: f\r\nsched_setaffinity(16493, 256, { f, 0, 0,\r\npid 16493's current affinity mask: f\r\npid 16493's new affinity mask: f\r\nsched_setaffinity(16494, 256, { f, 0, 0,\r\npid 16494's current affinity mask: f\r\npid 16494's new affinity mask: f\r\nsched_setaffinity(16495, 256, { f, 0, 0,\r\npid 16495's current affinity mask: f\r\npid 16495's new affinity mask: f\r\nsched_setaffinity(16496, 256, { f, 0, 0,\r\npid 16496's current affinity mask: f\r\npid 16496's new affinity mask: f\r\nsched_setaffinity(16497, 256, { f, 0, 0,\r\npid 16497's current affinity mask: f\r\npid 16497's new affinity mask: f\r\nsched_setaffinity(16498, 256, { f, 0, 0,\r\npid 16498's current affinity mask: f\r\npid 16498's new affinity mask: f\r\nsched_setaffinity(16499, 256, { f, 0, 0,\r\npid 16499's current affinity mask: f\r\npid 16499's new affinity mask: f\r\nsched_setaffinity(16500, 256, { f, 0, 0,\r\npid 16500's current affinity mask: f\r\npid 16500's new affinity mask: f\r\nsched_setaffinity(16501, 256, { f, 0, 0,\r\npid 16501's current affinity mask: f\r\npid 16501's new affinity mask: f\r\nsched_setaffinity(16502, 256, { f, 0, 0,\r\npid 16502's current affinity mask: f\r\npid 16502's new affinity mask: f\r\nsched_setaffinity(16503, 256, { f, 0, 0,\r\npid 16503's current affinity mask: f\r\npid 16503's new affinity mask: f\r\nsched_setaffinity(16504, 256, { f, 0, 0,\r\npid 16504's current affinity mask: f\r\npid 16504's new affinity mask: f\r\nqa04:oracle:qa4:\/u01\/orahome>\r\n<\/pre>\n
After doing this, we see that the first four CPU\u2019s are used by our threads.<\/p>\n
\r\nqa04:oracle:qa4:\/u01\/orahome>sar -P ALL 5 1 | grep ^Average\r\nAverage:        CPU     %user     %nice   %system   %iowait    %steal     %idle\r\nAverage:        all     50.46      0.00      0.93      0.03      0.00     48.58\r\nAverage:          0    100.00      0.00      0.00      0.00      0.00      0.00\r\nAverage:          1     98.00      0.00      2.00      0.00      0.00      0.00\r\nAverage:          2     99.60      0.00      0.40      0.00      0.00      0.00\r\nAverage:          3     98.80      0.00      1.20      0.00      0.00      0.00\r\nAverage:          4      1.42      0.00      0.61      0.00      0.00     97.98\r\nAverage:          5      1.62      0.00      0.81      0.20      0.00     97.36\r\nAverage:          6      1.01      0.00      1.21      0.00      0.00     97.79\r\nAverage:          7      1.21      0.00      0.81      0.00      0.00     97.98\r\nqa04:oracle:qa4:\/u01\/orahome>\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
Linux provides native threading capabilities. This allows a program to run with multiple processes all accessing the same shared data, without the headache of Interprocess Communication (IPC) calls. Essentially, a single process makes calls to the system pthread library to…<\/p>\n
Read more →<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[28,27],"tags":[],"_links":{"self":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2752"}],"collection":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2752"}],"version-history":[{"count":30,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2752\/revisions"}],"predecessor-version":[{"id":2782,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2752\/revisions\/2782"}],"wp:attachment":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2752"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2752"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2752"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}