Accurate calculation of CPU usage given in percentage in Linux?

It's a question which has been asked many times, however there is no well supported answer I could find.

Many people suggest the use of top command, but if you run top once (because you have a script for example collecting Cpu usage every 1 second) it will always give the same Cpu usage result (example 1, example 2).

A more accurate way to calculate CPU usage, is by reading the values from /proc/stat, but most of the answers use only the first 4 fields from /proc/stat to calculate it (one example here).

/proc/stat/ has 10 fields per CPU core as of Linux kernel 2.6.33!

I also found this https://stackoverflow.com/questions/5514119/accurately-calculating-cpu-utilization-in-linux-using-proc-stat question which is pointing out the same issue, -that most other questions only take into consideration 4 out of the many fields- but still the answer given here starts with "I think" (not certain), and except that, it is only concerned about the first 7 fields (out of 10 in /proc/stat/)

This perl script uses all of the fields to calculate the CPU usage, which again I do not think is correct after some further investigation.

After taking a quick look into the kernel code here, it looks like, for example, guest_nice and guest fields are always increasing together with nice and user (so they should not be included in the cpu usage calculation, since they are included in nice and user fields already)

/*
 * Account guest cpu time to a process.
 * @p: the process that the cpu time gets accounted to
 * @cputime: the cpu time spent in virtual machine since the last update
 * @cputime_scaled: cputime scaled by cpu frequency
 */
static void account_guest_time(struct task_struct *p, cputime_t cputime,
			       cputime_t cputime_scaled)
{
	u64 *cpustat = kcpustat_this_cpu->cpustat;

	/* Add guest time to process. */
	p->utime += cputime;
	p->utimescaled += cputime_scaled;
	account_group_user_time(p, cputime);
	p->gtime += cputime;

	/* Add guest time to cpustat. */
	if (task_nice(p) > 0) {
		cpustat[CPUTIME_NICE] += (__force u64) cputime;
		cpustat[CPUTIME_GUEST_NICE] += (__force u64) cputime;
	} else {
		cpustat[CPUTIME_USER] += (__force u64) cputime;
		cpustat[CPUTIME_GUEST] += (__force u64) cputime;
	}
}

So to sum up, what is an accurate way to calculate the CPU usage in Linux and which fields should be considered in the calculations and how (which fields are attributed to the idle time and which fields to non-idle time)?

According the htop source code, my assumptions looks like they are valid:

(see static inline double LinuxProcessList_scanCPUTime(LinuxProcessList* this) function at LinuxProcessList.c)

// Guest time is already accounted in usertime
usertime = usertime - guest;                             # As you see here, it subtracts guest from user time
nicetime = nicetime - guestnice;                         # and guest_nice from nice time
// Fields existing on kernels >= 2.6
// (and RHEL's patched kernel 2.4...)
unsigned long long int idlealltime = idletime + ioWait;  # ioWait is added in the idleTime
unsigned long long int systemalltime = systemtime + irq + softIrq;
unsigned long long int virtalltime = guest + guestnice;
unsigned long long int totaltime = usertime + nicetime + systemalltime + idlealltime + steal + virtalltime;

And so, from fields listed in the first line of /proc/stat: (see section 1.8 at documentation)

     user    nice   system  idle      iowait irq   softirq  steal  guest  guest_nice
cpu  74608   2520   24433   1117073   6176   4054  0        0      0      0

Algorithmically, we can calculate the CPU usage percentage like:

PrevIdle = previdle + previowait
Idle = idle + iowait

PrevNonIdle = prevuser + prevnice + prevsystem + previrq + prevsoftirq + prevsteal
NonIdle = user + nice + system + irq + softirq + steal

PrevTotal = PrevIdle + PrevNonIdle
Total = Idle + NonIdle

# differentiate: actual value minus the previous one
totald = Total - PrevTotal
idled = Idle - PrevIdle

CPU_Percentage = (totald - idled)/totald

The following is a bash script which is based on Vangelis's answer. It produces output like this:

total 49.1803
cpu0 14.2857
cpu1 100
cpu2 28.5714
cpu3 100
cpu4 30
cpu5 25

Create a file called get_cpu_usage.sh

Run it using the following command: bash get_cpu_usage.sh 0.2

The argument is the number of seconds to measure. In this case it's 200 milliseconds.

The contents are:

#!/bin/sh

sleepDurationSeconds=$1

previousDate=$(date +%s%N | cut -b1-13)
previousStats=$(cat /proc/stat)

sleep $sleepDurationSeconds

currentDate=$(date +%s%N | cut -b1-13)
currentStats=$(cat /proc/stat)    

cpus=$(echo "$currentStats" | grep -P 'cpu' | awk -F " " '{print $1}')

for cpu in $cpus
do
	currentLine=$(echo "$currentStats" | grep "$cpu ")
	user=$(echo "$currentLine" | awk -F " " '{print $2}')
	nice=$(echo "$currentLine" | awk -F " " '{print $3}')
	system=$(echo "$currentLine" | awk -F " " '{print $4}')
	idle=$(echo "$currentLine" | awk -F " " '{print $5}')
	iowait=$(echo "$currentLine" | awk -F " " '{print $6}')
	irq=$(echo "$currentLine" | awk -F " " '{print $7}')
	softirq=$(echo "$currentLine" | awk -F " " '{print $8}')
	steal=$(echo "$currentLine" | awk -F " " '{print $9}')
	guest=$(echo "$currentLine" | awk -F " " '{print $10}')
	guest_nice=$(echo "$currentLine" | awk -F " " '{print $11}')

	previousLine=$(echo "$previousStats" | grep "$cpu ")
	prevuser=$(echo "$previousLine" | awk -F " " '{print $2}')
	prevnice=$(echo "$previousLine" | awk -F " " '{print $3}')
	prevsystem=$(echo "$previousLine" | awk -F " " '{print $4}')
	previdle=$(echo "$previousLine" | awk -F " " '{print $5}')
	previowait=$(echo "$previousLine" | awk -F " " '{print $6}')
	previrq=$(echo "$previousLine" | awk -F " " '{print $7}')
	prevsoftirq=$(echo "$previousLine" | awk -F " " '{print $8}')
	prevsteal=$(echo "$previousLine" | awk -F " " '{print $9}')
	prevguest=$(echo "$previousLine" | awk -F " " '{print $10}')
	prevguest_nice=$(echo "$previousLine" | awk -F " " '{print $11}')    

	PrevIdle=$((previdle + previowait))
	Idle=$((idle + iowait))

	PrevNonIdle=$((prevuser + prevnice + prevsystem + previrq + prevsoftirq + prevsteal))
	NonIdle=$((user + nice + system + irq + softirq + steal))

	PrevTotal=$((PrevIdle + PrevNonIdle))
   	Total=$((Idle + NonIdle))

	totald=$((Total - PrevTotal))
	idled=$((Idle - PrevIdle))

	CPU_Percentage=$(awk "BEGIN {print ($totald - $idled)/$totald*100}")

	if [[ "$cpu" == "cpu" ]]; then
		echo "total "$CPU_Percentage
	else
		echo $cpu" "$CPU_Percentage
	fi
done

Hey i was also researching for the topic and found this thread really helpful. I used Vangelis Tasoulas formula to write a small python script for this. Attached is my Python code for the issue. It loads the cpu usage per cpu_id every second. Maybe its helps others as well. Also comments/suggestions are welcome :-)

#!/usr/bin/python 
# -*- coding: utf-8 -*-

'''
Created on 04.12.2014

@author: plagtag
'''
from time import sleep
import sys

class GetCpuLoad(object):
    '''
    classdocs
    '''


    def __init__(self, percentage=True, sleeptime = 1):
        '''
        @parent class: GetCpuLoad
        @date: 04.12.2014
        @author: plagtag
        @info: 
        @param:
        @return: CPU load in percentage
        '''
        self.percentage = percentage
        self.cpustat = '/proc/stat'
        self.sep = ' ' 
        self.sleeptime = sleeptime
        
    def getcputime(self):
        '''
        http://stackoverflow.com/questions/23367857/accurate-calculation-of-cpu-usage-given-in-percentage-in-linux
        read in cpu information from file
        The meanings of the columns are as follows, from left to right:
            0cpuid: number of cpu
            1user: normal processes executing in user mode
            2nice: niced processes executing in user mode
            3system: processes executing in kernel mode
            4idle: twiddling thumbs
            5iowait: waiting for I/O to complete
            6irq: servicing interrupts
            7softirq: servicing softirqs
        
        #the formulas from htop 
             user    nice   system  idle      iowait irq   softirq  steal  guest  guest_nice
        cpu  74608   2520   24433   1117073   6176   4054  0        0      0      0
        

        Idle=idle+iowait
        NonIdle=user+nice+system+irq+softirq+steal
        Total=Idle+NonIdle # first line of file for all cpus
        
        CPU_Percentage=((Total-PrevTotal)-(Idle-PrevIdle))/(Total-PrevTotal)
        '''
        cpu_infos = {} #collect here the information
        with open(self.cpustat,'r') as f_stat:
            lines = [line.split(self.sep) for content in f_stat.readlines() for line in content.split('\n') if line.startswith('cpu')]
            
            #compute for every cpu
            for cpu_line in lines:
                if '' in cpu_line: cpu_line.remove('')#remove empty elements
                cpu_line = [cpu_line[0]]+[float(i) for i in cpu_line[1:]]#type casting
                cpu_id,user,nice,system,idle,iowait,irq,softrig,steal,guest,guest_nice = cpu_line
                    
                Idle=idle+iowait
                NonIdle=user+nice+system+irq+softrig+steal

                Total=Idle+NonIdle
                #update dictionionary
                cpu_infos.update({cpu_id:{'total':Total,'idle':Idle}})
            return cpu_infos
            
    def getcpuload(self):
        '''
        CPU_Percentage=((Total-PrevTotal)-(Idle-PrevIdle))/(Total-PrevTotal)
        
        '''
        start = self.getcputime()
        #wait a second
        sleep(self.sleeptime)
        stop = self.getcputime()
        
        cpu_load = {}
        
        for cpu in start:
            Total = stop[cpu]['total']
            PrevTotal = start[cpu]['total']
            
            Idle = stop[cpu]['idle']
            PrevIdle = start[cpu]['idle']
            CPU_Percentage=((Total-PrevTotal)-(Idle-PrevIdle))/(Total-PrevTotal)*100
            cpu_load.update({cpu: CPU_Percentage})
        return cpu_load
            

if __name__=='__main__':
    x = GetCpuLoad()
    while True:
        try:
            data = x.getcpuload()
            print data
        except KeyboardInterrupt:
            
            sys.exit("Finished")                

idnt.net has a good description for how to use the /proc/stat cpu data, include a bash-script for extracting cpu and description of the lines. I just wanted to link it here, since I found it valuable.

I was also looking for the same. Here is my ruby program based on the Vangelis Tasoulas's answer:

#!/usr/bin/env ruby
$VERBOSE = true

prev_file = IO.readlines(::File.join('', 'proc', 'stat')).select { |line| line.start_with?('cpu') }
Kernel.sleep(0.05)
file = IO.readlines(::File.join('', 'proc', 'stat')).select { |line| line.start_with?('cpu') }

file.size.times do |i|
    data, prev_data = file[i].split.map(&:to_f), prev_file[i].split.map(&:to_f)

    %w(user nice sys idle iowait irq softirq steal).each_with_index do |el, index|
        eval "@#{el}, @prev_#{el} = #{data[index + 1]}, #{prev_data[index + 1]}"
    end

    previdle, idle = @prev_idle + @prev_iowait, @idle + @iowait
    totald = idle + (@user + @nice + @sys + @irq + @softirq + @steal) -
        (previdle + (@prev_user + @prev_nice + @prev_sys + @prev_irq + @prev_softirq + @prev_steal))

    puts "CPU #{i}: #{((totald - (idle - previdle)) / totald * 100).round(2)} %"
end

The following is a bash script builds on Fidel's answer and arberg's link.

I wanted to lower the usage of cat awk grep and date calls and spend less cpu usage trying to figure out the cpu usage.

Output:

total: 4%
cpu0: 10%
cpu1: 5%
cpu2: 1%
cpu3: 1%

create a bash script using this:

#!/bin/bash

# Paramiter one used to set time in sec between reads
sleepDurationSeconds=$1

# read cpu stats to arrays
readarray -t previousStats < <( awk '/^cpu /{flag=1}/^intr/{flag=0}flag' /proc/stat )
sleep $sleepDurationSeconds
readarray -t currentStats < <( awk '/^cpu /{flag=1}/^intr/{flag=0}flag' /proc/stat )

# loop through the arrays
for i in "${!previousStats[@]}"; do
  # Break up arrays 1 line sting into an array element for each item in string
  previousStat_elemant_array=(${previousStats[i]})
  currentStat_elemant_array=(${currentStats[i]})

  # Get all columns from user to steal
  previousStat_colums="${previousStat_elemant_array[@]:1:7}"
  currentStat_colums="${currentStat_elemant_array[@]:1:7}"

  # Replace the column seperator (space) with +
  previous_cpu_sum=$((${previousStat_colums// /+}))
  current_cpu_sum=$((${currentStat_colums// /+}))

  # Get the delta between two reads
  cpu_delta=$((current_cpu_sum - previous_cpu_sum)) 

  # Get the idle time Delta
  cpu_idle=$((currentStat_elemant_array[4]- previousStat_elemant_array[4]))

  # Calc time spent working
  cpu_used=$((cpu_delta - cpu_idle)) 

  # Calc percentage
  cpu_usage=$((100 * cpu_used / cpu_delta))

  # Get cpu used for calc cpu percentage used
  cpu_used_for_calc="${currentStat_elemant_array[0]}"

  if [[ "$cpu_used_for_calc" == "cpu" ]]; then
    echo "total: "$cpu_usage"%"
  else
    echo $cpu_used_for_calc": "$cpu_usage"%"
  fi

done