Monitoring EC2 with Ruby - The Diigo Meta page

• 30 Jun 08
  

  Praveen Sehgal

  ec2

• 22 Feb 08
  

  Harjeet Singh

  aws ec2 it_infrastructure ruby

  • Hardware: Monitoring Your CPU

      

    Your first consideration in determining what to monitor is to figure out whether monitoring is really a good idea. If you're doing CPU-intensive work on your system (that is, a lot of calculation work), then this is a good item to monitor. Obviously, if the system seems to crash a lot or it runs very slowly, CPU is definitely on the short list of items to monitor.

      

    One of the nice things about Linux is that it has a plethora of tools available for monitoring. Using vmstat, look at the run queue time (kthr - r column). Generally, the larger this number, the busier the CPU is. You can capture this number to a variable or store it on disk, possibly creating an XML file that you can parse and compare values. The following code shows an example of how to implement CPU monitoring.

       

    require 'EC2'  require 'thread'  # this function runs vmstat and parses the output to determine if the CPU is overloaded def get_cpu_load   cpu_load = 0    # run vmstat measurement 6 times with a delay of 10 seconds between measurements   IO.popen("vmstat -n 10 6") { |io|     # the first line of input is the header; we can discard it     io.gets          6.downto(1) {       # get an array of all integers returned by vmstat       stats = io.gets.split        # sum together the run queue load       cpu_load += stats[0].to_i     }   }   cpu_load end  access_key_id = ENV['AMAZON_ACCESS_KEY_ID']  secret_key = ENV['AMAZON_SECRET_ACCESS_KEY']  instance_id = ENV['AMAZON_INSTANCE_ID']  max_instances = 2 max_instances |= ENV['AMAZON_MAX_INSTANCES']  #create the instance of the ec2 object  ec2 = EC2::Base.new( :access_key_id => accessid_key_id, :secret_access_key => secret_key )  #run the instance  ec2.run_instances(:image_id => 'ami-f9907590')  #periodically check CPU load with vmstat  th = Thread.new do    logit = File.open('avg_cpu_load.xml', 'w')    loop do          cpu_load = get_cpu_load      #log CPU performance     logit.puts("<vmstat><time>#{Time.now}</time><avg_load>#{cpu_load}</avg_load></vmstat> ")      # let's assume 9 is too much     if (cpu_load => 9)        logit.puts("<vmstat><time>#{Time.now}</time><message>maximum CPU load exceeded</message></vmstat> ")        #start more instances       ec2.run_instances(:image_id => 'ami-f9907590',        :min_count => 2, :max_count => max_instances)      end      # try again in 5 minutes     sleep(60 * 5)      end    logit.close  end 

  • Hardware: Monitoring Memory

      

    Is monitoring memory a good idea? Because a system short on memory for running  applications can obviously cause performance issues, and because this is easy to check, this is a great candidate.

      

    Memory is easy to check. On Linux you can simply use the free command and use a regular expression to pipe the output you want to a variable (or store it in an XML file, if you want it to persist between program runs). The following code example shows a way to monitor memory.

      

    require 'EC2'  require 'thread'  # this function runs free and parses the output to determine if the memory is overloaded def get_free_memory   free_memory = 0    # run free measurement 6 times with a delay of 10 seconds between measurements   6.downto(1) {     IO.popen("free -m") { |io|       # we can discard the first two lines of output       io.gets; io.gets            # get an array of all tokens returned by free       stats = io.gets.split        # we're only interested in the fourth item in the array       free_memory += stats[3].to_i     }     sleep(10)   }   free_memory / 6 # return the average free memory available for all 6 measurements end  access_key_id = ENV['AMAZON_ACCESS_KEY_ID']  secret_key = ENV['AMAZON_SECRET_ACCESS_KEY']  max_instances = 2 max_instances |= ENV['AMAZON_MAX_INSTANCES']  #create the instance of the ec2 object  ec2 = EC2::Base.new( :access_key_id => accessid_key_id, :secret_access_key => secret_key )  #run the instance  ec2.run_instances(:image_id => 'ami-f9907590')  #periodically check memory usage with free  th = Thread.new do    logit = File.open('avg_free_memory.xml', 'w')    loop do          free_memory = get_free_memory      #log free memory available     logit.puts("<free><time>#{Time.now}</time><avg_free_mem>#{free_memory}</avg_free_mem></free> ")      # let's assume 64 megabytes is too little available memory     if (free_memory <= 64)        logit.puts("<free><time>#{Time.now}</time><message>maximum memory load exceeded</message></free> ")        #start more instances       ec2.run_instances(:image_id => 'ami-f9907590',        :min_count => 2, :max_count => max_instances)      end      # try again in 5 minutes     sleep(60 * 5)    end    logit.close  end 

  • Hardware: Monitoring Disk I/O

      

    Is monitoring disk input/output (I/O) a good idea? The answer in this case depends on whether you are actually experiencing an I/O problem. If you decide you need to monitor I/O, it would be reasonable to use iostat to monitor for the iowait parameter--the time the CPU is waiting while it has no pending I/O request. If this number is high, it's an indication there is an I/O bottleneck. As in the hardware monitoring examples discussed in previous sections, this number can be extracted using regular expressions and either  be assigned to a variable, or to output to XML for later processing.

      

    However, it's important to realize that iostat is a tool that should not be run too often. Because it has a high overhead, iostat can contribute to the very problem for which it is monitoring. So, monitor your I/O if you suspect a problem, but don't set it up for continuous monitoring.

      

  1 more annotation...

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.