Netcat, ruby and pstools for iislog searching

Today I had the need to find the longest query string in our IIS logs. These daily logs are large, at about 2GB per web server for dozens of web servers. I cobbled together a short Ruby script to parse the file:

#/usr/bin/ruby

longestq = 0

STDIN.each {|line|
  # if the entire line isn't long enough, skip
  next unless line.length > longestq

  # grab the order of the fields in case they change
  if line.match(/^#Fields/)
     line.split(/ /).each_with_index {|f,i|
        puts "#{f} => #{i}"
        @fields[f] = i - 1
     }

  # skip other # lines or if we don't yet have field defs
  elsif line.match(/^#/) or @fields.keys.length == 0
     next

  # start parsing
  else
     tmpstr = line.split(/ /)[ @fields['cs-uri-query'] ]
     if tmpstr.length > longestq
        print tmpstr.length
        puts " #{tmpstr}"
        longestq = tmpstr.length
     end
  end
}

However, since ruby is not installed on our web servers, I need to get the file to the linux server where I'll be doing the parsing. Netcat seems to fit the bill well. On the linux system, I started netcat listening on port 6666 and piped into gzip then pv and finally my script from above:

$ nc -l -p 6666 | gzip -d | pv | iislog-search.rb

nc in l(isten) mode on p(ort) 6666, use gzip to d(eflate), give me a transfer rate, then send to the ruby script.

Now I need a good way to fire it off on each web server, hopefully without having to RDP to each first. A small batch file using pstools should do the trick. This file will run on the remote system, so will need to access network shares to get to gzip.exe and nc.exe. 'Server1' is the machine with gzip and nc.exe on its D drive. 'LinuxHost' is where the netcat listener and ruby script are running.

\\Server1\d$\gzip.exe -c d:\logpath\ex090202.log | \\Server1\d$\nc.exe LinuxHost 6666

Gzip the log file, sending the compressed output to nc.exe, which will then send it to my LinuxHost on port 6666. With that (single line) in a file named 'd:\scripts\sendlogfile.bat', I can execute pstools for each server I'd like to examine:

c:\>psexec -u domain\myuser \\targetwebserver1 -c d:\scripts\sendlogfile.bat

Issues: I notice that netcat isn't quitting automatically after the transfer has completed, but I haven't had the time to track down why. When the progress from pv shows 0 B/s, I kill it and move on to the next target.