Splunk: get list or count of addresses

I need to see a list of unique IPs that generated errors. The IP of our visitors are tracked in a X-Forwarded-For header as our load balancers and firewalls override the original in the IP packet. I already had Splunk set to drop the value into a variable named xfwd:

Transforms.conf:

[ezlmxfwd]
REGEX = X-Forwarded-For: ([0-9.]+)
FORMAT = xfwd::$1

Props.conf:

REPORT-xfwd = xfwd

Now for the searches. First, a list of IPs, with a count of how many times each showed up:

* | stats count by xfwd

Next, a count of distinct IPs:

* | stats distinct_count(xfwd)

More interestingly, a list and a distinct count for /24 addresses:

* | rex field=xfwd "(?[0-9]+\.[0-9]+\.[0-9]+)" | stats count by classc
* | rex field=xfwd "(?[0-9]+\.[0-9]+\.[0-9]+)" | stats distinct_count(classc)

The 'rex' command is a little more involved. It's saying to search through the field xfwd and apply the given regex. The string between the angle-brackets is the name that will be given the grouping's (inside the parens) match. Essentially we are creating a new field on the fly using regex.

.

Using canvas for graphing fun

I've been using the graphing features of Adobe Flex to present data on our dashboard. However, using canvas and gnuplot looks like a potential replacement. Lighter weight and OSS, but maybe not as pretty. Further investigation needed...

Programming for kids

Marshall Brain has a great post regarding introducing your kids to programming.

Ubuntu/Gnome remote remote desktop activation

There may come a time when you need to get to your Ubuntu desktop remotely, but the Remote Desktop access is disabled. Fortunately, you can control these settings with the gconftool-2 utility (assuming you can ssh to the server of course :-). There are 3 vital settings that will need to be changed:

gconftool-2 --type boolean -s /dekstop/gnome/remote_access/local_only true

This setting will require you to connect to your remote machine through an ssh (or vpn) tunnel. If you leave this out, anyone that can reach your machine will be able to control it. Not Good(tm). With this setting, people with accounts have access, so still be cautious.

gconftool-2 --type boolean -s /dekstop/gnome/remote_access/prompt_enabled false

Which prevents the confirmation dialog from popping up.

gconftool-2 --type boolean -s /dekstop/gnome/remote_access/enabled true

Which enables the remote service.

Optionally, but recommended, set a password for the connection:

gconftool-2 --type list --list-type string --set /desktop/gnome/remote_access/authentication_methods '[vnc]'

gconftool-2 --type string --set /desktop/gnome/remote_access/vnc_password `echo -n somepass | base64`

If you don't have base64 installed, you can create the encoded string and replace the `echo...` statement with the string there.

Now connect by opening an ssh session with the vnc port forwarded:

ssh -L 15900:localhost:5900 user@yourhostname

Finally point your vnc client to localhost on port 15900, and you're good to go.

update: If you have a X server installed on the local system, just use 'ssh -X' and run 'vino-preferences' to see the standard remote access dialog.

.

Google's custom buttons

A very interesting breakdown of how Google developed the new look to their buttons in GMail. That's a damn lot of work for such a seemingly small bit of interface. The payoff is nice though :-)

LifeHacker: Fences

LifeHacker has a short report on the excellent Fences utility for you poor blokes still using Windows.

Netcat, ruby and pstools for iislog searching

Today I had the need to find the longest query string in our IIS logs. These daily logs are large, at about 2GB per web server for dozens of web servers. I cobbled together a short Ruby script to parse the file:

#/usr/bin/ruby

longestq = 0

STDIN.each {|line|
  # if the entire line isn't long enough, skip
  next unless line.length > longestq

  # grab the order of the fields in case they change
  if line.match(/^#Fields/)
     line.split(/ /).each_with_index {|f,i|
        puts "#{f} => #{i}"
        @fields[f] = i - 1
     }

  # skip other # lines or if we don't yet have field defs
  elsif line.match(/^#/) or @fields.keys.length == 0
     next

  # start parsing
  else
     tmpstr = line.split(/ /)[ @fields['cs-uri-query'] ]
     if tmpstr.length > longestq
        print tmpstr.length
        puts " #{tmpstr}"
        longestq = tmpstr.length
     end
  end
}

However, since ruby is not installed on our web servers, I need to get the file to the linux server where I'll be doing the parsing. Netcat seems to fit the bill well. On the linux system, I started netcat listening on port 6666 and piped into gzip then pv and finally my script from above:

$ nc -l -p 6666 | gzip -d | pv | iislog-search.rb

nc in l(isten) mode on p(ort) 6666, use gzip to d(eflate), give me a transfer rate, then send to the ruby script.

Now I need a good way to fire it off on each web server, hopefully without having to RDP to each first. A small batch file using pstools should do the trick. This file will run on the remote system, so will need to access network shares to get to gzip.exe and nc.exe. 'Server1' is the machine with gzip and nc.exe on its D drive. 'LinuxHost' is where the netcat listener and ruby script are running.

\\Server1\d$\gzip.exe -c d:\logpath\ex090202.log | \\Server1\d$\nc.exe LinuxHost 6666

Gzip the log file, sending the compressed output to nc.exe, which will then send it to my LinuxHost on port 6666. With that (single line) in a file named 'd:\scripts\sendlogfile.bat', I can execute pstools for each server I'd like to examine:

c:\>psexec -u domain\myuser \\targetwebserver1 -c d:\scripts\sendlogfile.bat

Issues: I notice that netcat isn't quitting automatically after the transfer has completed, but I haven't had the time to track down why. When the progress from pv shows 0 B/s, I kill it and move on to the next target.