New Project: Spyder

I decided to write a simple web spider in order to learn Python, and to generate a list of urls for webserver benchmarking & stress testing… and so Spyder was born. It is written in Python 3.



a simple web spider written in Python 3

When called on a url, it will spider the pages and any links found up to the depth specified.
After it's done, it will print a list of resources that it found.
Currently, the resources it tries to find are:

images   -  any images found on the page (ie: <img src="THIS"/>)
styles   -  any external stylesheets found on the page.  CSS included via '@import' is currently only supported if within a style tag!
(ie: <link rel="stylesheet" src="THIS"/>  OR <style>@import url('THIS');</style> )
scripts  -  any external scripts found in the page (ie: <script src="THIS"> )
links    -  any urls found on the page.  'Fragments' are discarded. (ie: <a href="THIS#this-is-a-fragment"> )
emails   -  any email addresses found on the page (ie: <a href="mailto:THIS"> )

An example script for doing something like this, '', is included.  It uses apache benchmark as an example.
Eventually I'll be experimenting with 'siege' for benchmarking & server stress-testing.

NOTE: Currently the spider can throw exceptions in certain cases (mainly character encoding stuff, but there are probably other bugs too)
Getting *working* character encoding detection is a goal, and is sorta-working... ish?  Help in this area would be appreciated!
Filtering the results by domain is almost working too

My Photoshop Artwork

Recently, I’ve received a couple requests to use some of my Photoshop artwork for various purposes.  Up to this point, I’ve released all of my large pieces of art under a Creative Commons Attribution Share-Alike 3.0 Unported license.  This means you can feel free to use them and create new works based upon them as long as you give me credit somehow.  Feel free to contact me, however, because I’d love to know that other people are using it ^_^

So, as long as you mention that I’m the creator of this work somehow, then it’s all good ;-)

In case it’s not clear, the creative commons site makes this a little less confusing.  Just click on the following image:

Creative Commons License

How to Get Windows 7′s Aero Snap Feature in Linux

So I’ve just finished creating my own script based off of the tips found at LifeHacker, OMG Ubuntu, ClickOnF5, and a video here.

You may ask: why create yet another script for this?  Well, I found a couple issues with the other implementations that bugged me.

  1. After snapping, windows lost their original dimensions :-/
  2. The left & right states failed to unset the horizontal maximized window property :-(
  3. Inputting the scripts directly into CCSM, or separate scripts was a bit messy

So, I created a single script that fixes these issues, and also adds another feature: Set a window’s size back to a default state.  Currently, this default is set using a bash variable, because I usually like to keep my terminals and nautilus windows one manageable size.  I also mainly find myself only using this snap feature on nautilus and terminal windows as well, so it works for me.  Perhaps in the future there may be some need to keep track of each window’s individual default size… but that’s too complicated for now.

How to use:

Use -l for left, -r for right, -m for maximized, and -d for a “default” sized window. The default window geometry is configurable as a variable called $WIN_DEFAULTGEOM.  If the variable is set at runtime, it will override the hardcoded value in the script.  You can use this to create however many default window sizes you need if you set them in multiple commands in compiz.  See the manpage for wmctrl for the format to specify window geometry arguments. (This is called <MVARG> in the manpage).  If you’re wondering why I chose such a weird default value… it cooresponds to an 80×26 line terminal window on my resolution.


aero-resize -l  # Snap left
aero-resize -r  # Snap right
aero-resize -m  # Maximize
aero-resize -d  # Default size (as hardcoded in script)
# You may also use whatever geometry you wish like so:
WIN_DEFAULTGEOM=0,20,80,800,600 aero-resize -d

How to install:

Get the script and save as “aero-resize” someplace in your PATH (I put mine in ~/bin). Then add the commands you wish to your compiz command config, or simply use it in a terminal window to resize it.

cd ~/bin
chmod +x aero-resize

Here are some screenshots of my compiz settings. If, you’re still having trouble, follow the instructions in this video, but replace the commands he uses with my script.

Compiz config manager commands for aero snap script

Compiz config manager commands for aero snap script

Compiz config manager key bindings for aero snap

Compiz config manager key bindings for aero snap

How to build git for a host with no compiler

Recently I found myself with shell access on a host without a git client installed, and also without the necessary build tools to compile it (gcc, make, etc…).  However, I did have access to a machine with the same processor architecture (in fact, the same exact processor).  If you mange to find yourself in this situation, what you need to do is compile git statically on the machine which does have gcc and make installed.

What’s static mean?

In the compilation process, the compiler usually must link in shared dynamic libraries.  In Windows, these are called .dll files, and in linux they are usually .so files.  These libraries are simply collections of compiled functions that can be reused for many different programs that require them to do a specific task.  By sharing these libraries, the computer can save RAM and hard drive space by only requiring one copy of a specific library to be present for many programs that have been compiled for it.

In order to avoid unexpected behavior, a program must sometimes be compiled with a specific version of a dynamic library in mind.  This isn’t always true, but in order to ensure portability and expected behavior it’s important.  In linux, your package manager takes care of making sure these version dependencies are satisfied correctly.  However, this can be a problem when you’re stuck on a machine for which you have no control over.  You can’t know for sure what version of a specific library is installed, or when it will be upgraded.  You could build your program on another machine with the same processor architecture, and with the same libraries and then just copy it over, but that leaves room for breakage down the line in case your target machine’s libraries are upgraded, or if any of the libraries on the target machine are compromised or replaced by malicious versions.  Here’s where statically building comes in handy.

How to build git with static linking

This example assumes you already have access to a machine with build tools already installed.  This build machine is also assumed to have the same processor architecture as your target machine.  You can find the latest stable release of git at:

Here are the steps to take:

1) On your build machine, get the source code for git, unpack it, and go into the source directory:

$ wget
$ tar -jxvf git-
$ cd git-

2) Configure git to install to a predetermined directory, with static linking.  (Replace /home/myuser/git-static
with whatever path you want):

$ mkdir /home/myuser/git-static
$ ./configure --prefix=/home/myuser/git-static CFLAGS="${CFLAGS} -static"

3) Make git and install it to your directory

$ make
# Optional: make man pages and documentation
# Assumes you have asciidoc and other required programs on your build machine
$ make doc
# Install to your target directory
$ make install

4) Assuming all went well, now you can pack it up into a tarball for transfer to your target machine.

$ cd /home/myuser/git-static/
$ tar -cjvf git-static.tar.bz2 ./*

5) Copy it over to your target machine however you can, and unpack it to your home directory there with tar:

$ cd ~
$ tar -jxvf git-static.tar.bz2

# Check that you can use git.
# If you can't, make sure that your ~/bin directory exists in your environment's $PATH
$ git --version

6) Enjoy!

My Yubikeys Three, OpenID, and Productivity

My three yubikeys

My Yubithree

I just received a couple of brand new version 2 yubikeys!  Previously, I had been one of the first round of people to purchase the version 1 yubikey.  Originally back when I had first received my version 1 key, I had high hopes of playing around with it as a home project.  However, things in real life managed to pull me away from getting very far with it.  I had planned to help with the yubico-pam module for linux, or perhaps mess around in PHP or Ruby to implement my own authentication server.  This reminded me about my current struggle with Getting Things Done (GTD).  Along the way, I’ve learned a couple of productivity tips that I’d like to share.

With college coming to a close, I ended up having too much to handle between my Security & Privacy, Java/Datastructures, and Senior Design courses.  (That’s not to mention the internet radio show and DJ-ing.)  However, I did learn a thing or two about SSL, Java programming, and RFID last year.  Knowledge is always good!

Being Productive with Projects

Lately, I’ve been really wanting to pick up some of my old abandoned projects so I can try to break my old habits of procrastination that I learned too well in college.  There are mainly three things that I’ve been battling with to do this:

Time Management

The first thing I started learning was to try and figure out how to schedule the various things that I need to get done each day.  It’s a lot harder than it sounds. The first bad habit I had to break here was the nocturnal sleep schedule that college had ingrained in me.  Getting a job helped with that, because it started forcing me to get up early.  I’ve always been somewhat of an insomniac anyway, and have learned that if I don’t get enough sleep one night, then the next I’ll definitely be really tired and want to get to bed early.  The first few weeks were painful, but I think most of it is out of my system.

Learn to break the inertia

The next obstacle to combat procrastination is to create a small goal for yourself and then just start working towards it.  I’ve found that the longer I work with a specific goal in mind, the easier it is to keep going.  This works even after stopping for the day.  Once you start your wheels turning, it’s definitely easier to motivate yourself.

Don’t get distracted

Here’s the one I’m currently fighting.  I’ve always been able to multitask, but there’s a point where this gets really distracting.  I’ve found that my first problem here is surfing the web to research one thing, and then the next thing I know I’m looking up why cats lick themselves so much.  I’m not sure if it’s ADD/ADHD related.  The other problem with getting distracted has to do with subtasks.  I usually start task A, and then realize that I need to get subtask B done first.  This can go on forever until I forget which task I started out on, or which order I needed to get them done in.  (Perhaps a stack could help here).

To combat this, luckily I’ve found a couple tools that can help.


The first is a plugin for Eclipse called Mylyn.  I first talked about it in my post titled “New Improved Development Environment“.  It’s made managing tasks in programming much easier.  The main idea here is that for any given programming “task”, you’ve got a set of files that are related to it.  For example, your task could be “fix CSS bugs”, so you would have a bunch of CSS files open that you’re editing.  Mylyn takes care of this problem by keeping track of which open file tabs are associated with what task.  It calls this a “context”.  The other thing Mylyn does is integrate with various bug trackers, so you’ll see all your current bugs as tasks in Eclipse.  There’s also a professional version of Mylyn called “Tasktop” that takes this idea of task-oriented tab management to other applications, including Firefox!  Sadly, you’ve gotta pay for it, and it’s kinda pricey ($99).

The next tool I found was inspired by my desire to have something like Tasktop.

Managing Tabs in Firefox

We’ve all probably had the problem of too many tabs in Firefox.  Even if the tab overload doesn’t bring the browser to a crawl, it sure does slow down my productivity!  That’s where the TabMix Plus plugin comes in handy.

TabMix is a session manager plugin for Firefox.  Firefox has it’s own built-in session manager, but it’s mainly limited to one session which it restores on the occasion of a crash.  TabMix lets you manage multiple sessions, which can be composed of window(s) that contain multiple tabs.  Before I had this plugin, I would generally have a bunch of subtasks (or even completely different tasks) going at once, all in a single Firefox window!  Now, I organize my FF windows by separate subtasks, and save each session with some description of the main task that I was doing.  It’s a bit more overhead to manage and organize tabs, but it definitely keeps me much more focused and clear headed.  The other nice thing is that I can set new tabs to open and load Google.  Now a web search is only a Ctrl+t away!

(Tip: Google desktop also has a nice shortcut Ctrl+Ctrl that will let you type in a query and search either your desktop or the web fast too)

It’s also got nice features like: make new tabs to open next to the current one, window and session histories, control what the mouse buttons do on each tab, and a bunch of other great configuration options that I won’t mention here.  Sadly the organizational features of TabMix are a bit lacking when compared to Tasktop, but overall it’s definitely a good thing to have.

LyraPhase 004 – Losing Sleep (Out of REM Phase)

Update: It’s also been uploaded to soundcloud & mixcloud now:

LyraPhase 004

Losing Sleep (Out of REM Phase)

The first “Out of Phase” mix in the podcast. This one was originally recorded on 5/20/2010, and mastered throughout the first week of June. I just put the finishing touches on it, and I’m very proud of the result.

Geez… Finally!

I’m really psyched to finally release this mix!

Getting to this point was really exciting, simply because I really wanted to pay attention to the quality of this mix.  Why?  Well, this one’s special to me because I hand selected some of the best 5 star tracks in my collection so far.

It’s a collection of the best dark, industrial sounding minimal techno tracks that I’ve come across in the past year and a half (plus some fun extras that don’t even fit in this category, but mix well to tell a story).

Playing with the nanoPAD

It was a blast mapping & testing out my new korg nanoPAD midi co

ntroller for Traktor, and pushing the effects to the limit while still blending the tracks seamlessly. I learned a lot of new effect techniques and fills that I showcase throughout the mix, so be sure to listen for them and see what you think. I lost a bunch of sleep mastering and preparing this one for release.

It starts out nice, a bit techy, tribal and smooth but soon becomes a sonic journey through darkness, playful rhythm d

econstruction, unrelentingly grimy basslines, hauntingly disharmonic soundscapes and finally ending with a light at the end of the tunnel.


Here’s the tracklist & the cuesheet

01) MCJ - Wou (Original Mix)
02) Lutzenkirchen - Nightforce
03) Sensient - Doof Bitch
04) Kilohertz - Breathe (Original Mix)
05) Alic - Evil Fairy (Kalumet Remix)
06) Umek - Destructible Enviroment (Loco & Jam Remix)
07) Circuit Bent - Ambiently Speaking
08) Rusko - Woo Boost (Subskrpt Remix)
09) Boris Brejcha - Schaltzentrale
10) Slinky Wire - Croaking Toads
11) Peat Noise - Show Me (Original Mix)
12) Moonbeam - Guru
13) Astronivo - Kikton
14) Derek Howell - Useless Sim Card (The Last Atlant Remix)
15) Lish - Breaking Point
16) Jori Hulkkonen - Man From Earth (Astronomer Remix)

New Job, New Plan

New Job

So I’m starting a new job today!  It’s going to be exciting, but sadly I’ll have less time to dedicate to other projects.  However, this could also be a good thing by forcing me to focus my attention on only a couple projects at a time.

Plan B

Ok, so I know I’ve been promising a new mix for a very LONG time now.  I admit, I’m a bit of a perfectionist, but lately whenever I start a project lately, either something unforeseen happens (ie: headphones destroyed in a tragic accident involving a disobedient 4 year old), or I get distracted with the many other projects I’m working on.

The large backlog of mixes for the original broadcasted LyraPhase radio show is now seeming quite insurmountable.  Because I have so much new music that I want to play, this has become an irritation to me.  I would really love to remix all my old sets, however I’m really itching to play some new stuff too.

Here’s the new plan, in easy to digest steps:

  1. Create new mixes, or choose an old set to remix
  2. Add new music
  3. PROFIT!

So I’m going to either abandon the old numbering scheme I had, or simply insert new mixes as I see fit.  The goal is to give myself more incentive to release more regularly and make the show more structured in the process.  I’ve already taken steps toward completing 2 mixes.  I’ve added new tracks and restructured my promised psytrance mix “PsycheDatum” (aka: LyraPhase 004), and I just completed take 1 of a brand new dark minimal mix which I’m currently calling “LyraPhase 004.0 – Out of (REM) Phase”.

Overall, I’m much happier with the track selection for the original 004, and I’m more excited to play some of my top rated picks from the past couple of months.  Finally got my mixing setup in working order, and I’m playing with my new Korg nanoPAD as well.  I’m in the process of mastering 004.0, and may just need to touch up a couple parts due to some new MIDI mappings in Traktor that I’m not quite used to yet.

Either way, I’m feeling a large surge in musical motivation that I hope will continue through the summer.  This time I promise I’ll make a release before the end of the month.

Studio Monitor’s are Back!

So I finally got my KRK Rockit 8′s back from getting their tweeters replaced!  I’m extremely happy about having a good reference system to work with now.  It was quite hard to go back to standard consumer speakers after getting so used to the quality of the Rockits for so long.


Anyway, expect a remixed re-release of LyraPhase 004 – PsycheDatum soon!  Because I’m not planning on releasing many Psytrance mixes, I wanted to redo this one with a new track selection.  I may be releasing the original alongside it just because it contains a couple tracks I decided to scratch out from the setlist, yet I really did like quite a bit.  The main reasons for the re-release of this one is that I was unhappy with the flow, and the fact that I seemingly couldn’t stop myself from putting multiple tracks from the same artists on it.  I definitely blame this on the disorganized and frantic setlist planning done back when I had to make & practice a new set each week and still get myself through those last college courses in the fall.

I’m quite self critical sometimes, and really was expecting better although I was very rushed at the time.  Plus, I also have had a chance to give quite a large number of new tracks a listen in the past couple months, and just had to add a couple of them to the set.

Either way, if Psytrance isn’t your bag, then you probably won’t care either way.  In that case, look forward to LyraPhase 005, in which I made an adventurous attempt to mix Electro, House, Progressive, and even a bit of Trance-ish stuff.  It was a bit experimental, that’s for sure.  Still I’m really looking forward to doing some completely new sets, especially with all the excellent Minimal and Tech House I’ve been listening to lately.

On to other news:

Job Offer:

I’ve finally got a job offer!  Really looking forward to interviewing for a hardware (EE) related job this time.  I really have been needing some good electronics experience to put on my resume.  Definitely feeling like I need to brush up on my skills though.  Wish me luck!

Sound Production:

In the meantime, I’m checking out some sound engineering and production stuff while I still have the time to do so.  So far I’ve started reading “Computer Sound Design Synthesis Techniques and Programming” by Eduardo Reck Miranda.  I had no idea where to start, and this book seems like it’s a good point.  The first chapter goes over some basics of sound synthesis, and some real basic stuff about audio sampling, file formats, and ultra basic programming concepts.  It’s definitely written for someone who doesn’t know much about programming and how computers work, but skipping over those parts just makes me feel more productive anyway ;-) .  The way in which it describes sound generation using oscillators and function generators is also very intuitive from an Electrical Engineering standpoint.  I hope to learn a lot quickly from this book, and move on to other sources more directly applicable to Ableton and VST synths.

Any suggestions for reading material on electronic music production would be greatly appreciated!

New Improved Development Environment!

I finally got my home development server completely updated, including a freshly compiled Gentoo hardened kernel!  Now that I’ve got my server setup and working smoothly again, I started looking into the IDE side of the equation so I could do PHP web development on my laptop.

So after looking around a bit, I stumbled upon the idea of using Eclipse to do PHP development.  In the past I have disliked Eclipse due to it’s tendency to have problems with it’s workspace “.metadata” files over time, along with it’s slowdowns and/or freezing.  However, after seeing a presentation about Mylyn I reconsidered.  After looking up some other plugins, I was convinced that Eclipse is definitely worthy of a second look.  What’s Mylyn you ask?  In a nutshell: Mylyn is a task oriented plugin to Eclipse, giving you the benefit of saving what files & tabs you have open in Eclipse for a specific task.  A task can be anything, a bug report in Bugzilla that you’re working on, or simply a powerpoint presentation (An example given in the presentation with Tasktop Pro, the fully featured task oriented desktop app from Tasktop Technologies).

Why am I reconsidering Eclipse?  Well for starters:

  1. It’s built on Java, so I won’t be tied to using Windows for my laptop forever (Eventually I’m looking into getting a Mac)
  2. Mylyn allows integration with Bugzilla, along with a solution to my constant “too many tasks with too many tabs” problem.
  3. It includes built-in task scheduling features, perfect to start training myself to do better time management.
  4. Allows for developers to share “contexts” for each task (or bug) with one another, allowing for easy views on what parts of the code a bug/feature affects.  Collaboration is made that much easier!
  5. The PHP Development Tools (PDT) project gives PHP code completion, PHP debugging (once you install an apache server library), and all the other nice standard features of Eclipse.  For the Apache module, you’ve got the choice of either the free & open source XDebug or the binary blob Zend Debugger.
  6. The Subclipse plugin allows for nice integration with SVN (although I prefer git, I am forced to use for a couple projects).  I was also familiar with using this plugin in my college’s Software Development class, where we used Eclipse & SVN to do Agile Java programming with many different teams over the course.
  7. The Ajax Tools Framework (ATF) gives many of the features that the FireBug plugin for Firefox supports including: DOM Inspector, JavaScript Debugging, live CSS style editor, and all that good stuff.  It does this by embedding Mozilla into Eclipse!

I’m really excited to start debugging PHP code on the server.  Previously I’d been using jEdit, an SSH terminal, and Firefox to develop.  This upgrade should improve my productivity a lot.

Sanity at last!

In case anyone’s been wondering why my blog hasn’t been updated for a while… there are a bunch of reasons.  A 1 sentence reason would be: my environment went insane!  It all started with the lack of organization in the past couple months, which began with me graduating from college (w00t!).  After this, I ended up with a lot more time than I’m used to having, along with the lack of ability to use it properly.  All of this, along with big changes in my life led to a big disorganized mess building up over time concerning my site, development environment, and life in general.

Here’s a quick summary:

  1. Started looking for a job, put off development for a couple weeks while I celebrated graduating, tried to figure out my living situation, and dealt with my first job offer fizzling due to interviewing for a position that wasn’t sure to exist.
  2. Moved in with my girlfriend, and moved my development webserver with me (a Gentoo hardened machine named Polaris).
  3. The tweeters on my studio monitor speakers blew out during a frequency sweep in the 19-20kHz range!  Note to self: CHECK LEVELS FIRST IN THE AUDIBLE RANGE! Especially before trying to reproduce frequencies on the border of the inaudible range!
  4. Finally ran an ‘emerge -avuDN system’ and ‘emerge -avuDN world’ on Polaris.  Had to resolve a couple blocks that arose & rebuild a bunch of stuff multiple times (ran ‘etc-update’ and ‘revdep-rebuild’ ad nauseum).
  5. Fixed database backup scripts that kept emailing me once a day about how they were failing.
  6. After server was updated, tried updating WordPress via the method I previously posted about & ran into a big issue with SVN.  I ended up having to rebuild from a fresh SVN checkout of the latest WordPress release, and re-checkout my installed plugins & other changes on top of that.
  7. After rebuild, so far site seems stable & development server is also mostly sane & stable.  Success at last!

This whole process happened over the course of months, alongside taking care of stuff with life in general.  My monitor speakers are currently in the process of making their way back to me after getting new replacement tweeters.  As such, I’m still waiting to be able to release and/or redo the mixes for the new LyraPhase episodes that were aired on after October 12th, 2009 (Episodes 004 -> 011 !!).

NOTE: I’m no longer in college, so the live radio show is now over with.  However, I am planning on doing a podcast online.  This will start with the remaining original LyraPhase episodes, and will continue from there.

I’ve currently got a recording of 004 almost ready to go, but I’m really thinking about re-recording it now, along with some of the others.  The reasons for this are: I had 1 poor track that didn’t flow well in 004, and I’m confident I can now mix much better, and have learned a couple new tricks.

To my few fans that have been waiting patiently: don’t worry, the episodes ARE coming as soon as I get my speakers back.  I’d estimate that 004 should be out sometime in the next 2 weeks (depending on the time it takes them to ship the speakers back to me).

Get Adobe Flash player