Jesse Sanford

How to be a portable programmer.

2011-11-02T06:02:11-07:00

Everyone and their mother should at this point realize that portability is a virtue in the software community. So much so that it is often the primary goal of a project. Why is it then that we programmers who are striving to create portable libraries/applications are not also ourselves striving to be more portable. Too often I am confronted by a developer/engineer wanting to do something in a specific, non-standard way because thats the way “they” do things. Or “thats the way it’s done over in project X” or “thats the way we have been doing it for years.” Instead of trying to understand why you should setup your environment in the same fashion that the “community” does or using the design patterns that the core developers creating software library/framework X do, they try to backport their personal preferences into the use of the library/framework and then complain when they can’t get things to work. It should be a goal amongst software engineers to not only create portable software but rather to live in a portable mindset entirely. At least professionally. I can say for sure that if you do try at all times to be more maleable in your preferences. To use vanilla/standards-based/best-practice proven patterns and idioms in all of your professional duties. You will find your jobs easier. You will find working in groups easier. You will find life easier.

My developer mantra a.k.a rant on php frameworks

2011-09-14T09:48:05-07:00

Below is the reply I sent to the NYC PHP SIG mailing list on yet another “What are the top php frameworks” subject line… Read. Enjoy. Comment CRITICALLY and INTELLIGENTLY please.

"

damn sure… it has a wealth of CURRENT documentation and a lively community.

My mantra… (a.k.a. rant)

I don’t use software that tries to solve all problems. The worst software grows out of goal-less development. Look for something with clear, concise, decisive design decisions. The nix moto… do one thing and do it right. LESS CODE IS BETTER! A good example that will hit close to home for some and I hope won’t start a flame war is Drupal. Until recently it was a bloated mess. It has started to clean up it’s act come version 7 but in my opinion it still does WAY too much for it’s own good.

I don’t use software if the documentation is not up to snuff. Don’t take api/(insert language here)doc as enough. When you are getting up to speed in a new technology you need much more handholding or face hair loss. You need cookbooks, working examples, CURRENT api and method signatures. etc.

I dont use software if it hasn’t already been put into production some very visible places. Use something you know works. Why be the first on the block to see if X scales? You know how I know what scales. I have seen larger stacks use it in the past. Safe bets are best.

I dont use software if it hasn’t had at least 2 minor releases. Unless you need a specific feature of a bleeding edge release just use what you know will work! How do you know… because it’s been around for a while and people have used it and tested it.

I dont use software if it has no community involvement. (Don’t underestimate other peoples schedules… think about how busy you get during your average work year. You may just have a project dropped from under you. One core developer on an open source project is not enough to bet the house on.)

I prefer my software to come with unit/functional tests. If nothing else they can be used as examples. But the piece of mind you get as a developer if you know what you are using has been tested thoroughly is worth more than the most amazing untested functionality you can dream of. Just think of all the times you banged your head on your desk trying to figure out why something isn’t doing what the documentation said it was supposed to do. Only to find out yes… it is a bug in the library you are using!

I hate using software. Less is better period. If you can get away without using an extra daemon then do it. Don’t add 5 more caching layers and 3 more database abstractions just because they are the new cool acronyms. It’s only going to make your code less maintainable. Over architecting and early optimization are what keep projects from ever getting to launch.

The same thing goes for languages. Don’t write your services in more than 1 language if you can avoid it. If you are writing webapps from the ground up limit yourself to one backend and one frontend language.

Don’t add caching until you need it! The same thing goes for sharding of your database schemas! KISS… across your entire stack. Not just your code.

Whew… sorry about that. Back to the topic at hand.

Symfony 1.X is probably still more widely used than Symfony 2.x and both are good. Although 2.x is a far more abstracted and multipurpose framework 1.x is still great for the usual 3 tier webapps. Both are very well documented.

CodeIgniter is a really great lightweight framework. If you are looking for something straight forward, well documented and full of examples this is also a good choice.

Zend had so much steam early on and then where did it go? I feel like they have been sleeping.

Yii has been around for a while and same with Kohana but both have less numbers than the above.

I have to revisit cake. I haven’t used it in a while but for a while it was seriously suffering on the documentation front.

If I was to pick a framework to run with on a new project I would probably pick Symfony 2 because of it’s forward thinking architecture. It can seem overly complex so be warned. If you haven’t already heard of things like dependency injection or service containers you may feel a little lost. The good thing is that there is a bunch of documentation to get you up to speed. Also it makes use of useful things like ESI and… HTTP… who knew!

"

My name is Jesse and I am an addict

2011-09-09T07:03:47-07:00

I have a confession to make…

I am addicted to information.

I can’t stop reading, learning, processing, consuming information.

If I could I would research day in day out with little time for else. Wait! I already do!

How does one deal with this?

I have been to conferences where the speakers have talked about trying to stay ahead of the wave by becoming very knowledgeable in only a few areas but maintaining a cursory understanding of a much much larger array of topics.

But I am an addict. And as other nerds know it’s very hard to tame the monkey. I would say I spend about 70% of my time working/playing/learning new technologies. Sounds like a dream?… welcome to my nightmare.

Currently I have 45000 unopened emails. Sure many of them are spam but I do have spam filters on my inbox. The others… Hundreds of mailing lists that I have signed up over the ~10 year tenure of my current email address.

I also have over 10000 bookmarks that I have accumulated over the years with every intention of returning at some point. If I even get to a small percentage of them I will be happy.

Then there is the “clipped/read later/evernote” bunch that are saved and waiting for my eyes to give them the time of day. But they are also too populous to be manageable.

Finally I have a library of pdf’s and ebooks measuring gigabytes in size (that’s text only people!) waiting to be read. By the time I read them the technologies they describe are quite likely to be outdated.

So I ask you. What am I to do? How can I slow down this landslide of information? How do I decide where to spend my time? Right now I am pulled between programming microcontrollers to work with android devices and working on massively scalable distributed multiversion control concurrent databases. Or do I spend my waking hours on further tuning linux kernels? Finishing my iOS project? Building a public cloud agnostic SAAS for erlang/otp applications? AHHH! Down the rabbit whole I go…

Have to have mac software

2011-09-08T05:07:32-07:00

I am in the process of setting up a new machine so I figured it would be a great time to compile a list of the software I have come to rely upon in my day to day life. I would try to keep it as cross platform as possible but fortunately/unfortunately I have been a mac head for a few years now and although I am most comfortable in front of a command line I have to use a gui from time to time.

Let’s start with the root of it all: Zsh Oh-My-Zsh: https://github.com/robbyrussell/oh-my-zsh

Also I am really liking this guy’s dotfiles: https://github.com/jelera/dotfiles

Next a good terminal Iterm2: http://www.iterm2.com/#/section/home

Now a good editor VIM/Macvim: http://code.google.com/p/macvim/

Checkout my MINIMAL list of VIM plugins and vimrc.

Ruby/Rack

http://pow.cx/

Turns out that pow starts a node server and reset’s the mac firewall to point ALL http traffic at it. This will cause you MANY “WTF”s as you try to find out why you can’t run apps in any other http application stack. Sorry pow you are out. Standar RVM install and then macruby or your favorite ruby blend on top.

PHP/Mysql Why not use MAMP and save yourself some time: http://www.mamp.info

Package Management I was an avid macports user for a long time but now I am strictly Homebrew: http://mxcl.github.com/homebrew/

Now install your favorite git/svn/hg/postgres/mongodb/python/perl versions!

Don’t forget wget/curl/tmux/screen

A way to browse Chrome: http://www.google.com/chrome

A way to browse with only the keyboard Vimium: http://vimium.github.com/

A way to communicate: Adium: http://adium.im/

The new iChat in lion is pretty nice too!

Arrange windows intelligently: Optimal Layout: http://most-advantageous.com/optimal-layout/

Share files DropBox: http://www.dropbox.com

Backup files: Jungle Disk: http://www.jungledisk.com Also looking forward to the new iCloud.

Keep Notes http://www.evernote.com

Mail/Calendar/Documents GOOGLE! I know it’s crazy to trust them with everything but it’s so friggin easy! I download and backup all of my content regularly on s3 and locally.

App Launcher For the most part I use spotlight but I have been using Alfred: http://www.alfredapp.com lately as well.

Virtualization I have tried other hypervisors but keep coming back to VMware fusion: http://www.vmware.com/mac

IDE I am trying not to use these big heavy guys these days but if I need to step through a debugger I have to admit it’s much more fun to use a gui than it is to use the cli of (insert first character of language here)db.

One Eclipse for javascript/node.js: Latest with v8 debugger: https://github.com/joyent/node/wiki/Using-Eclipse-as-Node-Applications-Debugger

Another Eclipse for Android/java: Latest with the android sdk: http://developer.android.com/sdk/eclipse-adt.html

Netbeans 6.9.x with the Ruby/Rails support is nice: http://netbeans.org/features/ruby/index.html

Netbeans 7.x with the PHP/Symfony support is also nice: http://netbeans.org/features/php/

Of course. Textmate. But I have to say this software is getting more and more outdated. Can I just re-iterate for the millionth time Where the hell is textmate 2? http://macromates.com/

There will certainly be more to come. This is just what I could think of in an afternoon.

Vim plugins I am using

2011-09-07T15:48:43-07:00

I wanted to keep a running list of vim plugins that I find useful. I don’t use many because I use a lot of different machines and find that portability almost always outweighs customization. As I come to add more to my repertoire I will edit this post.

Of course NerdTree https://github.com/scrooloose/nerdtree

Here is a plugin that allows you to kill your current buffer without loosing your window (splits) Buffkill: http://www.vim.org/scripts/script.php?script_id=1147

Color Sampler pack (checkout twilight): http://www.vim.org/scripts/script.php?script_id=625

Syntastic syntax checking (haven’t used this much but have heard good things): https://github.com/scrooloose/syntastic

TopFunky’s Peeopen (macvim only): http://peepcode.com/products/peepopen

Breaking into a ubuntu box with access to chef

2011-08-30T08:03:42-07:00

You would think this would be easier than it was. The Chef daemon that Rightscale runs on the ec2 instances it manages runs as root. However I had a few requirements. A) I had to do it all from my galaxy tab. B) I had to use only the Rightscale admin pannel. C) I could only write my commands in bash (I think this is all the rightscripts support) D) The bash commands I wrote were all executed by chef in a forked process so normal nix piping and such wouldn’t work. E) I didn’t want to open up the root user for direct login.

SO to start I had to create a new user, give him a home directory and add him to the admin group.

Then I had to put the galaxy tab’s public key into the user’s authorized_keys file.

Then I had to allow for passwordless sudo for that user.

The last item on the list was simply because I could not find a way of having chef in it’s execution of my bash scripts to pipe a password into passwd. Passwd even when run as root expects a password to be entered in on the tty. So of course I attempted all kinds of echo "password" | passwd jesse and

REXML could not parse this XML/HTML: 
<code>passwd jesse

Also creating the authorized_keys file proved very tricky. Since it didn’t already exist I had to create a new file from scratch. I started by simply touching the file touch /home/jesse/.ssh/authorized_keys and then trying to echo "pubkeymaterial" >> /home/jesse/.ssh/authorized_keys but again because of the piping issue explained above nothing would work. I kept ending up with an empty file.

Sed to the rescue! I broke out sed and started by simply trying to add a line to the file: sed -i -e "1a\\ pubkeymaterial" /home/jesse/.ssh/authorized_keys

Again blank file!

So then I tried to replace the empty line…

sed -i -e "s/^$/pubkeymaterial" /home/jesse/.ssh/authorized_keys

Again blank file!

Turns out sed needs SOMETHING in the file before you can insert add or replace text within it.

So I had to figure out how to get some data into the authorized keys file without an interactive console (no vi or emacs sorry!) AND without pipes (no echo redirection sorry!)

DD to the rescue! So how do you get some data into a file? Read it from somewhere! Where? well there are of course a few std places you could “mv/cp” a file from (I had thought about moving the .bash_history into place and then deleting everything but the first line. I assume that would have worked fine but would it be TRULY portable?)

Instead I decide to just put a zero into place.

dd if=/dev/zero of=/home/jesse/.ssh/authorized_keys bs=1 count=1

Now sed would work!

sed -i -e "s/^.*/pubkeymaterial" /home/jesse/.ssh/authorized_keys

Back to my terminal on the galaxy tab I was now able to login as jesse interactively. BUT because my password had never been set I couldn’t actually sudo.

sudo ls ~/

SO I need sudo to allow my Jesse user to sudo without prompting me for a password.

Back to sed.

sh -c “/bin/sed -i -e ‘5a\ Defaults:jesse !authenticate’ /etc/sudoers”

Wallah! now back in the gallaxy tab’s terminal I could sudo ls ~/ to my hearts content.

I finished up interactively in the gallaxy tab terminal. setting a password for myself.

sudo passwd jesse

Also it should be mentioned that I created my user within the admin group from the start using the following.

REXML could not parse this XML/HTML: 
<code>adduser --ingroup admin --home /home/jesse/ --shell /bin/bash jesse<code>

And of course if you caught it I did have to create the .ssh directory and own it.

<code>mkdir /home/jesse/.ssh/</code>

<code>chown -R /home/jesse/.ssh</code>

All in all it took a few hours. A hell of a lot longer than it would have if I had simply cheated and added my key directory the root user and then allowed root login but that would have been against the ubuntu way and a heck of lot less interesting :)

Brain Dump: VIM commands

2011-08-25T14:48:26-07:00

This is the first of a many quick brain dumps I will be doing of helpful keyboard-commands for things like bash, vim, OS X, etc. Check back for more goodies and feel free to comment with your own!

REXML could not parse this XML/HTML: 
<pre>Miscelaneous:
:set number (:set nu) shows line numbers.
:set nonumber (:set nonu) removes line numbers.

:sh opens shell in current directory
:! [command] executes command and returns output.

Copy and paste:
yy copy 1 line.
5y copy 5 lines.

dd cut 1 line.
5dd cut 5 lines.

p Paste under current line.

m[char] mark spot [char]. *(char can be any alpha-numeric character)
`[char] go to mark [char]. *(alpha-numeric character used as a mark)

y`[char] yank from current cursor to mark [char]

. repeat last command.

Moving:
:n go to line n
gg move to top of file.
G move to end of file.
w move one word forward.
b move one word backward.
} move one paragraph forward.
{ move one paragraph backward.

Line shifting/indenting:
>> shifts a line one shiftwidth to the right.

Bash Background Exec aka daemonize your executable

2011-07-26T13:08:38-07:00

I was having trouble getting a node process to work correctly with start-stop-daemon’s –background switch and also allowing for me to pipe stderr and stdout wherever I deemed necessary.

Turns out that the –backround switch expects the process being daemonized to take care of it’s own logging and does not allow access to the stderr and stdout file descriptors as you would normally expect.

SO I bring you Bash Background Exec

<SNIP> Waiting for license review SORRY! I will post a very detailed english explination of the solution shortly!</snip>

With it you can daemonize whatever executable you want and pass it parameters like you normally would while also piping stderr and stdout someplace nice. Here is an example of backgrounding a node process.

$ ./bbexec.sh -d /usr/local/bin/node -p /var/run/node -l \ 
> /var/log/node -a "app.js --debug-brk"

Remember the above is most useful when being called from an init.d script utilizing start-stop-daemon. I will be adding an example init.d script for node shortly.

Check back soon!

P.S. I am fully aware that this does not actually Daemonize in the strict sense. In particular it violates (on purpose) the following rule of Daemons:

"Closing all inherited open files at the time of execution that are left open by the parent process, including file descriptors 0, 1 and 2 (stdin, stdout, stderr). Required files will be opened later." -http://en.wikipedia.org/wiki/Daemon_(computing)

A quick rant about nosql and "webscale" grrr...

2011-07-11T06:04:28-07:00

I wrote a comment on a post over at 10gen’s mongo blog today. To their credit they are probably the most transparent corp. backed nosql product out there. So often I find them saying… well “we aren’t trying to solve that problem” or “our product is not good for this situation” (paraphrased obviously). I praise them for such honesty and frankly I love it when engineers keep it real. A warning to anyone trying to solve the “webscale” problem. You can fool the business types but the engineers are going to get to the bottom of your BS! Anyway here is the contents of my post: (link)

“10gen has always done a great job of trumpeting the benefits of mongodb while being completely transparent about it’s drawbacks.Â I think that is very helpful.Â I wish I could say the same for all of the non-10gen staff that have drank the mongo kool-aid and trumpet it as THE solution for “webscale” (I hate that acronym). Â I can say for sure that mongodb’s document storage and rich query language goes a long way for simple schemas. It is so important to break through to everyone that this is not a solution for complex schemas and by complex I mean anything that has a datamodel that cannot have every single field living inside the same document. It would almost be better to have blog posts about the sort of problems mongo is NOT intended to solve. Imagine trying to maintain a consistent view of a product library where each product has hundreds or thousands of individual fields. Then imagine each of the products in the library having a quantity field. Then imagine having a total quantity for all products within the library. Then imagine users checking in and out products and keeping the exact quantity of not only each product but the total of all products currently checked in consistent. $inc and similar atomic operators within mongo will work for the individual product totals but not the total over all products. Thus the need for acid in certain problem sets. As with all things you can get 2 but not 3. Fast/large(r than memory dataset)/consistent, the persistence daemon you choose will dictate which two you can have.”

Oh and quickly a little rant about “webscale” and why I hate that yet-another-unoriginal-sales-inspired-technical-buzzword. As with most of these “buzzwords” it means nothing and is not new at all. Since the dawn of computing there have been physical limitations. Limitations on physical memory being one of the more prevalent and permanent. “Webscale” I thinkÂ intends to describe the ever growing dataset of the web and the difficulty of working with it. This “web” dataset has seen a huge spike recently and as such solutions to how to process the information have changed… SLIGHTLY. I say slightly because processing data has been going on since the dawn of man and the solutions to processing this data have been permuted on over and over again. Sure the software was different then but the ideas behind the software are age old. Yes there are breakthroughs in how to look at a problem from time to time (ahem… map reduce) but for the most part the solution has always been to keep the data as close to the processor (in memory dbs) as possible and keep the volume of changes to that data low or suffer consistency issues (nosql). Webscale has always been around only with a different number of INsignificant digits following the numbers. However the relationship of dataset size to physical memory (available to a single machine) is still the same old problem as it’s always been.

Learnings on javascript scope chains, object prototype hierarchies, javascript "inheritance" and "constructors"

2011-07-06T06:44:53-07:00

I had to debug some very funny stuff going on in some of my models yesterday/this morning. It lead to some learnings about the internals of javascript inheritance and scope. Please excuse my run-ons and bad grammar!

First one of the interesting things about javascript is that variables can be declared locally inside a function and that function can be turned into an object using the “new” keyword but those local variables are not available to the object that the new keyword creates out of the function!

Jesse = function() { var name = jesse };jesse = new Jesse();typeof

However if you use the this object within the context of the “constructor” it does work. The reason being that the “new” keyword creates a new object then inserts that object into the body of the “constructor” function’s “scope chain” with the property name “this” that way any references to “this” within the constructor function reference the “new object” and not the this of any scope surrounding the constructor function or the call to the “new” keyword.

So if you were to think of the scope chain as an array like data structure where “global” akaÂ ”window”Â scope is normally the last element in the array aka = arrayarray.length

AND if you were to imagine the behavior of the javascript interpreter trying to de-reference a property as a walk through the array from element 0 to element array.length looking at every property on every object in each “cell” of the array and checking if their property names match the one in question.

Then the new keyword takes a blank object assigns the property name this to it and sticks it at the front of the aforementioned scope chain array and then evaluates the contents of the constructor using it.

Functions like call/apply/bind and the keyword “with” can be used to manipulate the scope chain programatically. However call/apply/bind can only manipulate the “this” property name and the “with” keyword comes with many caveats and performance hits.

In my opinion call/apply/bind are probably all shortcuts to simply creating a closure around a function with a pointer to the current “this”.

Here is an example:

var Jesse = function(){};var jesse = new Jesse();Â //Look ma we now have an object with a reference to itself called "this"jesse.firstName = "JESSE";Â //aka jesse.lastName = "SANFORD";jesse.getFirstName = function() { return this.firstName; }; //yup you guessed it "this" is pointing at jessejesse.getLastName = function() { return this.lastName; };//now here comes the fun!var tom = { name: "TOM" }; //look ma another object! except this is an "object literal" who cares you say?... I agree I don't care... yet... see below.jesse.getFirstName.call(tom); //yup this will return "TOM"jesse.getFirstName.bind(tom)(); //same as before! //note the extra parens at the end are simply calling the function that is returned by the bind object which is simply the jesse.getFirstName function with the scope chain modified so that the reference to "this" is replaced with tom

Note that all of the above examples are very trivial and they are in the imperative style so they may not seem that handy. But they can also be used when doing asynchronous callback passing functional style

//lets create a new function called printName which takes in two callback getter functions and print's their return values;jesse.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };//now you might expect the following to work:jesse.printName(jesse.getFirstName, jesse.getLastName); //undefined undefined//but it doesn't because now "this" points to the GLOBAL scope!Â //check it out:firstName = "global"; //note the lack of the use of var (pushing the variable into global scope)Jesse = function(){};Jesse.firstName = "parentobj";Jesse.prototype.firstName = "prototype";jesse = new Jesse();jesse.firstName = "instance";jesse.print(cb) { console.log(cb()) }; //just print whatever the callback returnsjesse.getFirstName = function() { return this.firstName }; //supposed to return the instance's foo variablejesse.print(jesse.getFirstName); // 'global' !!! WHAT? :)

This may not be news to you but scope and functional programming can get wonky! So this is where the call and bind functions from above come in handy!

If we were to call the jesse.printName(fcb, lcb) function above with them we would get what we expected rather than “undefined undefined”

jesse.printName(jesse.getFirstName.bind(jesse), jesse.getLastName.bind(jesse)); //"JESSE SANFORD"

OR if we re-wrote the printName function we could do the same with call (daisy chaining the passing of “this”)

jesse.printName =Â Â function(fcb,lcb) { console.log(fcb.call(this)+" "+lcb.call(this)) };jesse.printName.call(jesse, jesse.getFirstName, jesse.getLastName);Â //"JESSE SANFORD"

But bind/call/apply may not look as clean as you might like. Also manipulating the scope chain just feels a little wrong to me. SO in some situations it makes more sense to use closures!

let’s re-define our getFirstName and getLastName functions to remove the “this” from them. Instead let’s have them reference a synonym called “self”

jesse.getFirstName = function() { return self.firstName; };Â jesse.getLastName = function() { return self.lastName; };

but what is self you say?Â it’s whatever you make it!

check this out…

var self = jesse;jesse.printName(jesse.getFirstName,jesse.getLastName); //"JESSE SANFORD" !!

Yes because now the interpreter when running through the blocks for getFirstName and getLastName will search the scope chain for a variable called “self” and the first one it finds will be used within them.

Again this example is very trivial and may not seem very useful. But think about it in reference to objects and their constructors. Remember a constructor when called ALWAYS has “this” pointed at it’s new blank object instance. SO…

Let’s re-factor the Jesse parent object like so:

//We can also use the inheritance to define the Jesse parent obj more robustly for re-use… Hell let’s just rename it to Person as you might have expected.

var Person = function() {Â var self = this;Â //now self will point at WHATEVER new instance object is constructed by the "new" keyword when this "constructor" is called.Â Â self.getFirstName = function() { return self.firstName; };Â Â Â self.getLastName = function() { return self.lastName; };Â Â self.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };};//now we can simply instantiate a new Person as jesse, name it and print the name!jesse = new Person();jesse.firstName = "JESSE";jesse.lastName = "SANFORD";jesse.printName(jesse.getFirstName, jesse.getLastName); //"JESSE SANFORD"!!!

NOTE: You might expect the following to work: (If you are confused as to why not… see the first example all the way at the top!)

var Person = function() {Â Â var self = this;}Person.prototype.getFirstName = function() { return self.firstName; };Â Person.prototype.getLastName = function() { return self.lastName; };Person.prototype.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };jesse = new Person();

jesse.firstName = “JESSE”;

jesse.lastName = “SANFORD”;

jesse.printName(jesse.getFirstName, jesse.getLastName); //ReferenceError: self is not defined (Because self is only available to the instance… NOT to the prototype which is where getFirstName and getLastName are defined)

THIS ALSO WILL NOT WORKÂ ( again If you are confused as to why not… see the first example all the way at the top!)

var Person = function() {Â Â Â var self = this;Â };//if we construct a new jessejesse = new Person();jesse.firstName = "JESSE";jesse.lastName = "SANFORD";//and define the following as above:jesse.getFirstName = function() { return self.firstName; };Â jesse.getLastName = function() { return self.lastName; };jesse.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };//we get as expected:jesse.printName(jesse.getFirstName, jesse.getLastName); //ReferenceError:Â self is not defined (as above)

A quick (albeit dangerous) fix to the above examples would be to use the “with” keyword and a local variable called self.

var whoami = {self: jesse};with(whoami) {Â Â jesse.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };Â //"JESSE SANFORD"!!!}

However “with” is dangerous and slow and almost everyone regards it as something you should never use. The problem is what happens if you don’t define “whoami” correctly RIGHT before you call “with” keyword. Will you remember WHERE you defined it? Will you have even been the one who defined it? “with” will cause the interpreter to walk the scope chain looking for properties on variables that match the variable names referenced within the block following it’s use. This can cause variables from all different scopes to get used and as such very unpredictable output! It get’s even worse if you redefine variables within the with block. You may try to set a value of a property within “whoami” and very well be creating new GLOBAL variables if you are not careful to make sure all variable names used are already contained within “whoami” as properties. See:Â http://stackoverflow.com/questions/61552/are-there-legitimate-uses-for-javascripts-with-statement/61676#61676

A safer alternative is in the works for javascript 1.7 using the “let” keyword

let(self = jesse) {Â Â jesse.printName = function(fcb,lcb) { console.log(fcb()+" "+lcb()) };Â //"JESSE SANFORD"!!!}

Of course there are not many interpreters for 1.7 yet though. (The above does not work in the current version of node.)

A couple of other interesting notes creating objects/constructors and on prototype chains and when you can access them. (see note above about object literals)

Basically if you create an object literal… you will never get access to it’s prototype chain UNLESS you modify the “Object” object’s prototype. This is the super-prototype in javascript. The only thing higher up in the order than Object is Object.prototype and then there is null. SO don’t expect this to work:

var tom = { firstName: "tom", lastName:"pytleski" };tom.prototype.getLastName = function() { return this.lastName; }; // returnsÂ TypeError: Cannot set property 'getLastName' of undefined (because tom has no prototype! it's an object literal... yeah now we care :)//However we can do this:Object.prototype.getFirstName = function() { return this.firstName; };tom.firstName = "tom";Â tom.getFirstName(); //'tom'

But that is BAD! because you just added a getFirstName function to every object in your virtual machine.

var one = new Number();one.firstName = "two"; // funny huh :)one.getFirstName(); //'two' // great now numbers have a getFirstName function!

So the correct way is definitely to create a constructor first. Even a blank constructor works:

var Tom = function(){};Â tom = new Tom();tom.firstName = "tom"; //'tom'

BUT remember that the instance does NOT have a prototype chain. Trying the following will not work:

tom.prototype.getFirstName = function() { return this.firstName; }; //TypeError: Cannot set property 'geFirstName' of undefined (because instances of objects don't have prototypes!)

However as you have seen above this does work:

Tom.prototype.getFirstName = function() { return this.firstName; };//and now:tom.getFirstName(); // 'tom' //yes as expected.

Finally it may help to setup instance variables within the constructor right off the bat.

var Tom = function() { this.firstName = "jesse"; this.lastName = "sanford"; };Tom.prototype.getFirstName = function() { return this.firstName };tom = new Tom();tom.firstName = "tom"; //you can set their values just as you might expecttom.getFirstName(); //'tom' //yup!

A note on inheritance… Everyone has probably figured this out already but I am repeating it here at the end if it’s still not quite clear. The prototype heirarchy provides the single parent inheritance you have seen.

Person = function(){};jesse = new Person();tom = new Person();Person.prototype.getFirstName = function() { return this.firstName; };jesse.firstName = "jesse";jesse.getFirstName(); //'jesse'tom.getFirstName(); //'jesse' yup!tom.lastName = "pytleski";Person.prototype.getLastName = function() { return this.lastName; };Person.prototype.lastName = "sanford";tom.getLastName(); //'pytleski' yup! //Also note that the hierarchy is respected when the scope chain is inspected by the interpreter as expectedjesse.getLastName(); //'sanford'

Another quick tidbit about inheritance:

//Guess what jesse.foo will be :)Jesse = function(){};Jesse.foo = "parentobj";Jesse.prototype.foo = "prototype";jesse = new Jesse();jesse.foo; //? (scroll down)

//yeah it'sÂ 'prototype'!

Here were some of the good articles that lead to my solution:

http://www.academa.si/?content=http://www.academa.si/html/articles/js/professionalJavaScript/prototipesAndScopeChains.htm http://stackoverflow.com/questions/61552/are-there-legitimate-uses-for-javascripts-with-statement https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/call https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/function/apply

more interesting stuff:

https://developer.mozilla.org/en/JavaScript/Reference/Functions_and_function_scope/arguments http://unspecified.wordpress.com/2011/06/05/simulating-classes-with-prototypes-in-javascript/

Successful SAAS/PAAS. How to architect maintainable services without killing yourself.

2011-03-09T07:10:40-08:00

Even the most simple SAAS/PAAS ain’t easy. As soon as you have a single client you are on the hook to make sure your software works as expected 24/7 or suffer the tongue lashing. I recently launched a SAAS to support a multi-player game title on top of ~30 bare-metal servers. I have had a lot of experience building large scaleable web-apps, but this was the first time I had to support other developers directly. This series of blog posts will cover how I was able to architect a hardware/software stack that could be developed by a team of two, iterated on quickly and then maintained with minimal downtime. For all of you nerds out there…I will try to get into the nitty gritty technical details. More importantly though I hope to also focus on the high level principles which helped me make the right decisions in the right places and ultimately led to a calm, cool and collected launch. Here is a top level view of what I believe to be the most important parts of the software lifecycle of a modern platform/service.

1) Testing. (Unit, Functional, Regression and ESPECIALLY LOAD testing) 2) Tooling. (Language, Framework, Libraries, IDE, Best practices/Recipes, Documentation) 3) Architecture/Platform. (Decoupled, PROVEN logical architecture on top of redundant, highly available hardware) 4) Logging. (Unified logging across all of your app servers with tunable logging levels vs shared nothing logging with distributed shell access our best friend GREP) 5) Ops Team/Hosting SLA. (24/7 support for hardware/Low level software related issues, Backups, Disaster Recovery) 6) Metrics. (Realtime, Historical, Query-able metrics) 7) Client facing support staff.

Other things worth mentioning: Version control (Branching, Merging, developing features and maintaining releases in parallel), Behavior driven development (BDD), Agile/Iterative software development, Deployment management, Systems configuration management, Change control, Software/Service selection. The list goes on…

Looking forward to brain dumping all of this. I hope this helps make your project run easier.

How to increase your .bash_history size a.k.a. what to do when you suffer from short term command line syntax loss

2010-10-12T11:25:05-07:00

vi ~/.bashrc add HISTSIZE=1000 (default is 500 on most nix) then source ~/.bashrc echo $HISTSIZE To see the old size echo $HISTFILESIZE

If you really want to get tricky you can set your .bash_history size REALLY high and then use logrotate to make sure you don’t fill up your hd. See here: http://linux.derkeiler.com/Mailing-Lists/SuSE/2008-12/msg00864.html

Compiling percona xtradb on ubuntu 10.04

2010-08-02T05:35:15-07:00

I recently had to compile from source the Percona blend of mysql 5.4 and ran into a few hiccups that I don’t normally see when doing a vanilla mysql install. In particular I ran into the same issue described here and here. "/bin/bash ../ylwrap sql_yacc.yy y.tab.c sql_yacc.cc y.tab.h sql_yacc.h y.output sql_yacc.output -- -d --verbose ../ylwrap: line 111: -d: command not found make[1]: *** [sql_yacc.cc] Error 1" Apparently having nothing to do with percona but with mysql and yacc not finding bison and possibly flex. So I installed both of those via APT and ran a make clean, make, make install and still same error. Then I realized that the configure script is probably what sets the make switches for using bison so I re-ran CC="gcc -static-libgcc" CFLAGS="-O3 -pipe -m64 -fPIC -fomit-frame-pointer" CXX="gcc -static-libgcc" CXXFLAGS="-O3 -pipe -m64 -fPIC -fomit-frame-pointer -felide-constructors -fno-exceptions -fno-rtti" sudo ./configure -prefix=/opt/xtradb -enable-local-infile -enable-thread-safe-client -enable-assembler -with-client-ldflags=-all-static -with-mysqld-ldflags=-all-static -with-unix-socket-path=/opt/xtradb/tmp/mysql.sock -with-plugins=partition,archive,blackhole,csv,federated,heap,ibmdb2i,innobase,innodb_plugin,myisam,myisammrg -with-big-tables -without-debug -with-tcp-port=3306 -with-readline -enable-profiling -disable-shared -enable-static -with-extra-charsets=complex -with-pic -with-fast-mutexes -with-zlib-dir=bundled -with-ssl and then another make make install

And it worked successfully.

Hope this helps!

PS. If you want the full list of libraries that I needed to install in order to compile correctly: apt-get install gcc build-essential automake ncurses libncurses5-dev libtool termcap bison flex

Update: since I was using the stock ubuntu 10.04 server install apparmor was installed and keeping mysql from correctly installing it’s databases in a non-standard directory. I kept getting this error: /opt/xtradb/bin/mysql_install_db --user=mysql --ldata=/opt/xtradb/data Installing MySQL system tables... 100802 10:26:54 [Warning] Can't create test file /opt/xtradb/data/ubuntu.lower-test

So just had to run a apt-get purge apparmor restart and then run /opt/xtradb/bin/mysql_install_db --user=mysql --ldata=/opt/xtradb/data and finally I could start mysql

REXML could not parse this XML/HTML: 
<code>
/opt/xtradb/bin/mysqld_safe --defaults-file=/opt/xtradb/conf/my.cnf --user=mysql --basedir=/opt/xtradb --datadir=/opt/xtradb/data --log-error=/var/log/xtradb/mysql.err &
</code>

Undertstanding Apache HTTPD mod_fcgid variables: PHP via suexec fastcgi

2010-07-27T06:46:32-07:00

Had to talk through another php mod_fcgid setup today. It is amazing to me how difficult it is to find clear information on this setup out there on the internet.

If any of the below are wrong please tell me. I am not always right but I have studied this setup quite a bit and have come across many permutations and the below finally make sense.

Assumptions:

The most MEMORY efficient way to use php with apache is via the WORKER MPM with a fastcgi plugin forking PHP-CGI processes. I specify MEMORY here because I am pretty sure that CPU wise mod_php is still faster. I can only assume that is because all of the data structures etc that are used within php are within direct access of the apache processes servicing the request. Faster still is probably compiling mod_php directly into apache rather than using the shared module.

The most MEMORY efficient way of using PHP-CGI is to use as FEW parent processes as possible if not 0 and use as many child processes as possible under those parent processes. This is especially true when using an opcode cache like APC. There is a known bug with the mod_FCGID that causes it to be unable to share memory between FCGI processes. mod_FASTCGI on the otherhand does not have this bug. Why not use mod_FASTCGI module instead? Because unfortunately it is a dead project (nothing since 2003): http://freshmeat.net/projects/mod_fastcgi/

Thus APC is not very effective when using mod_fcgid. It just can't share its memory across subsequent requests to the same php code if those requests get routed to seperate php-cgi processes. See here:

"PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled.

The popular APC opcode cache for PHP cannot share a cache between PHP FastCGI processes unless PHP manages the child processes. Thus, the effectiveness of the cache is limited with mod_fcgid; concurrent PHP requests will use different opcode caches."

Anyway I used the following config on a relatively low traffic 4gb virtual instance with a VERY memory intensive PHP app.

In apache httpd.conf:

REXML could not parse this XML/HTML: 
<code>
<p>
&lt;IfModule mod_fcgid.c>
AddHandler    fcgid-script .fcgi
&lt;/IfModule>
</p>
<p>
## Sane place to put sockets and shared memory file
#FcgidIPCDir run/mod_fcgid #to use sockets on unix or pipes on windows
#FcgidProcessTableFile run/mod_fcgid/fcgid_shm #to use shared memory on unix
</p>
</p>

REXML could not parse this XML/HTML: 
<p>

FcgidInitialEnv PHPRC “/etc” # is also set in the wrapper script per virtual host. both ways will send the variables on to the cgi instances

the following MaxRequestsPerProcess is also set in the wrapper script. both ways will send the variables on to the cgi instances

REXML could not parse this XML/HTML: 
</p>

FcgidInitialEnv PHPRC=/etc/php5/cgi #working directory of php-cgi

#"By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem."

FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 5000 #same as below just as environment variable. sometimes good to set as well. due dilligence

FcgidMaxRequestsPerProcess 5000 #restarts child after so many requests to take care of memory leaks. needs to equal or be less than the PHP_FCGI_MAX_REQUEST variable sent to the cgi process

FcgidIdleTimeout 300 #allows process to sit for 5 mins while idle FcgidIdleScanInterval 120 #how often fcgid parent polls children who are idle FcgidBusyTimeout 3000 #allows php to spin for 50 mins before throwing 500 error FcgidBusyScanInterval 120 #how often fcgid parent polls children who are busy FcgidErrorScanInterval 60 #how often fcgid parent polls children for errors FcgidZombieScanInterval 60 #how often to scan for zombie processes FcgidProcessLifeTime 7200 #max time a child is alive by default FcgidMaxProcesses 15 #max children PER PARENT PROCESS (remember this is not max TOTAL) FcgidMaxProcessesPerClass 15 #Max fcgi processes FcgidMaxProcessesPerClass 15 #Min to leave lying around FcgidIPCConnectTimeout 3600 #Max time to wait for first "packet" on port or socket from child process FcgidIPCCommTimeout 3600 #Max time to wait since the last "packet" on port or socket from child process FcgidOutputBufferSize 128 #comm buffer between mod_fcgid and the cgi process. flushed to client after full.

</IfModule>

In vhost definition:

SuexecUserGroup some_unprivileged_user_with_rights_to_webroot some_unprivileged_group_with_rights_to_webroot #often these are www or apache user and group

FcgidFixPathinfo 1 #"This directive enables special SCRIPT_NAME processing which allows PHP to provide additional path information. The setting of FcgidFixPathinfoshould mirror the cgi.fix_pathinfo setting in php.ini."

<Directory "/var/www/html"> Options +ExecCGI AllowOverride All AddHandler fcgid-script .php

FcgidWrapper /var/www/php-fcgi-scripts/php-fcgi-starter .php #location of your suexeced bash script

Order allow,deny Allow from all </Directory>

</IfModule> </VirtualHost>

Inside the suexec starter script /var/www/php-fcgi-scripts/php-fcgi-starter

#!/bin/sh PHPRC=/etc/php5/cgi/ #again the php-cgi working directory export PHPRC #also set in the httpd.conf. I think this just stomps the one set in the httpd.conf.

export PHP_FCGI_MAX_REQUESTS=5000 #again setting the max number of requests per child process. Again also in httpd.conf so I think these are redundant.

#"PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled. The popular APC opcode cache for PHP cannot share a cache between PHP FastCGI processes unless PHP manages the child processes. Thus, the effectiveness of the cache is limited with mod_fcgid; concurrent PHP requests will use different opcode caches."

export PHP_FCGI_CHILDREN=0 #THIS IS WHERE YOU SET THE NUMBER OF PARENT PROCESSES These will each spawn 15 children in this example. If set to 0 then you will have 1 parent with 15 children and apache will handle spawning the children not php.

exec /usr/bin/php-cgi #the php-cgi binary

REXML could not parse this XML/HTML: 
</code>

References:

http://www.linode.com/forums/viewtopic.php?t=2982&postdays=0&postorder=asc&start=15 http://www.magentocommerce.com/boards/viewthread/29264/http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html

How multiple caching layers can confuse the hell out of everybody a.k.a. when will my story publish!?!

2010-07-22T09:27:02-07:00

When using a varnish/squid/nginx or any reverse proxy +cache / http accelerator on top of a modern web framework / web application that has it’s own caching layer you may run into some strange seemingly uncontrollable TTL’s on your content. I have come to rely heavily on varnish in the last year and squid before that. If your reverse proxy is setup defensively (and in my opinion it should be) then you probably tend to ignore somewhat the http headers coming out of the application layer and put an additional TTL on top of all of the (non-dynamic) content. A for instance: There is a certain editorial site that I maintain that I cache everything on the site in varnish for 30 mins… period. This means that once an editor pushes a publish button and the content is picked up into the cache it won’t change for a half an hour. OR WILL IT? Turns out that since the application layer also has a cache lifetime of something like 5 mins the object in varnish may or may not refresh for an hour or more. How? Well that’s where it gets difficult to explain. It all depends on when the change get’s published to the content with respect to the application servers cache expiry on the object AND varnish’s cache expiry on the object AND the time of the next request for the object after Varnish’s cache expiry has happened. HAH! Before we go any further a picture is needed.

As you can see from the graph there are two sawtooth looking lines representing the two types of cache's in use on this site. I have made the assumption that an editor publishes a change to an object (say a homepage of a site) every half hour. Really they may publish 25 times within that half hour but let's just keep it simple. So in the best case example an editor publishes a change. That publish event happens RIGHT BEFORE the cache in varnish AND the cache in the application server is invalidated for that object and the user request comes in RIGHT AFTER the cache is invalidated in both places so that the user request flows all the way upstream to the application server which re-renders the page with the new content, caches it and then sends it back up to varnish which also caches it and then sends it to the client. That is the BEST POSSIBLE case. That RARELY happens. What normally happens is that a user request comes in, hit's the varnish cache and is served a moderately old object. I say moderate because more than likely on average the object is somewhere near the middle of it's TTL. IF the varnish TTL has expired on an object what is next most likely is that the APP server is somewhere within the middle of the TTL for the object within IT'S cache and the app server will just serve the cached response and then varnish will do what it knows best and CACHE THE OBJECT FOR ANOTHER CACHE CYLE. That means ANOTHER 30 mins in this scenario. Hmmm editors are not to happy about this situation. Thing is. It get's worse. What CAN happen is that editor publishes a change immediately AFTER the app server rolls over on it's cache for the object, re-caches the old content and never sees the editors change. The re-cached old content in the appserver is then served again to varnish who could (as within the last example) roll over on it's cache RIGHT at the end of the app servers cache cycle for the object BUT BEFORE the appserver's cache actually invalidates. As explained in the last example the appserver serves varnish the old object, varnish re-caches the old object and then that old object is within the varnish cache for the extra cache cycle. SO in the worst case the editors story doesn't get picked up for what is essentially the sum of TWO FULL CACHE CYCLES of both cache's TTL's.

All of this is VERY hard to explain to laypeople. Hence the picture above with the perty colorful stars.

I hope this helps someone out there. I know I will use it over and over again..

P.S. I do realize you can always just get rid of one of the application servers cache. Varnish is performing the same exact function so why bother. Fact is I simplified this example to html buffer caching in both layers so that you it would be easier to understand. What Is really going on is that the application layer is not caching the html buffer. It is actually caching the results of the queries from the database in memcache for a period of time. In addition there are "partials" strewn through out the app that are cached independently of context within the app which gives finer grained control over areas that change less frequently than varnish allows for. BTW I do realize varnish supports ESI allowing certain of these partials to be re-calculated at the app server rather than at varnish even when varnish serves some of the page from it's cache. I hope to utilize ESI in the future but that will require an app re-write. ALSO I would love to have the application server send the cache invalidation requests upstream to varnish when it's own cache cycles roll over on objects. Unfortunately the off the shelf app we are using is not that smart and again it would require a re-write to get that in there.

On EIP, Camel, Spring and Object Recognition!?

2010-04-15T05:34:29-07:00

I have recently been doing some work with Camel routing library/engine for java that plays really nice with Spring and JMS via ActiveMQ. Camel makes it really easy to implement EIP (Enterprise Integration Patterns) into your Java app and ultimately allows you the beautiful flexibility describe in Gregor Hohpe’s work on enterprise integration patterns… great book btw!

Anyway I got to thinking how fun it would be to create a custom predicate that employs a machine learning algorithm (possibly based on a neural network) to do image recognition and route messages (images) based on it’s understanding of them. Given enough time and enough input this could really open up some awesome possibilities!

The most immediately applicable solution is moderation, allowing the AI to do the first pass, pipe positives to an administrative interface where an actual human being could determine if the image was actually naughty or if the algorithm flagged a false positive. The feedback on false positives could then be driven back into the pipeline thus tuning the algorithm further (false positives) or deleting the image from the servers and warning the contributing user (actual positives).

Another cool application would be to do something like google is doing where you could use the routing engine and the image recognition predicate to make a first guess at what the image is basically of and then query all sorts of different data stores/apis to find more information about the image. You wouldn’t need to do so much legwork query everything or storing a “master index” you could simply allow the algorithm to make an educated guess and say for example “this is a painting” or “this is a barcode” and then ONLY query the applicable datasources.

Of course this will be full of false positives. BUT if you make it fun and allow feedback back into the routing engine from the end users the algorithm will get smarter over time!

Installing xdebug on centos 5

2010-02-04T17:36:59-08:00

Performance profiling php in my opinion is best done with xdebug and kcachegrind (install it via APT on an Ubuntu virtual machine and save yourself some trouble. I know you can get it to work with xwindows natively on a mac but damn it’s not easy and it certainly isn’t as pretty as using it in KDE 4 )

How toÂ performanceÂ profile an app is an art not covered here. Seriously if you want to start somewhere I suggest reading this article over at Zend developer zone (It’s from 2007 but it’s still relevant):

http://devzone.zend.com/article/2899-Profiling-PHP-Applications-With-xdebug

Also there is a great article if you can find it from the September 2004 edition of php|Architect… If you can’t find it on the internet email me and I will send you a PDF version of it. It is a complete and comprehensive article and should not be missed if you are working with xdebug at all.

So finally back on topic:

Installing xdebug is pretty easy assuming you have php/pecl/pear installed… If you don’t I suggest you use yum and if you don’t have that installed then just follow this tutorial: http://www.matteomattei.com/en/install-yum-and-php-pear-on-centos-5

Anyway once you have pecl installed you can run the following:

REXML could not parse this XML/HTML: 
<blockquote># pear install pecl/xdebug

downloading xdebug-2.0.5.tgz … Starting to download xdebug-2.0.5.tgz (289,234 bytes) ……………………………………………………done: 289,234 bytes 67 source files, building running: phpize Configuring for: PHP Api Version: Â Â Â Â 20041225 Zend Module Api No: Â Â Â 20050922 Zend Extension Api No: Â 220051025 /usr/bin/phpize: /tmp/tmpOcEvNL/xdebug-2.0.5/build/shtool: /bin/sh: bad interpreter: Permission denied Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF environment variable is set correctly and then rerun this script.</blockquote> Woops!

Looks like something is up with running /tmp/tmpOcEvNL/xdebug-2.0.5/build/shtool

I have seen this type of build error before when using other package management systems and it turns out that what I thought was true. That the /tmp directory was mounted with the noexec switch.

See here:

# mount -l | grep /tmp simfs on /tmp type simfs (rw,noexec) simfs on /var/tmp type simfs (rw,noexec)

# pear install pecl/xdebug

downloading xdebug-2.0.5.tgz ...

Starting to download xdebug-2.0.5.tgz (289,234 bytes)

............................................................done: 289,234 bytes

67 source files, building

running: phpize

Configuring for:

PHP Api Version: Â Â Â Â 20041225

Zend Module Api No: Â Â Â 20050922

Zend Extension Api No: Â 220051025

/usr/bin/phpize: /tmp/tmpOcEvNL/xdebug-2.0.5/build/shtool: /bin/sh: bad interpreter: Permission denied

Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF

environment variable is set correctly and then rerun this script.

So let’s remount it with the exec switch:

mount -o remount,exec /tmp

And now let’s see what mount -l says:

mount -l | grep /tmp simfs on /tmp type simfs (rw) simfs on /var/tmp type simfs (rw,noexec)

So we can see that now the /tmp does not have noexec listed.

Ok let’s try and install via pecl again:

# pear install pecl/xdebug downloading xdebug-2.0.5.tgz ... Starting to download xdebug-2.0.5.tgz (289,234 bytes) ............................................................done: 289,234 bytes 67 source files, building running: phpize Configuring for: PHP Api Version: Â Â Â Â 20041225 Zend Module Api No: Â Â Â 20050922 Zend Extension Api No: Â 220051025 building in /var/tmp/pear-build-root/xdebug-2.0.5 running: /tmp/tmp3tHVR2/xdebug-2.0.5/configure checking for egrep... grep -E checking for a sed that does not truncate output... /bin/sed checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... configure: error: cannot run C compiled programs. If you meant to cross compile, use `--host'. See `config.log' for more details. ERROR: `/tmp/tmp3tHVR2/xdebug-2.0.5/configure' failed

Well were not in the clear yet the error looks surprisingly similar… let’s just try to mount /var/tmp without the noexec switch as well:

# mount -o remount,exec /var/tmp

And again the pecl install:

REXML could not parse this XML/HTML: 
<blockquote># pear install pecl/xdebug
downloading xdebug-2.0.5.tgz ...
Starting to download xdebug-2.0.5.tgz (289,234 bytes)
..................................................done: 289,234 bytes
67 source files, building
running: phpize
Configuring for:
PHP Api Version: Â  Â  Â  Â  20041225
Zend Module Api No: Â  Â  Â 20050922
Zend Extension Api No: Â  220051025
building in /var/tmp/pear-build-root/xdebug-2.0.5
running: /tmp/tmpkWolpH/xdebug-2.0.5/configure
checking for egrep... grep -E
checking for a sed that does not truncate output... /bin/sed
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...

<SNIP>

Build process completed successfully Installing ‘/var/tmp/pear-build-root/install-xdebug-2.0.5//usr/lib64/php/modules/xdebug.so’ install ok: channel://pecl.php.net/xdebug-2.0.5 You should add “extension=xdebug.so” to php.ini</blockquote>

Nice. Ok so now it worked. Notice the path that it was installed to (on this 64 bit machine it was):Â /usr/lib64/php/modules/xdebug.so

We will need that later when we put it in our new /etc/php.d/xdebug.ini: (the contents of which I will explain in another blog post but for now the comments inside of it should be self explanatory)

REXML could not parse this XML/HTML: 
<blockquote>#extension_dir = "/usr/local/lib/php/20060613"

doc_root=”/usr/local/apache2/htdocs/web”

the following was added by jesse to fix the path variables when run in cgi mode

cgi.fix_pathinfo=0

zend_extension=”/usr/local/lib/php/20060613/xdebug.so”

xdebug.remote_enable=1

;JESSE: the following was taken from: http://code.google.com/p/syslogr-utils/wiki/XdebugHelper ;When using Eclipse with PDT and xdebug – make sure to change your ;tools –> addons –> xdebug helper –> preferences –> idekey = ECLIPSE_DBGP ;(This is the setting that XDEBUG_SESSION_START is set to on your web browser URL)

;JESSE: I have put the following notes in from some research into profiling with webgrind: ;(they were taken from: http://www.chrisabernethy.com/php-profiling-xdebug-webgrind/ )

;NOTE: usually use cachegrind.out.%t.%p when I want one output file per script run, ;but if I want to run a script multiple times and see the aggregate numbers ;I use cachegrind.out.%s and set xdebug.profiler_append = on

;NOTE: A Nice to know, for different cachegrind files per host you can use %H in the ;profiler_output_name, according to ;http://www.xdebug.org/docs/all_settings#trace_output_name

;Always profile scripts with xdebug: ;xdebug.profiler_enable = 1

xdebug.profiler_enable=1

;Alternatively, enable profiling with GET/POST parameter XDEBUG_PROFILE, ;e.g. http://localhost/samplepage.php?XDEBUG_PROFILE: ;xdebug.profiler_enable_trigger = 1

xdebug.profiler_enable_trigger=1

xdebug.profiler_output_dir=”/tmp/xdebug/”

;the following forces xdebug to append to the outfile rather than ;overwrite it on each exec of a script ;xdebug.profiler_append=On

;the patterns for the names of the outfiles ;xdebug.profiler_output_name = cachegrind.out.%s xdebug.profiler_output_name = cachegrind.out.%t.%p

;the following opens debug output links in textmate xdebug.file_link_format = “txmt://open?url=file://%f&line=%l”

xdebug.remote_host=”127.0.0.1” xdebug.remote_port=9000 xdebug.remote_handler=”dbgp” xdebug.remote_mode=req xdebug.idekey=1</blockquote> Now check the php-cli and see if xdebug shows up in it’s ini dump.

php -i | grep xdebug /etc/php.d/xdebug.ini, xdebug xdebug support => enabled xdebug.auto_trace => Off => Off xdebug.collect_includes => On => On xdebug.collect_params => 0 => 0 xdebug.collect_return => Off => Off xdebug.collect_vars => Off => Off xdebug.default_enable => On => On xdebug.dump.COOKIE => no value => no value
<snip>

Looking good!

Oh and obviously restart apache… /etc/init.d/httpd restart

Sweet.

Why Drupal polls don't play well with full page caching or how a 3-tier architecture works.

2009-12-17T08:56:25-08:00

I had a client who didn’t understand why caching was breaking things on their site. Of particular nuisance was the polls. To be fair, grasping why something as easy as polls works without caching and then does not work with caching can be confusing. Here is the email I wrote to help educate all of us. Below you will see the contents of my email to them.

<snip>

To begin I think it is necessary to explain why things are cached and why caching has an impact on editorial workflow and functionality. Specifically this is in reference to polls. It’s going to take a bit to grasp what is happening so please read slow and bare with me.

Full web page caching or HTTP acceleration is done in part to effectively reduce the load experienced by an application and also to mask performance bottlenecks in the application to users.

Without caching, every time a user makes a request, that request eventually has some action that is performed by business logic in the application and ultimately the database. That action can actually be one or THOUSANDS of connections and queries. Those thousands of queries per user request are normally NOT unique as in one request will normally cause the same exact connections and queries to be performed as the next request for the same content. So as the number of users goes up the number of requests go up and ultimately the number of connections and queries on the db will go up (however far more dramatically).

Here are some scenarios:

1 user makes 1 request for 1 page that takes 500 queries to render = 500 queries on the database.

1 user makes 3 requests for 3 different pages each taking 500 queries to render = 1500 queries on the database.

2 users each make 3 requests for 3 different pages each taking 500 queries to render = 3000 queries.

It’s linear when you look at it in this fashion but if you think about how the database actually performs transactions you will see that the performance hit becomes EXPONENTIAL.

The database performs each transaction in a fashion that is sometimes blocking only allowing a single transaction to request a certain record at a time (rare in read only scenarios) OR the io needed to service requests can actually cause the processor to wait while the hard disks are saturated (common on high volume, data intensive applications). This wait time experienced by the processor can quickly add up to minutes experienced by an individual request. Minutes can become tens of minutes even hours as queries stack up waiting for the hard disks to return the data to the queries in the queue ahead of them.

There are many ways to solve this problem but the first is always to remove the waste.

Why perform all those non-unique calculations over and over again when you can just do them once, store the results and then return those stored results when the same non-unique request comes back again?

That is caching.

There are many levels of caching.

We can cache the query results for each query performed on the database. Every single one of the thousands of unique queries is currently kept cached automatically by the datbase for a certain period of time until it is thrown away to make room for “Hotter” queries.

We can also cache the results of the processing of the business logic (in the case of web applications this is usual some sort of html or xml output) those results can be used to produce an actual .html file on the hard disk.

This called “page level” caching.

Hotness explained

If a certain request is made for a certain piece of data lets assign it an importance value of 1.

If a subsequent request is made for that same data lets add 1 to that importance value so that it is now 2

If no other request is made for that data it’s value stays at 2 for some period of time.

Now let’s graph this importance or hotness value:

The x axis is some deliniation of the volume of data. Lets just presume that all your data is in a single directory and it all has random alphabetical filenames. Then X would possibly start at at AKURALEJD and end at ZNFNALHE. NOTE the filenames are RANDOMLY assigned to the data.

The Y axis is the hotness or importance value assigned to each piece of data so the Hotter the data the higher it is plotted on the Y axis.

What you will find is that SOME portion of the data is hotter than others. The plot of said hotness follows a bell curve.

So why not just cache ALL possible data ever produced by an application?

Memory is limited. We can’t possibly account for every single unique view of the data. SO we can only keep a portion of that data in the cache. There for we draw vertical lines on our graph starting at the middle of the bell curve and expanding them until the content contained in distance between them is equal to amount of available memory. After that the “tail ends” remain un-cached.

Now what happens when what’s hot changes? Well we have to make room in the cache so that this new data can be put in. How do we know what to throw out of the cache to make room? Well we introduce a lifetime or a TTL (Time To Live) on each element in the cache.

So in it’s most basic form a TTL will tell the cache to “Throw out anything older than X”

That means that as time goes on if something HAS not been requested in a longer period of time than the TTL is set to then it will eventually make it’s way out of the cache.

This aging process also allows for the content that is cached to be “refreshed” or updated periodically. SO if you add NEW data or change data in the application then those changes will eventually make their way into the cache and ultimately be seen by people making requests for cached data.

Ok so that all sounds good so why all the issues with caching?

Before we get into this we need to take an even farther step back. Let’s look at how our content is uniquely named.

When it comes to the web our content is uniquely named by it’s URL (uniform resource locator) See: http://en.wikipedia.org/wiki/Uniform_Resource_Locator

With a unique URL the cache is able to determine “what is what” or more technically what the “STATE” of the application is for that URL. The web used to be so simple.

What webservers do with URL’s: I will briefly explain the two most common functions a web server performs. These are:

Responding to GET requests. (commonly associated with clicking a link)

Responding to POST requests. (commonly associated with pushing a submit button)

NOTE that BOTH are “REQUESTS” there for if you click a link OR click a submit button you are “REQUESTING” something from the webserver. In the case of a POST request most times you are also sending something extra to the webserver (for instance a contact form OR your answers to a poll!)

Once the webserver receives your request it will check to see if it is a GET or a POST, then it will check to see what file was requested and then if the file is of a certain type it will sometimes offload the request to some application server (for instance PHP running as a cgi process). This application server (PHP for instance) will then process the request (taking into account the information in the GET or POST, making needed database queries etc.) finally the application server returns a response back to the webserver which will in turn return the response to your browser.

That response may or may return unique information depending on what was passed in the GET or POST request to the webserver (and ultimately to the application server) NOTE: In the past ALL requests both GET and POST caused the browser to refresh the page. THAT fact alone is what defined WEB 1.0.

Enter AJAX.

Now web 2.0. We no longer have Unique URLS for every piece of content. More importantly we don’t have page refreshes upon every request. The same URL/page might look one way when you first visit it and then let’s just say (for the sake of a timely example) you submit a poll for instance. If the submission of that poll does not navigate you away from the page to a new URL then with page level caching you will never see the results of the poll as your submission will only return the same exact cached page that you first saw since the url is the same. THEREFOR you will see the UNSUBMITTED poll once again.

So then how do we deal with caching AND AJAX?

We do so in one of two ways.

1) By maintaining STATE or at least some subset of the state on the client side (within the browser’s cookies).

2) By caching only small parts of a page that don’t change and allowing the other parts of a page to remain UNCACHED. This is commonly called “partial caching”.

So why not use partial caching on my project?

Partial caching is entirely application dependent. It has to be architected into the application from the start. You unfortunately cannot “just add” partial caching because someone has to take the time to determine which portions of an application CAN be cached and which portions CANNOT. Most times off the shelf web applications cannot have partial caching added on top of them. Most traditional CMS applications and legacy web applications are not Web 2.0 “savy” enough.

Drupal falls into a gray area. Some modules are smart enough. Others are not. In general you cannot use partial caching with drupal without some legwork.

SO we use the page level caching described above. Now what?

Well now we have to take into account the functionality implications of a page looking the same way no matter what every time you visit it. Sounds like it doesn’t fix anything, and it doesn’t, without some tricks.

To start we can tell the cache to NEVER respond to a POST request with cached content. That means that every time someone submits a poll then show them the results. The results are generated each time by the application server and then the unique response is sent back to the browser.

Why does the poll still NOT work on the site then?

The long and the short of it:

It does. BUT unfortunately the poll is one of those modules that was NOT written to be used with page level caching.

What’s happening is this:

The first time you visit the site (or at least before ever taking the poll) you get there by typing www.yourpage.com into your browser. That is a GET request and the homepage is retrieved from the cache. You see the poll and you ARE allowed to vote and you DO see your results. Your request is sent via a POST to the application servers and your response is returned by the application servers… NOT the cache because it was a POST request.

Then you navigate away from that page and everything is fine. Then let’s say you come back to the home page by clicking the logo in the top left. That is a GET request. The homepage is returned FROM THE CACHE this time and you do NOT see your results in the poll. You see a poll that has not been taken yet. Because that is how the homepage was cached originally.

SO then you try to take the poll again and this time it doesn’t let you submit. Why? Because the poll module in drupal was built to only allow ONE vote per person per poll. That means that you CANT submit the poll more than once.

So since you have already submitted and since the poll is cached as unsubmitted you are stuck with an unsubmittable poll.

DARN! How do we fix this?

The easy way: Make the polls allow more than one vote per person per poll. Really why is that so bad?

The hard way: Rewrite the poll module correctly to refresh itself via another Ajax call for an un-cached version of the results. This means essentially writing a new poll module.

I hope that explains why the polls are not working. Maybe you picked up a thing or two a long the way.

</snip>

More research on Youtube racism with Hadoop and Nutch

2009-11-29T07:16:36-08:00

I will go into the details later but I am thoroughly convinced that youtube is one of the largest most racist places on the internet. You would be hard-pressed to find a more high profile site with the N-word used so frequently in a derogatory sense. Anyway more recently in parallel I have been looking for some way of getting my hand’s on enough data to put Hadoop to the test. About a week ago the two idea’s converged in my head and I had a vision of creating an application that would consume all of the comments on youtube (or at least as many as it can swallow) and then using a little linguistic regexp magic figure out who is the MOST racist user on Youtube. I started out by writing the basic mapper in python in my previous post. It is only good for a single video however. I new I would need a spider and I was planning on writing my own (python has some awesome dom and xml parsing: beautiful soup mini-dom, expat) but decided that for future use it might be better to make use of something based on lucene. I have been using solr quite a bit lately but was hoping for something a little more lightweight. Luckily I was able to remember the name of the spider that came out shortly after lucene was made available. Nutch! It hasn’t had the glory that Solr has had in recent years (I don’t understand why these two projects exist and don’t converge?) but it turns out that it is now based on Hadoop by default! In fact little to my knowledge but it looks like Hadoop was actually spawned by Nutch?! Anyway I spent some time setting up nutch on a hadoop cluster today and I have to say it is still not the easiest thing in the world to work with. I am still new to Hadoop and was following this Nutch tutorial very closely. Unfortunatley it glazes over setting up the slave nodes so I plan to fill in the details here and offer them to the community. One thing that it fails to mention that I should have know from my other Hadoop tutorials (never underestimate the minutia!) hdfs configurations need to use FULLY qualified hostnames! you can’t just user your machine name even though you may be able to ping each machine on a local subnet with just the hostname and even though you can telnet from the slave machine to say an Httpd instance running on the namenode you WILL get silent failure when trying to telnet to the namenode daemon’s port. Check your firewalls of course but even if they are turned off on both your master and slave it is quite pssible you will just see:

telnet: connect to address 127.0.0.1: Connection refused

$telnet master 9000

telnet: connect to address 192.168.0.1: Connection refused

however if you try the same thing from the master machine itself it works!? (assuming your name node is currently running and supposedly listening on port 9000)

This of course through me for a loop and I assumed it was everything from firewalls to bad interconnects between my machines. Strange that I never thought about the hostnames. I guess it was because all other services between them have been working fine. If anyone has any idea why hdfs has a problem with this I would love to know.

Map-Reduce, Hadoop, Hadoop Streaming, Python and racism

2009-11-27T11:25:08-08:00

Here is my python script for grabbing the latest 1000 comments (the api only allows access to the latest 1000 unfortunately) and then checks them against a regexp for matching agains known racist words. Right now it is just looking for the N word. This script will be one of the inner MAP tasks in a series of Map-Reduce steps.

!/usr/bin/env python

import sys

import gdata.youtube

import gdata.youtube.service

import re

racist_pattern = re.compile(‘.igger.’, re.IGNORECASE)

import pprint

pp = pprint.PrettyPrinter(indent=4)

yt_service = gdata.youtube.service.YouTubeService()

yt_service.developer_key = ”” Â Â #turns out the developer key isn’t necessary

urlpattern = ‘http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=%d&max-results=50’

for line in sys.stdin:

video_id = line.strip()

index = 1

url = urlpattern % (video_id, index)

print url

comments =

while url:

if index < 20:

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

comments.extend(comment.content.text for comment in comment_feed.entry)

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print ‘%s\t%s\n’ % (comment.author0.name.text, comment.content.text)

print ‘Author: %s\t Comment: %s\n’ % (comment.author0.name.text, comment.content.text) for comment in comment_feed.entry

url = comment_feed.GetNextLink().href

index += 1

else:

currently the google youtube gdata api will not support over 1000 comments

url = ‘http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=951&max-results=49’ % video_id

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print ‘%s\t%s\n’ % (comment.author0.name.text, comment.content.text)

comments.extend(comment.content.text for comment in comment_feed.entry)

print ‘Author: %s\t Comment: %s\n’ % (comment.author0.name.text, comment.content.text) for comment in comment_feed.entry

break

#!/usr/bin/env python

import sys

import gdata.youtube

import gdata.youtube.service

import re

racist_pattern = re.compile('.*igger.*', re.IGNORECASE)

#import pprint

#pp = pprint.PrettyPrinter(indent=4)

yt_service = gdata.youtube.service.YouTubeService()

#yt_service.developer_key = "AI39si7MDdkK_3HKW7C-NykJxoCuBYSBk3GfFDdjEG7tHWmNIZKyLgnvLR9sj6D4wss3IXWQ-oIWm_hB29vb7oOFUCMk8OClMQ"

urlpattern = 'http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=%d&max-results=50'

for line in sys.stdin:

video_id = line.strip()

index = 1

url = urlpattern % (video_id, index)

#print url

comments = []

while url:

if index < 20:

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

#comments.extend([ comment.content.text for comment in comment_feed.entry ])

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print '%s\t%s\n' % (comment.author[0].name.text, comment.content.text)

#print [ 'Author: %s\t Comment: %s\n' % (comment.author[0].name.text, comment.content.text) for comment in comment_feed.entry ]

url = comment_feed.GetNextLink().href

index += 1

else:

#currently the google youtube gdata api will not support over 1000 comments

url = 'http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=951&max-results=49' % video_id

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print '%s\t%s\n' % (comment.author[0].name.text, comment.content.text)

#comments.extend([ comment.content.text for comment in comment_feed.entry ])

#print [ 'Author: %s\t Comment: %s\n' % (comment.author[0].name.text, comment.content.text) for comment in comment_feed.entry ]

bre

Varnish and Squid working together... What?!

2009-11-12T12:43:35-08:00

I have been using Varnish for quite some time and have always wished that there was some way for Varnish to know to serve “Stale” pages when the upstream application servers are swamped. There is actually a feature request for this on the Varnish Trac system here. NOTE: this feature should not really be necessary unless you have underestimated the ability of your application servers to handle your traffic. However even after proper capacity planning sometimes you get well… DUGG. We all know the “digg effect” (formerly referred to as the “slashdot effect”) and it’s repercussions (500, Guru meditation, Houston we have a problem!) There are many ways to skin a cat, but none would be as simple as this (considering we have an existing varnish setup). I should note that simply getting “Dugg” or “Slashdotted” normally wouldn’t take down a site with a proper reverse proxy setup based on Varnish. If your TTL is appropriate and you are using an appropriate GRACE value (for you Squid readers: ”stale-while-revalidate”) then you will probably not saturate your app servers. Unfortunately if your content is good and your UI is right then maybe, just maybe a certain percentage of your new readers will stick around. And here is where it gets scary for the app servers. Maybe just, maybe your new readers will start to navigate in ways that your cache is not used to. Maybe they will start to hit those really OLD articles that haven’t been requested in months! If you think about your sites content vs it’s popularity you it will look something like this:

No matter what you do there will always be something that falls into those “long tails” if your traffic patterns shift suddenly you can very well start to make a lot more request to your upstream servers than you (or more importantly your reverse proxy) expected.

Back to the task at hand. What can I do while I wait for the Varnish team to put this feature through? EASY… use Squid! There are so many debates over which reverse proxy is currently the fastest, which one is easier to setup or integrate with legacy apps etc. I'm certainly NOT trying to get into that! In fact I will skirt the issue entirely saying this: when the features are right and you can afford to use it then why not? NOW don’t get me wrong. Afford can mean a lot of things. Take it as you will. I personally HATE using software, ANY software when I don’t have to. In fact I try to design my stacks as small as possible. As a general rule LESS SOFTWARE IS BETTER! It means less maintenance, less quality assurance… less hastle! However there are situations like the one I described above when you are put between a rock and a hard place. I can either:

A) Swap Varnish out completely and start using squid.

B) Augment my http acceleration layer with squid.

C) Buy more application servers and avoid the issue.

I wish, I wish, I wish C was always an option. Unfortunately not all client’s can afford to simply throw more money at the problem. If I had my choice I would scale horizontally off to the…horizon. SO I now get to choose between A and B. A is what my Sysadmin gut feeling (aboutÂ never using more software than necessary) is telling me to do. BUT A also has the Test Engineer in me screaming “You will have to test everything all over again!”

Sooo here is another instance where the REAL WORLD comes crashing down on good systems engineering. B) is the cheapest most cost effective solution. It could be said that maintaing another piece of software over time is going to be more costly than the upfront cost of swapping out Varnish entirely. But consider this…the Varnish feature that I was mentioning earlier… has already been assigned. It is only a matter of time before someone decides to pick it up and implement it. Hell I might even go ahead and do it if I can find the time. (BTW if your reading this month’s past the publish date of this post then you should definitely check that Trac ticket and see what has become of it.)

So B) it is. Now I am going to have to dust off my Squid skills and install that beast again. (Of course I couldn’t get through an article about varnish and squid with out some opinion…. Setting up Squid is not the easiest thing in the world!)

When good code goes BAD or The Unfortunate Misuse of Good Software Development Principles

2009-11-09T19:29:31-08:00

I had a bit of a quandary today as I reviewed some code today and I decided to write up this little post on the experience. In essence it came down to choosing between code readability and code reuse. I am a huge fan of both and they almost never conflict in fact they almost always exist in a symbiotic state. But not this time.

This title of this post has been buzzing around in my head recently as I have been doing a lot of performance tuning on a very large and HIGHLY trafficked web site of a major print publication. Anyway I was recently reading (reference coming soon! until then dig through my delicious links) a blog post on the topic of performance tuning and why it is silly how often people micro-tuning. Think of that in the same negative respect that you think of micro-managing. It’s a waste of time and resources. The moral: even though performance tuning is normally a good thing, after a while it looses it’s value. You spend your money on the low hanging fruit (AHEM more hardware!)

Back to today. The code that I was reviewing had a few very cryptic stanzas that included a call to a function that consumed a single integer based parameter which after thorough inspection simply ended up determining if the query it finally triggers against the database is ascending or descending. Now I honestly doubt this developer (no matter how Jr) actually thought they would be increasing the performance of the application by including this integer parameter. In fact I am sure that they assumed that they were doing a good thing by reusing code (which I am a HUGE proponent of normally!) but when it came time for another developer to get into the code to augment it slightly it took exponentially more time then it would have if the original developer had simply a) created the database queries inline or b) created two separate nearly identical functions. Instead (fyi we are using an ORM that follows the ActiveRecord Pattern) the object has one single method getNextArticle(int foo) that returns both the previous AND the next article! How confusing! Anyway the best solution that required the least new untested code at this point was to wrap that function with two new functions that map to the integer values. Now we have:


/*Note that the following function has a different method signature than the original and thus can coexist due to the method overloading feature of java*/
getNextArticle(){ 
 return this.getNextArticle(1);
}


getPreviousArticle(){
  return this.getNextArticle(2);
}

I can already hear someone out there saying: “That’s silly why not re-factor the whole object so that it has:

getNextArticle()getPreviousArticle()

wrapping some new function that is called with a String paramater like ASC or DESC.” Well to put it simply, time. Regression testing takes time and we would have to do a heck of a lot more of it on this project that unfortunately does not yet have 100% unit test coverage (to my DISDAIN) and does not have an automated testing setup. So the moral of the story is this. Even though code reuse is almost always a GOOD sign of proper software development practices sometimes it can lead to poor readability which then becomes more of a maintenance problem than the code reuse actually solves. In retrospect it would have been great to have found this code before it went through thorough user acceptance testing. However due to the accelerated nature of meeting client demands and working on unrealistic project schedules we were not able to do enough peer code reviews of code nor were we able to create a team large enough to do pair programming.

AHH how the real world always ruins principles, techniques, patterns and paradigms that work so well on paper!

First Post

2009-11-08T18:06:37-08:00

After nearly a decade of the “cobler’s children have no shoes” I have decided to pick this blog back up again and write about my experiences as a software architect. I am planning a few articles on my work specializing in highly durable, highly available, horizontally scalable open source architectures pertaining not only to the web but to any massively concurrent application. Any questions, comments or topic suggestions are gladly accepted.