Saturday, August 8, 2015

What I've Learned in Four Months Answering Questions on Stack Exchange

I interview a lot of software developers. I used to ask where they went for answers when they were stuck, but I stopped asking the question because literally everybody answered StackOverflow. I knew it. I go there daily, too.

StackOverflow is a question and answer forum for programmers. Their format is so effective for getting answers on how to do things, that the users branched out into other topics, and not necessarily techie topics. Though I've been a StackOverflow consumer for years, I signed up and answered my first question about four months ago (April 2015). Someone asked if it were possible to use the Xcode text editor as a standalone application (without all the IDE stuff). I happened to have been playing around with it like that one day after updating XVim, and happened to know how it could be done. Granted, I had to agree with the others who responded to the question with a "Why on earth would you want to do that?" But I registered as a user and provided an answer to the question. It was an old question from 2013, obviously long-forgotten by the original poster ("OP" in online forum speak), and nobody has up-voted my answer. It's all good. I have 127 reputation points ("rep") on StackOverflow. It's not impressive. Basically, every question I can think to ask, already has an answer, and you don't usually get up-votes for saying something that's already been said.

There is a footer on each StackOverflow page with links to other forums in the StackExchange network. I hadn't bothered to notice any of those in the years I've been a user, though many of them looked familiar, as I had been sent to them by some of my Google searches. I remember thinking when I would see some of those other sites, "Hey, they must use the same forum software as StackOverflow." Silly me. Now I know.

But since I had already signed up for StackOverflow, I figured I'd check out some of their other networks. Since I spent five years aspiring to be a successful financial advisor, I immediately took an interest in the Finance and Money site. I found I could answer a lot of folks' questions, and in looking at some of the other answers, I felt like my conservative approach to money and investing might be appreciated. In four months, I've built up about 2000 rep, which is respectable.

Eventually I ended up spending most of my time in The Workplace network. This site deals with dilemmas, problems, and pitfalls in the workplace. I've really enjoyed offering advice on how I would approach a situation, and reading about others' advice for the same situations. I've learned a lot from people around the world. As of August 2015, I've built up over 6000 rep in The Workplace, which puts me in the top 0.66% of rep growth for the year. Not bragging, though, as there are others who are growing faster than I am, and they deserve it, too -- they give good answers, and generally avoid making someone feel bad for having asked a question.

I've enjoyed spending time on StackExchange. It has filled my need to write, which explains the long gap in this blog (which is something I intend to remedy).  I've learned a thing or two about how to be successful in an online discussion/Q&A forum. I thought I'd share.

#1: Be Nice

I started joining into discussions on Reddit just before getting into StackExchange. When it was appropriate, I'd post a link to a blog post or one of my iOS apps, just to try to generate some traffic to my site. Reddit is similar in many ways to StackExchange, with multiple "sub-reddits," or groups, organized along topics of interest. However, Reddit is more of a discussion forum, whereas StackExchange is a Q&A forum. In those discussions, Redditors (users of Reddit) can often become very rude and crude. To quote the great literary figure, Eliza Doolittle (My Fair Lady), "I'm a good [boy], I am." I've lost most of my interest in Reddit, simply over this one point. F-bombs are not the only way to express strong feelings.

The top rule in each StackExchange site is Be Nice. (See the Help section of each forum: The Workplace, for example). And they enforce it. I actually appreciate that profanity and vulgarity are frowned upon. It elevates the conversation, and keeps the content focused on the questions and answers. I've noticed that many users who join the forum and come on too strong, either soften their tones, or drift away from the site.

When I signed up for StackExchange, I decided to use my real name as a way to encourage myself to always say things I would be willing to have attributed to me later. It has worked well, so far. There have been times where I've been tempted to send out a little extra "Internet Snark" and have decided against it.

The World Is a Huge Place

StackExchange is used world wide. Although the official language of the site is English, it is used by people from almost every continent (I can't confirm that I've seen anything from Antarctica). I am amazed at how common our questions are, and at the same time, how diverse our cultures are. In The Workplace, it is very common for responders to ask for clarification about which country the OP (original poster) is in. The answers that will follow will be very different when it is known that someone is working in London as opposed to New York City. Labor laws are vastly different. And if you ask me, based on the headaches our European and Asian friends have, I am perfectly happy to stay in the USA and work. But I appreciate seeing the different ways people view the same issues. 

Not Everyone Will Agree With  You

...and that's ok. It happens to everyone who contributes regularly to such a forum. You spend a few minutes crafting the best answer you've ever written. Your advice is sound. Your spelling is checked. You hit the Submit button, and within five minutes, you have two down-votes, and a comment telling you why your advice is bad. (It's usually said in a nice way, but it's still clear that they disagree with you.)

It's ok to have the Internet disagree with you. You have a choice: blow it off and disregard it, or look at it and see if maybe you can learn something from it. I've done both. 

A Final Tip: Be the First to Answer ;)

Ok, this is unabashed self-aggrandizement, but if you want to build your rep, you get a boost from being the first answer (as long as it's a good answer). Most good questions get multiple good answers. Most readers who vote on answers (about 10% of viewers, by my observations) vote for the first one they come across and agree with. Answers are ordered according to the number of votes they receive, so when a question has just been asked, the first answer is often the one that gets the most votes, and would then be ranked at the top of the page, thus helping it to get even more votes.

I recently tried an experiment. I answered a question (first, mind you) and it got 96 up-votes. If it had gotten to 100, I'd have earned a gold badge for a Great Answer. Those are fun to get. It's always nice to have strangers from around the planet validate your self-worth. I was left wondering whether four more kind souls would come along and agree with me. 

But then, I remembered seeing another user (who has more than 88,000 rep) update his own answers from a couple of years ago. Anytime a question or answer is updated, it shows up at the top of the Active Questions list (the default page when you arrive at the site). So it gets an old question in front of new users. I have no reason to believe he was trying to boost his rep. From what I've observed, he values refactoring his answers (to borrow a programming term - it means fixing things now that we've learned a thing or two). But I'm not above juicing my rep a little. So I made an update to the question with the intent that it would get listed in the recent update list, which it did. Now, my update was legitimate. I incorporated advice that was provided by others as comments to my answer. It improved my answer, and if StackExchange had a way to share rep with other users whose ideas make your answer better, I would do it.  But it worked! I got my four more votes, hit 100, and got my fourth gold badge. I feel fulfilled.


You know the funny thing about me is this... I get a little obsessive about checking whether anyone agrees with what I've said. I'd post an answer and then check 100 times throughout the rest of the day to see if anyone voted for it, or commented, or responded in any way. I'm a little pathetic that way. After publishing this post, I have the strongest urge to go check my stats and see whether I'm getting any extra page hits on this blog. Sigh. I need a life.

Monday, March 30, 2015

Kibana Quick-Tip: Save those Dashboards!

I like to learn things the hard way, I guess. In an attempt to keep my installation clean, I went to remove the kibana-int index from my system. Why? Because I didn't know what it was, and it had a date from way back when I originally set up my ELK stack on this system. (No, I don't actually read docs. I apparently must always learn by sad experience.  Sigh.) It was easy enough to delete, and I felt good that I had cleaned up some cruft.

Except, now my dashboards are gone. Turns out, Kibana creates its own index, named kibana-int for storing the dashboards and stuff. Oops. 

Did I have a backup of my indices? No - it's just log messages, and the originals are still on their respective systems. We use ELK as a convenience when things are amiss, which rarely happens (except when I go around "cleaning" things!). So I had to spend an hour rebuilding my dashboards. The new one is actually better than the old one, so I suppose that's the optimistic view of the story.

Here's the tip: Export your dashboards to a local file, and keep them under source control, like git.

Then, if you ever need to reload your dashboard after a disaster (or a "cleaning"), you can just load it from a local file in the Open Dashboard menu. Bonus: You can also open a dashboard from a Gist using the same Advanced screen under the Open icon.

Wednesday, February 25, 2015

How I Made It: TreeDudes Launch iOS Game

The Game: Arrange various tools in Rube Goldberg fashion to help the character land on a target after being flung from the launch platform. (See it in iTunes)


Cocos2d-iPhone: This is a great library. I would have used SpriteKit but it didn't exist when I started this project.

Chipmunk Physics: I bought the license so I could have the Objective-C library. Was it worth the $250? Yes. I believe it was. I didn't have to waste time mentally "bridging" between Obj-C and C. It's a high quality library.

iDraw: Vector drawing software. All artwork is drawn in iDraw. I started with InkScape, but I found iDraw was easier to use and the file format was easier to read. What? What could be easier than SVG? Well, I generated my physics environment by drawing it as a layer in the file, then parsing it and writing Objective-C to build the Chipmunk physics objects at runtime. InkScape uses a lot of transforms, and reverse engineering those got really hard. iDraw uses a zipped plist as a file format, which made it really easy to crack it open and see what it was doing. Its internals are very easy to read and understand, and made my code generator a lot easier to write. It was definitely worth the $25 on the App Store.

Audacity: Sound effects were recorded using things around the house (like a finger in the cheek for the balloon popping sound), recorded on an iPod Touch. I used Audacity to filter out the background noise and tweak the pitches and things to get the right sound.

GarageBand: My wife came up with the main menu audio loop, which I believe is just a pre-canned Apple Loop with a bit of a rhythm added in. She used the iPad version.

bmGlyph: Used for creating bitmapped fonts. I started by doing all my text as graphics in iDraw, but when I went add Spanish localization, I had to switch to text. It was also worth the $10 on the App Store. And the developer is very responsive. I e-mailed him with a couple of questions and a problem, and he asked me to send a sample file so he could see what I was seeing. He sent back some suggestions to accomplish what I was trying to do.

TexturePacker: Used to create sprite sheets to optimize my graphics footprint (a little).

GraphicsMagick: Command line tool used to crop images and prepare them for loading into sprite sheets.

Xcode: This is the de facto for Apple development, of course. I am a Vi-guy, so I used the Xvim plugin so I could feel a little more at home.

iOS Simulator: I used this mainly to debug the In-App purchases and the GameCenter interaction. It was very difficult debugging the gameplay, since the Cocos2d and Chipmunk animations were run by the simulator and were very slow.


Objective-C: All of the game is written in Obj-C. This was my first exposure to it and after the learning curve, I would prefer Obj-C over C++ for almost any development effort. (Note: I am excited by Swift, and am using it on a current project that is just getting underway.)

Python: My supporting scripts are written in Python. idraw_to_gamelevel is the script that cracks open the iDraw files and generates the Objective-C classes that implement each level. export is the script that took the manually exported png and 2x png output from iDraw, cropped out individual images, created sprite sheets, and copied them over to the right place in the Xcode project. I decided to export the 2x and 1x images separately from iDraw. I used GraphicsMagick to crop, but the quality of the scaled files (downscaled from 2x to 1x) wasn't as good.


Once I finally zeroed in on the game concept, it took about year and a half of mornings, evenings, weekends, and burning out a few times, to get it ready to submit to the App Store.  There was a lot of learning I had to do, and I'm convinced I did a lot of things wrong. There is some major technical debt rolled into this app that I will definitely have to refactor if I do any more work on it.

I started by getting a basic GameLayer class in place. This is the class that sets up the level (menus, scenery, etc) and then is called by Cocos2d each frame. It would call Chipmunk to calculate all the physics. It also implements all the Chipmunk callbacks for collision handling. It was basic at first, then evolved into much more and took on a lot of technical debt (i.e. things I should have done better, but didn't) because I wasn't disciplined enough to refactor as I went. I just got into heads-down mode and made stuff work.

With the GameLayer in place, I could start adding levels. To create a level, I would draw it out in iDraw, then add the physics layer. The physics layer is just line segments and shapes with specific attributes that would point to more details in a JSON file. The code generator would load both files at the same time and use the attributes to generate line segments and shapes with certain collision behaviors in an Obj-C class that is loaded by the GameLayer for each level (see below for an example).

The rest is just the grinding work of producing artwork (I am not an artist, but would someday love to take the time to develop that skill), and adding functionality to the GameLayer to support new things I wanted to add to the game.

Would I do it differently if I started over?

Yes, I would. While Cocos2d and Chipmunk are fantastic libraries, the workflow around SpriteKit is just so much easier. In order to support Android, I have considered whether this game would work well as a hybrid web app using Cocos2d and Chipmunk Javascript libraries on top of the Apache Cordova framework. It's not a complex game, but some of the levels have a couple thousand line segments and other physics objects. The ChipmunkJS demo app for the Logo Smash, which also has a couple thousand objects, is pretty slow.

Have I made any money?

A little. $40, roughly. About half in early sales to friends and family, and the rest in ads after making it a free download with In-App purchase. $0 from In-App purchases, by the way. I'm not sad about it. I wrote about it recently.


Here are a few screenshots of my Physics generation...

Level 1
Level 1 Scene

Level 1 With Physics
Level 1 Scene with Physics Layer Visible

Level 5 Physics
Level 5 Physics Layer

And finally, here is the code generated for Level 1.

#import "GameLevel.h"
@interface GameLevel_1 : GameLevel

#import "Level1.h"
#import "Character.h"
#import "FieldObject.h"
#import "CollisionTypes.h"
#import "Screen.h"
#import "Animation.h"

#define POSSIBLY_UNUSED(A) if (A)

@implementation GameLevel_1
- (bool) debug { return NO; }

- (id) init {
  if ((self = [super init])){
    self.ID = 1;
    self.gameSize = CGSizeMake(SCRNX(2048),SCRNY(1024));
    [self.gems addObject:[GemDefinition gem:0 at:ccp(SCRNX(936.102388),SCRNY(599.268964))]];
    [self.gems addObject:[GemDefinition gem:0 at:ccp(SCRNX(1015.272542),SCRNY(599.268964))]];
    [self.gems addObject:[GemDefinition gem:0 at:ccp(SCRNX(1094.442735),SCRNY(599.268964))]];
    [self.gems addObject:[GemDefinition gem:0 at:ccp(SCRNX(1051.625752),SCRNY(679.538748))]];
    [self.gems addObject:[GemDefinition gem:1 at:ccp(SCRNX(973.155487),SCRNY(679.538748))]];
    [self.gems addObject:[GemDefinition gem:2 at:ccp(SCRNX(1049.410717),SCRNY(837.354861))]];
    [self.gems addObject:[GemDefinition gem:0 at:ccp(SCRNX(1173.612969),SCRNY(599.268964))]];
    [self.gems addObject:[GemDefinition gem:1 at:ccp(SCRNX(1130.096015),SCRNY(679.538748))]];
    [self.gems addObject:[GemDefinition gem:1 at:ccp(SCRNX(1007.524312),SCRNY(759.808502))]];
    [self.gems addObject:[GemDefinition gem:1 at:ccp(SCRNX(1096.744143),SCRNY(759.808483))]];

  return self;

- (ChipmunkSpace*) createSpace {
  ChipmunkSpace *space = [[ChipmunkSpace alloc] init];
  space.gravity = cpv(0.000000,-400.000000);
  space.damping = 0.900000;
  ChipmunkSegmentShape *wall;                      POSSIBLY_UNUSED(wall);
  self.launchPad = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(-2.000000),SCRNY(597.000000)) to:cpv(SCRNX(230.000000),SCRNY(597.000000)) radius:0];
  self.launchPad.elasticity = 0;
  self.launchPad.friction = 0;
  self.launchForce = cpv(SCRNX(15.000000),SCRNY(0.000000));
  self.launchPad.collisionType = CT_LaunchPadSegment;
  [space add:self.launchPad];
  self.launchSensor = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(-2.000000),SCRNY(597.000000)) to:cpv(SCRNX(230.000000),SCRNY(597.000000)) radius:0];
  self.launchSensor.sensor = YES;
  self.launchSensor.collisionType = CT_LaunchPad;
  [space add:self.launchSensor];
  //Target(s) = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(1815.437917),SCRNY(121.097526)) to:cpv(SCRNX(1915.267717),SCRNY(121.097526)) radius:2]; = 0; = 2.5; = CT_Target;
  Sensor *sensor;                                  POSSIBLY_UNUSED(sensor);
  ActiveFieldObject *afo;                          POSSIBLY_UNUSED(afo);
  RotationAnimation *rotation;                     POSSIBLY_UNUSED(rotation);
  OscillationAnimation *oscillation;               POSSIBLY_UNUSED(oscillation);
  FollowPathAnimation *followPath;                 POSSIBLY_UNUSED(followPath);
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(223.266024),SCRNY(584.583267)) to:cpv(SCRNX(209.997305),SCRNY(509.934742)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(223.266024),SCRNY(584.583267)) nextNeighbor:cpv(SCRNX(209.997305),SCRNY(509.934742))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(209.997305),SCRNY(509.934742)) to:cpv(SCRNX(200.227196),SCRNY(414.490535)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(209.997305),SCRNY(509.934742)) nextNeighbor:cpv(SCRNX(200.227196),SCRNY(414.490535))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(200.227196),SCRNY(414.490535)) to:cpv(SCRNX(192.135846),SCRNY(195.342987)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(200.227196),SCRNY(414.490535)) nextNeighbor:cpv(SCRNX(192.135846),SCRNY(195.342987))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(192.135846),SCRNY(195.342987)) to:cpv(SCRNX(186.883136),SCRNY(34.529300)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(192.135846),SCRNY(195.342987)) nextNeighbor:cpv(SCRNX(186.883136),SCRNY(34.529300))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(1798.693016),SCRNY(38.837067)) to:cpv(SCRNX(1816.270816),SCRNY(93.071957)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(1798.693016),SCRNY(38.837067)) nextNeighbor:cpv(SCRNX(1816.270816),SCRNY(93.071957))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(1816.270816),SCRNY(93.071957)) to:cpv(SCRNX(1817.306316),SCRNY(119.317667)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(1816.270816),SCRNY(93.071957)) nextNeighbor:cpv(SCRNX(1817.306316),SCRNY(119.317667))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(1931.003741),SCRNY(37.626398)) to:cpv(SCRNX(1918.337241),SCRNY(61.221718)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(1931.003741),SCRNY(37.626398)) nextNeighbor:cpv(SCRNX(1918.337241),SCRNY(61.221718))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(1918.337241),SCRNY(61.221718)) to:cpv(SCRNX(1913.339041),SCRNY(120.294938)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(1918.337241),SCRNY(61.221718)) nextNeighbor:cpv(SCRNX(1913.339041),SCRNY(120.294938))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  wall = [[ChipmunkSegmentShape alloc] initWithBody:[space staticBody] from:cpv(SCRNX(-2.723418),SCRNY(46.361666)) to:cpv(SCRNX(2497.276587),SCRNY(46.361666)) radius:4.000000];
  [wall setPrevNeighbor:cpv(SCRNX(-2.723418),SCRNY(46.361666)) nextNeighbor:cpv(SCRNX(2497.276587),SCRNY(46.361666))];
  wall.elasticity = 0.000000;
  wall.friction = 1.000000;
  [space add:wall];
  for (GemDefinition* gem in self.gems){
    ChipmunkBody *body = [ChipmunkBody staticBody];
    body.pos = gem.position;
    ChipmunkShape *shape;
    int size = 40 - (5 * gem.type);
    shape = [ChipmunkPolyShape boxWithBody:body width:SCRNX(size) height:SCRNY(size)];
    shape.sensor = YES;
    shape.collisionType = CT_Gem;
    gem.shape = shape;
    [space add:shape];
  for (ActiveFieldObject *obj in self.gameConfig.activeFieldObjects){
    obj.body = [obj.definition createInSpace:space position:obj.position angle:obj.settings.rotation rogue:NO];
  return space;
} // end of createSpace()

- (void) addSceneryToNode:(CCNode*)node staticNode:(CCNode*)staticNode {
  CCSprite *image;                                       POSSIBLY_UNUSED(image);
  image = [self loadImage:@"L1-Bkg-0-0.png"];
  image.anchorPoint = ccp(0,0);
  image.position = ccp(SCRNX(0.000000),SCRNY(0.000000));
  [node addChild:image z:-999999];
  image = [self loadImage:@"L1-Bkg-1024-0.png"];
  image.anchorPoint = ccp(0,0);
  image.position = ccp(SCRNX(1024.000000),SCRNY(0.000000));
  [node addChild:image z:-999999];

Friday, February 20, 2015

Should I Still Be Using Vim in 2015? (Yes)

Last summer, I wrote a post about why I keep looking for something better than Vim, and why I keep coming back to it when it's time to actually sit down and get some work done. I've been editing the Vi way since 1992, but there are some things about Vim that I wish were better (easier to script, prettier to look at, better context-aware completion). So I occasionally wander away from Vim to see if things are actually better. The past few months have included the longest such wandering episode I've had in years. There are newer options for text editors out there, and they have passionate followers. Here's what I've seen on my little text editor excursion (in no particular order).


Github's text editor was released as open source last summer after an invitation-only beta. It's off to an impressive start with a great community around it.

What did I like about it? 

  • Cross-platform. I used it on Mac and Windows (performs a little less smoothly on Windows).
  • Autocomplete2 plug-in is pretty good (though it was very slow looking up C completions, presumably calling out to clang).
  • Package manager built-in. 
  • Extremely easy to find and set configuration options via the settings pane search field.

Why didn't I stick with it? It has a huge memory footprint at runtime, and spawns at least two processes just to edit a single file. It gets sluggish if you try to open very large text files. Built first on webkit, then on Chrome, it is a platform suited for displaying rich content, but it does so at great expense.


Adobe's text editor geared toward web developers has been around for a few years. It's community is less productive than Atom's. But the editor is solid.

What did I like about it?

  • Cross-platform.
  • Inline editors for things like CSS options and colors.
  • Live preview

Why didn't I stick with it? It does a great job in web development, but is just a regular editor for other types of projects. It is built on a similar platform as Atom (webkit or Chrome). I criticize it the the same as Atom - huge resources are required to edit a file. 


This is a very interesting project. It aims to be a platform for evolving how we develop software. The problem is it's really slow.  Really slow. And unlike Atom, which is highly usable right out of the box, the cool features shown on the website require configuration to achieve, and the docs weren't helpful, and it just wasn't worth it. 

Sublime Text

Loads of people love this editor. I just don't. It didn't impress me that it could offer anything I didn't already have available to me in TextMate. Being a paid product, I couldn't bring myself to pay for it (which I believe you should, if you use it) when it didn't provide a clear advantage over other options I have that don't cost anything.


This venerable editor for Mac is fast, lightweight and has a loyal following. The fan base has waned, though, as the long-awaited TextMate2 took too long to come about. In reading posts and forums, it seems like most folks that bailed went toward SublimeText. But TextMate2 is now available as free software. 

What did I like about it?
  • You can tell its author(s) understand Unix workflows in the way bundles (plug-ins) work. It's a comfortable environment for me. 
  • It's fast, and lightweight.
  • Once you become familiar with the keyboard shortcuts, you can be very efficient navigating and editing code and text in general.
  • Rmate is nice - it's a ruby script you can install on remote machines you access that allows you to easily edit a remote file in TextMate on your local machine. Works great.

Why didn't I stick with it? At my day job, I have to work in a Windows environment very often, and it's not cross-platform. Also, although you can get very efficient at editing text, the keyboard shortcuts don't seem to have any rhyme or reason to the composition. I may be getting old, but I just couldn't remember the (seemingly) random key combinations to do things. I always had to have a cheat sheet handy.


Yes, no editor walkabout would be complete without at least attempting (once again) to like Emacs. I just don't. Maybe it's the learning curve? Maybe I'm just really picky? I ended up using Aquamacs this time around, and was impressed by it. They managed to put a glossy finish on a stodgy old editor and have pulled it off really well. Their default settings make it more of a good Mac citizen, which I appreciated. But, in the end, I just didn't enjoy the learning process (same as always). One thing I liked about Emacs this time around, though, was the ability to sudo while editing a remote file via Tramp. I wish Vim's netrw could do that.

The Return to Vim

Anyone reading this article might infer that I'm just itching to leave Vim as soon as I find a viable alternative. There is some truth to that. I think everybody wants to improve their lives if they can. But anything that is going to aspire to replace Vim in my toolbox is going to have to overcome more than 20 years of familiarity with the Vi way of navigating text. 

For me, there is just no more efficient way to navigate and manipulate text, especially source code, than the Vi way. 

Vim is cross platform in a way that no other cross platform editor can be, except maybe Emacs, but the key chords are too burdensome (yes I'm that petty), even after remapping my CapsLock and Return keys. The problem all other editors have between Mac and Windows is the difference between Cmd and Ctrl. Yep - on Mac it's Cmd-S to save. On Windows it's Ctrl-S. Using an Apple keyboard, this is a big deal, because muscle memory causes me to hit Cmd-S, which on a PC keyboard usually ends up being Alt-S, and in VirtualBox, is Win-S. See my dilemma? In Gvim, or MacVim, I can use the common keystrokes if I want to, but I don't have to. Instead, I can use :w, which I don't even have to think about. I think 'Save' and my fingers have typed :w within milliseconds. Even in Emacs, after remembering what it is, I'd still have to hit Ctrl-x, Ctrl-s. That's just too much :)

No other editor provides the notion of Text Objects. That is one of Vim's secret killer features. Having the ability to address a chunk of text (a block of code, the contents within parentheses, array brackets, within quotes, etc) as a unit and the act on it (select it in visual mode, copy it, cut it, change it, use it as input to a search command, etc.) is a huge benefit.

And just like I said in that article last year, returning from my excursion to Vim feels like coming home. It's like drinking water, or breathing air. What I said last summer still applies...

Here's where they all fell short, and what [takes] me back to Vim every time...
  • hjkl for navigation (EDIT: 'hjkl' is really an identifier for the whole concept of using plain old keys to do what you need to do. No special modifier keys are necessary.)
  • / for searching (really, no other editor has made it this simple)
  • m to bookmark a spot, ' to hop back to it
  • and ctags
  • q to record a macro; @ to execute it again
  • 1G to go to the top of the file, G to go to the bottom (EDIT: commenters helped me relearn this one: gg will do the same thing, cutting out the need to hit the Shift key).
  • ^, $ to go to the front, or end, of a line.
  • Text Objects! Holy cow, if you're a Vim user and you're not using these, you're missing something special.
  • Platform ubiquity. (Mac, Linux, Windows - Vim works great on all the platforms I use day in and day out.)

What it boils down to, really, is that the above keys have become a part of me. They are so ingrained in my brain and reflexes, that I think of what I want done in the text and my hands just naturally do it for me. It's like drinking water. I don't have to think about how to go about bringing the cup to my mouth and tipping it just right. I just do it, and keep going on about my business.
Please don't hate me because I wander. It's just more validation that Vim can stand up to any competition.

I don't claim to have the most dazzling .vimrc. But if you're interested, you can see mine on github: Some of it is really old. I'd love to hear about any recommendations for changes to it.

Saturday, January 31, 2015

Learning From Mistakes

Here's a good tagline for this post: Mistakes happen. They are not the end of the world. They can be fixed or compensated for. What matters most is learning from them. And, yeah, in software, proper testing is important!

A quote attributed to Thomas John Watson, Sr. reads, "Recently, I was asked if I was going to fire an employee who made a mistake that cost the company 600,000. No, I replied, I just spent 600,000 training him. Why would I want somebody to hire his experience?"

I appreciate that philosophy. Mistakes, when they truly are mistakes and are not the result of ill intent or a demonstrated inability or refusal to learn, should be regarded as teaching opportunities. I firmly believe that success and failure together are the best way to get a "well-rounded" education in life.

This week I made a minor mistake at the day job. I renamed all the internal groups in our JIRA (issue and task management software) in preparation for synchronizing groups from our directory server. All seemed to go fine until a coworker appeared in my door and said that everyone on his team had just lost their workflow buttons. They had been there a few hours earlier. Hmmm... Turns out, JIRA refers to groups by name in lots of places, and due to the ultra-generic (flexible?) way big Java applications all seem to be designed these days, it was impossible for me to have written a script to find all these corner cases. Now that I've figured out where they were, I have been able to deal with the two or three other cases during the week where someone said they couldn't see something they were supposed to be able to see.

Lessons learned? First, seriously, why was I doing this in production first? Second, I learned something about the tool I would never have learned otherwise. I am better for it, and therefore my company can be better for it as well. Third, most mistakes are are not unrecoverable. And fourth, even if a mistake is unrecoverable, there is always a way forward from it, and things can get back on track.

Ok, I could end this post right here, and you might judge it to be a nice little piece of possible wisdom that most people already share. But, like I said, this was minor mistake. I've been the cause of some major headaches, mostly very early in my career. Read on for some entertainment...

What happens when I delete the C:\DOS directory?

I used a student loan and grant money to buy my first computer in college (a 386SX with 2MB RAM and an 80MB hard drive, I think this was 1992). This way I wouldn't have to spend hours upon hours  in the computer labs to work on assignments, and could be home with my beautiful wife. Having a computer at home gave me plenty of opportunities to tinker and try things. Curious about what would happen, I one day decided to delete the C:\DOS directory.  Imagine my surprise when the computer wouldn't boot after that. I learned a little more about what an operating system actually does (I was in my first year of computer science at this point). After a little time panicking, seeing my wife's realization that I might have just thrown away $2000 only added to the horrible feeling in the pit of my stomach. In desperation, I put in floppy disk #1 and tried rebooting. Relief! It booted and offered to install DOS again. More learning.

Too much whitespace in the code in a 24-line 3270 terminal is a problem

Fast forward a couple of years. I am now working as a student programmer in the university's financial department, assigned to work on the payroll system, among other things. One day, upon arriving at work, the payroll team lead asked me if I had tested my recent changes thoroughly enough. Of course, I thought I had, but... Accounting had called. Somehow, this payroll was double what it should have been.

Here's what the problem was. We accessed the development environment on the mainframe via 3270 terminal emulators from our PCs. When paging up and down, the editor leaves one line from the previous page visible to help provide some continuity between the pages. I knew at one point in the logic I was updating I would need to make the call to the ApplyPayment subroutine (or whatever it was called). I found the place in the code where I thought it should be called and added the appropriate line. I was right about it being the proper place for the call, because the call was already in that place, but it had scrolled off the screen. The line that was carried over from the previous page was a blank line, and I was only barely scanning the code as it was scrolling by. There were two or three extra blank lines after the call, which increased the chances that the line I needed to see would not be the line that carried over from the previous screen.

Luckily the accountants caught it before any real damage was done. Every payroll run gets audited (thankfully) before releasing it to the printers and the direct deposit systems. A quick fix, a re-run of the payroll, and a few hours later, the checks were in the mail.

More learning. Testing is important. Good test data is important. But here's an observation. It's difficult to teach college students how to write good tests when they don't have enough experience to understand all the things that usually go wrong in software. Experience is so important to becoming competent in whatever you're doing in life.

Where did all the buildings go?

Fast forward another couple of years. The payroll mistake didn't cause me to be tossed from the career path, doomed to a life of ___(insert_your_least_favorite_menial_job_here)___. Just before graduating, they offered me full-time employment. (I should add that although I'm only highlighting mistakes in this post, I worked my tail off and tried to learn all I could. That overcomes a multitude of sins.)

I was now in charge of the capital equipment inventory system, which is a fancy accounting term that means watching the value of buildings and desks and things go down, usually for tax reasons. I don't even remember what change I was making to the system that day. I coded it up and tested it in the development database, and upon seeing that it did what I expected it to do, promoted it to production.

The next morning, another more experienced developer, the one whose role I had taken over on this part of the system, asked me whether I had tested the changes I had made. I said, Yes. To which he responded, "Oh, maaaan. Something went really wrong last night." More than half of all the capital equipment (the buildings and everything bolted to the floors in them) was missing from the database. It turns out there was a condition in the data I hadn't accounted for, and that wasn't represented in the dev data we had to test with. Crud. I spent 36 straight hours at the office trying to reconstruct the data, but there wasn't enough there to reliably recreate everything. Month-end was coming, and pressure was high.

What's that? Just restore from the backup? Yeah. After a day and half of trying to avoid doing that, we finally turned to the operations department to request the restored data. After a few hours of closed door conversations, they came sheepishly back and said that their backup job had been failing every night for the last six months. The operators had been ignoring the messages on the console. After so long, they just started telling each other that message always happens so it doesn't mean anything. College students. You could sort of give them a bit of a pass, since most of them didn't really know what those messages meant anyway. But the system admins? Never, in six months, did they bother to review the logs?

At this point, yelling and screaming accomplishes nothing, and the business folks over in the administration building were pretty calm people. The accounting director decided to close the month early, using data manually entered from the last report, two weeks old. I fixed my bug and everything proceeded forward from there. Harm was done. Time was lost. Multiple people contributed to this disaster.

What did I learn? Again, testing is important. And having good test data is critical. And if you have any sysadmin responsibilities, it's a good idea to develop a habit of surveying the logs every day as a sanity check.  And, if you're going to work in a job, be proactive and learn all you can about what you're working with.

These whoppers all happened 20+ years ago. To be sure, I have continued to learn a few things by not doing them right the first time. But nothing has really come close to the levels of disaster that I have caused, or potentially caused, in those early years. I suppose this is what we call EXPERIENCE. Life would have been better in those occasions had I not made the mistakes I did. But knowing what can go wrong, really knowing it, makes me more capable now. The only real mistake in life is not learning from the experience when it doesn't go right.

Saturday, January 17, 2015

Monitoring Web Service Performance with Elasticsearch, Logstash and Kibana

Elasticsearch, Logstash and Kibana ( are fantastic open source products for collecting, storing, monitoring and analyzing events.

Here is one way you can configure Logstash and Kibana to monitor system loads and web application response times.  (Elasticsearch, while powerful, does its basic job so well that I did not need to configure anything for it in this example, other than to set it up so Kibana could connect to it, which Kibana told me how to do the first time I tried.)

Configuring Logstash

Note: This configuration works on Ubuntu servers. Your environment may need slight adjustments.

input {
   # System Load
    exec {
        # We are generating our own events here, so we 
        # get to assign a type of "system-loadavg" so 
        # Logstash can apply a specific filter to them.
        type => "system-loadavg"

        # Produce the system loads on a single line.
        # Example:
        #     0.23  0.18  1.03
        command => "cat /proc/loadavg | awk '{print $1,$2,$3}'"
        # Do this every 60s while Logstash is running.
        interval => 60

    # Web Application Response Time
    exec {
        # Again, generating our own events, with type 
        # "ws-ping"
        type => "ws-ping"

        # (Optional) Add a field to our custom event.  If 
        # you monitor more than one service this way, you 
        # will be able to distinguish them using this field.
        add_field => { "service" => "jira" }

        # Execute a couple of commands to get a web 
        # page from JIRA
        #    /usr/bin/time    Uses the Linux time command, 
        #                     not the Bash built-in function
        #    -f '%e'          Specify the output should just 
        #                     be the real time, in seconds.
        #    2>&1             Redirect time's output from 
        #                     stderr to stdout for Logstash
        #    curl             Just google this for an idea 
        #                     of its sheer awesomeness.
        #    -s               Silent mode, don't want the 
        #                     progress bar in the output.
        #    -k               Don't check SSL certificate 
        #                     (this is acceptable for our 
        #                     self-signed internal cert)
        #    -o /dev/null     Send the output to /dev/null. 
        #                     We don't care what the page
        #                     contains, just how long it 
        #                     takes.
        #    -u someuser:somepass 
        #                     We do not want to measure the 
        #                     static login page, so we'll 
        #                     need curl to authenticate with 
        #                     JIRA so it can get to a dashboard.
        #                     The page to load.
        command => "/usr/bin/time -f '%e' curl -sk -o /dev/null -u someuser:somepass 2>&1"

        # Do this every 60s while Logstash is running
        interval => 60

filter {
    # Transform the system-loadavg event by adding names 
    # to the numbers
    if [type] == "system-loadavg" {
        grok {
            # Parse/match the event using the pattern 
            # of three numbers separated by spaces. Map 
            # the numbers to float numbers in Elasticsearch 
            # (otherwise, we won't be able to plot them.)
            match => { "message" => "%{NUMBER:load_avg_1m:float} %{NUMBER:load_avg_5m:float} %{NUMBER:load_avg_15m:float}" }
            # Only store matched values that have names 
            # provided for them (this is the default)
            named_captures_only => true
    # Transform the ws-ping event by adding a name to 
    # the number created by the input event.
    if [type] == "ws-ping" {
        grok {
            match => { "message" => "%{NUMBER:responsetime:float}" }

output {
    # Send all events to Elasticsearch.  Be sure to configure 
    # the correct host.
    elasticsearch { host => "" }

Logstash will now generate performance monitor events every minute and send them to Elasticsearch.

Visualizing in Kibana

Here's how to configure your Kibana dashboard to watch and visualize these events.

Add the following queries to your dashboard, as two separate queries.  You can optionally set the color and alias for the queries so they look better on the screen.  This is done by clicking the color dot in the query field in Kibana. AND type:system-loadavg

  type:ws-ping AND service:jira

Notice that Logstash automatically adds a "host" field to the event.  You can see that our custom event types and fields have made it, too.

To graph the system load average, add a new panel to your dashboard with the following parameters.

Panel type: Histogram
Title: 5 Minute Load Average
Chart Value: mean
Value field: load_avg_5m  (We named this in the Logstash configuration)
Chart option: Line (not bars), not stacked
Queries: selected, then select the query that corresponds to the system-loadavg search you defined above.  It will be in the list Kibana provides for you to select.

To graph the response time, do the same thing as above, but select the responsetime field for the Value field, and the query that corresponds to the ws-ping event type as defined above.

That's it!

Here are a couple of things I learned along the way.

  • Kibana didn't update my charts at first.  The updating icon on each panel just sat there, spinning.  The problem was that I hadn't told Logstash to send the numbers into Elasticsearch as numbers, so it sent them as strings.  Kibana was getting errors back from Elasticsearch, but wasn't telling us anything about it.  Using the Inspect tool (the "i" icon on the panel) gives you a curl command to see the result of the query for yourself.
  • Kibana didn't give us the option of plotting the actual value, so we use the mean to provide the closest representation of the value over the time between plot points.  It's a good compromise since it smoothes the plot somewhat.
  • We collect more events from our application logs and Linux system logs. Applying a grok pattern to those was more difficult, and the docs could be better.  But there is a Grok Debugger that allows you to paste in a sample of your event data and play with the grok patterns to see what it is able to match.  Once I found this, my work for the week was done in no time at all.
This is a short intro to a very powerful event management platform.  I hope it helps you get up and running a little faster.

Tuesday, January 6, 2015

Paying the Indie Dues and Simplifying My Life

Let's be honest.  It's lucky for me you happened by my little blog.  I'm nobody of any sort of fame.  I wouldn't say that I'm one of the world's, or the industry's, or even my circle of friends', best thinkers. I'm just a regular person.  A software developer.  I have a day job that more than pays the bills.  I'm really happy there.

Why, then, would I dream of becoming an independent software developer?  It must be because the life of an indie is so glamorous.  I mean, who wouldn't love working from anywhere you want, on whatever you want. And who wouldn't want hundreds of thousands of dollars thrown at you each year as the general population of the world falls all over themselves to pay you a couple of bucks to have the pleasure of using your beloved software creation?

Oh yeah. Apparently, it's not like that.

I am thoroughly impressed by indie developers that can make a living at what they're doing. They work hard and most of them enjoy none of the benefits mentioned a couple of paragraphs ago. But they're making a living.  I listen to many of them on podcasts, and one thing I've come to understand (we all know it, but we don't all understand it) is that their success is the result of years of working at it.

If you look at My Apps, you'll see that I've developed a couple of games for iPhone and iPad.  They're fun, or so I'm told by all my family and friends who have played them (not all my family and friends, mind you, just the ones who have played my games). I admit I had dreams of going viral and retiring on it, or at least paying off my mortgage and sending my kids to college on a TreeDudes scholarship.  The reality is, they're only my second and third attempts at publishing indie software.  My first attempt was about 12 years ago, when I wrote a hex file viewer for Palm OS and sold two copies of it via Handango.  So, between that app, and my two recent iOS games, plus the ad revenue from the games, I have earned so far a grand total of about $40. I suppose the dues to be paid for success are somewhat higher than where I am currently at.  It's all good.  I'm not angry or disillusioned.  Of course I'm disappointed, but it's the same kind of disappointment you experience when you buy a raffle ticket and you don't win.  What else would I expect?

Rovio produced 51 titles before Angry Birds made them instantly famous. David Smith (@_DavidSmith) has a whole portfolio of apps he maintains.  Daniel Jalkut of Red Sweater does, too.   His podcasting partner, Manton Reece, is the same.  (Can you tell which podcasts I listen to the most?)  Interestingly, Manton's app business is his side job.  He has mentioned often in his Core Intuition podcast that he has a day job, too.

Still being honest... I did more than just dream of wild success. I prepared for it. I registered my own domain,  formed an LLC, got an employer ID number (EIN) from the IRS, opened a business account at the bank, and converted my personal Apple Developer account to a business account.  After all, I had to look legitimate so nobody would realize I was just a small potato.

Looking back, I can now understand why my wife (she is so patient and understanding) always looked at me and said, "Well, I don't think all that is necessary, but you're the one who knows about these things, so I trust ya."  $40 in revenue doesn't even come close to covering the hosting fees, domain registration fees, Apple Developer fees, and small account fees at the bank.  And, oh yeah, I had to pay extra for Turbo Tax to help me do my small business taxes, since I had a small (very small) business.  But, after operating on a loss, at least my tax bill was in my favor!

I didn't need any of that stuff to publish my software.  I've learned a valuable lesson.  I'm letting my domain expire. It was always a bit of a pain having to explain that it was "cornerstone" but spelled "kor-ner-sto-ane." We did that so the domain name would be unique.  Yep.  The only thing I hosted on that site was this blog, which was easily moved over to and I actually like the platform.  It's nice not having to worry about Wordpress updates anymore.  My LLC has been terminated.  It's not like it was really going to protect me from a lawsuit, anyway.  The bank account is closed, eroded entirely away by low-balance fees.  Oddly enough, the EIN gets to stay around forever.  I'm sure I'll forget about it someday.  And, yes,  I've opened a new Apple Developer account in my own name again since Apple doesn't convert business accounts back to personal accounts, darn it.

I am not giving up on developing my own little portfolio of apps, though.  I've got a few ideas that I'm thinking about.  One of them will soon be moved from the back burner to the front, and I'll be off developing something new again.  But before I start that, there's an awful lot of home repair and yard work that needs to happen!

I am not foolish enough (anymore) to hope for overnight success,  but I am still counting on life providing for me what it does for others who pay the price required for success. Success takes steady, consistent work.  As I consider it now, maybe the reason I have a good day job is because I've worked at it nearly every day for two decades.  Gosh, imagine that!

I really do feel fortunate that you've read my blog post all the way to the bottom.  Thanks for that.  I'd love to hear from you in the comments below.