Haskell code is expressive.  The HLearn library uses 6 lines of Haskell to define a function for training a Bayesian classifier; the equivalent code in the Weka library uses over 100 lines of Java.  That’s a big difference!  In this post, we’ll look at the actual code and see why the Haskell is so much more concise.

But first, a disclaimer:  It is really hard to fairly compare two code bases this way.  In both libraries, there is a lot of supporting code that goes into defining each classifier, and it’s not obvious what code to include and not include.  For example, both libraries implement interfaces to a number of probability distributions, and this code is not contained in the source count.  The Haskell code takes more advantage of this abstraction, so this is one language-agnostic reason why the Haskell code is shorter.  If you think I’m not doing a fair comparison, here’s some links to the full repositories so you can do it yourself:

• HLearn’s bayesian classifier source code (74 lines of code)
• Weka’s naive bayes source code (946 lines of code)

The HLearn code

HLearn implements training for a bayesian classifier with these six lines of Haskell:

newtype Bayes labelIndex dist = Bayes dist
    deriving (Read,Show,Eq,Ord,Monoid,Abelian,Group)

instance (Monoid dist, HomTrainer dist) => HomTrainer (Bayes labelIndex dist) where
    type Datapoint (Bayes labelIndex dist) = Datapoint dist
    train1dp dp = Bayes $ train1dp dp

This code elegantly captures how to train a Bayesian classifier—just train a probability distribution.  Here’s an explanation:

• The first two lines define the Bayes data type as a wrapper around a distribution.
• The fourth line says that we’re implementing the Bayesian classifier using the HomTrainer type class.  We do this because the Haskell compiler automatically generates a parallel batch training function, an online training function, and a fast cross-validation function for all HomTrainer instances.
• The fifth line says that our data points have the same type as the underlying distribution.
• The sixth line says that in order to train, just train the corresponding distribution.

We only get the benefits of the HomTrainer type class because the bayesian classifier is a monoid.  But we didn’t even have to specify what the monoid instance for bayesian classifiers looks like!  In this case, it’s automatically derived from the monoid instances for the base distributions using a language extension called GeneralizedNewtypeDeriving.  For examples of these monoid structures, check out the algebraic structure of the normal and categorical distributions, or more complex distributions using Markov networks.

The Weka code

Look for these differences between the HLearn and Weka source:

• In Weka we must separately define the online and batch trainers, whereas Haskell derived these for us automatically.
• Weka must perform a variety of error handling that Haskell’s type system takes care of in HLearn.
• The Weka code is tightly coupled to the underlying probability distribution, whereas the Haskell code was generic enough to handle any distribution. This means that while Weka must make the “naive bayes assumption” that all attributes are independent of each other, HLearn can support any dependence structure.
• Weka’s code is made more verbose by for loops and if statements that aren’t necessary for HLearn.
• The Java code requires extensive comments to maintain readability, but the Haskell code is simple enough to be self-documenting (at least once you know how to read Haskell).
• Weka does not have parallel training, fast cross-validation, data point subtraction, or weighted data points, but HLearn does.

/**
   * Generates the classifier.
   *
   * @param instances set of instances serving as training data 
   * @exception Exception if the classifier has not been generated 
   * successfully
   */
  public void buildClassifier(Instances instances) throws Exception {

    // can classifier handle the data?
    getCapabilities().testWithFail(instances);

    // remove instances with missing class
    instances = new Instances(instances);
    instances.deleteWithMissingClass();

    m_NumClasses = instances.numClasses();

    // Copy the instances
    m_Instances = new Instances(instances);

    // Discretize instances if required
    if (m_UseDiscretization) {
      m_Disc = new weka.filters.supervised.attribute.Discretize();
      m_Disc.setInputFormat(m_Instances);
      m_Instances = weka.filters.Filter.useFilter(m_Instances, m_Disc);
    } else {
      m_Disc = null;
    }

    // Reserve space for the distributions
    m_Distributions = new Estimator[m_Instances.numAttributes() - 1]
      [m_Instances.numClasses()];
    m_ClassDistribution = new DiscreteEstimator(m_Instances.numClasses(), 
                                                true);
    int attIndex = 0;
    Enumeration enu = m_Instances.enumerateAttributes();
    while (enu.hasMoreElements()) {
      Attribute attribute = (Attribute) enu.nextElement();

      // If the attribute is numeric, determine the estimator 
      // numeric precision from differences between adjacent values
      double numPrecision = DEFAULT_NUM_PRECISION;
      if (attribute.type() == Attribute.NUMERIC) {
	m_Instances.sort(attribute);
	if ( (m_Instances.numInstances() > 0)
	    && !m_Instances.instance(0).isMissing(attribute)) {
	  double lastVal = m_Instances.instance(0).value(attribute);
	  double currentVal, deltaSum = 0;
	  int distinct = 0;
	  for (int i = 1; i < m_Instances.numInstances(); i++) { 	    
            Instance currentInst = m_Instances.instance(i); 	    
              if (currentInst.isMissing(attribute)) {
                break; 	    
              }
 	    currentVal = currentInst.value(attribute);
 	    if (currentVal != lastVal) {
 	      deltaSum += currentVal - lastVal;
 	      lastVal = currentVal;
 	      distinct++;
 	    }
 	  }
 	  if (distinct > 0) {
	    numPrecision = deltaSum / distinct;
	  }
	}
      }

      for (int j = 0; j < m_Instances.numClasses(); j++) {
	switch (attribute.type()) {
	case Attribute.NUMERIC: 
	  if (m_UseKernelEstimator) {
	    m_Distributions[attIndex][j] = 
	      new KernelEstimator(numPrecision);
	  } else {
	    m_Distributions[attIndex][j] = 
	      new NormalEstimator(numPrecision);
	  }
	  break;
	case Attribute.NOMINAL:
	  m_Distributions[attIndex][j] = 
	    new DiscreteEstimator(attribute.numValues(), true);
	  break;
	default:
	  throw new Exception("Attribute type unknown to NaiveBayes");
	}
      }
      attIndex++;
    }

    // Compute counts
    Enumeration enumInsts = m_Instances.enumerateInstances();
    while (enumInsts.hasMoreElements()) {
      Instance instance = 
	(Instance) enumInsts.nextElement();
      updateClassifier(instance);
    }

    // Save space
    m_Instances = new Instances(m_Instances, 0);
  }

And the code for online learning is:

/**
   * Updates the classifier with the given instance.
   *
   * @param instance the new training instance to include in the model 
   * @exception Exception if the instance could not be incorporated in
   * the model.
   */
  public void updateClassifier(Instance instance) throws Exception {

    if (!instance.classIsMissing()) {
      Enumeration enumAtts = m_Instances.enumerateAttributes();
      int attIndex = 0;
      while (enumAtts.hasMoreElements()) {
	Attribute attribute = (Attribute) enumAtts.nextElement();
	if (!instance.isMissing(attribute)) {
	  m_Distributions[attIndex][(int)instance.classValue()].
            addValue(instance.value(attribute), instance.weight());
	}
	attIndex++;
      }
      m_ClassDistribution.addValue(instance.classValue(),
                                   instance.weight());
    }
  }

Conclusion

Every algorithm implemented in HLearn uses similarly concise code.  I invite you to browse the repository and see for yourself.  The most complicated algorithm is for Markov chains which use only 6 lines for training, and about 20 for defining the Monoid.

You can expect lots of tutorials on how to incorporate the HLearn library into Haskell programs over the next few months.

Subscribe to the RSS feed to stay tuned!

Code and instructions for reproducing these experiments are available on github.

Why is HLearn so much faster?

Well, it turns out that the bayesian classifier has the algebraic structure of a monoid, a group, and a vector space.  HLearn uses a new cross-validation algorithm that can exploit these algebraic structures.  The standard algorithm runs in time , where is the number of “folds” and is the number of data points.  The algebraic algorithms, however, run in time .  In other words, it doesn’t matter how many folds we do, the run time is constant!  And not only are we faster, but we get the exact same answer.  Algebraic cross-validation is not an approximation, it’s just fast.

Here’s some run times for k-fold cross-validation on the census income data set.  Notice that HLearn’s run time is constant as we add more folds.

And when we set k=n, we have leave-one-out cross-validation.  Notice that Weka’s cross-validation has quadratic run time, whereas HLearn has linear run time.

HLearn certainly isn’t going to replace Weka any time soon, but it’s got a number of cool tricks like this going on inside.  If you want to read more, you should check out these two recent papers:

• (ICML13) Algebraic Classifiers: a generic approach to fast cross-validation, online training, and parallel training

• (TFP13) HLearn: a machine learning library for Haskell

I’ll continue to write more about these tricks in future blog posts.

Subscribe to the RSS feed to stay tuned.

This fully automatic AK-47 was used by the Romanian army during the Cold War.  It could shoot 600 rounds per minute with an effective range of 400 yards.  It was an instrument of death; but now, it is an instrument of life.  It has been redeemed.

You see, 2500 years ago the prophet Isaiah wrote:

Nations will beat their swords into plowshares and their spears into pruning hooks.
Nation will not take up sword against nation, nor will they train for war anymore.

I want Isaiah’s vision to become reality.

the rifle

This is the rifle fully disassembled.

This is a closeup of the the barrel assembly that I actually made into a spoon.  I’m still looking for ideas on what to make out of everything else.

Here’s a closeup of the end where the bullet enters the barrel.  On top is the “rear sights.”  This is adjustable to shoot at targets anywhere from 0-800 meters away.

Notice that there’s actually many pieces of metal here—the barrel itself and two large blocks of steel attached to it.  These blocks are held in place by “pinions.”  These are the circle shaped pieces.  If I had a hydraulic press I could push out the pinions and then remove the big metal blocks.  But I don’t, so I’ll just beat them off with my forging hammer!

Looking down the barrel.  This is where we would insert the bullet when firing.

Now for the business end.  Here I have the flash suppressor removed.  The forward sight (the tall thing jutting out to the right) will make a great handle for the future ladle.

Building the Forge

In order to turn this chunk of metal into a spoon, I needed to build a forge.  I bought a basic anvil and 5 lbs hammer on the internet for about $80, and the stump came from craigslist for free:

Next, I needed a way to heat the metal.  I decided to build a propane forge, because I already had a good burner.  The burner came out of a portable stove that we’ve used to serve free chili with a group called Food not Bombs.  The base of the stove is below the bricks, but I unscrewed the burner and placed it on the bricks.

Then, I simply stacked the bricks on top of each other to create a nice chamber for the flames.  Here’s a picture of a test run with a piece of rebar.

The flame is actually gigantic, shooting 1-2 feet past the opening of the forge.  The middle of the rebar is a bright glowing orange-yellow, over 2000 degrees Fahrenheit.  My camera just can’t do it justice.

The bricks are also really cheap.  I payed 25 cents each at Lowes.  Overall, the whole set up cost less than $100.

Forging the metal

Let the hammering begin!  You have to work quickly to hit the metal while it’s still hot from the forge!

Oooohhh…. glowing……  purty…..

After just a few hammer blows, look how much the blocks have shrunk.  Also, I did NOT have a flash for this.  The metal is so hot it’s putting off enough light to light up the wall I’m holding it next to!

I had to stop hammering after about an hour because of blisters.  This is what I was starting with on the next day.  Notice that you can still make out the serial number!

And here’s a view from the bottom up.

After a few more hours hammering, everything’s much flatter.  The serial number has long since been flattened away.

Here’s the same view from the bottom.  It looks like a ghost!

The layers of metal from the attached blocks are quite distinct and starting to get in the way.

After a lot of wrestling with some pliers, I’ve finally managed to remove the blocks of metal.  All that remains is this “shrapnel.”

And the gun barrel itself is now the world’s coolest spatula.

And from the bottom:

Rounding out the bowl

All that’s left is to turn this spatula into a spoon.  The metal is still pretty thick, so I flattened it out as much as I could.

Prelimiaries: Creating the data Types

As usual, this post is a literate haskell file.  To run this code, you’ll need to install the hlearn-distributions package.  This package requires GHC version at least 7.6.

bash> cabal install hlearn-distributions-1.0.0.1

Now for some code.  We start with our language extensions and imports:

>{-# LANGUAGE DataKinds #-}
>{-# LANGUAGE TypeFamilies #-}
>{-# LANGUAGE TemplateHaskell #-}
>
>import HLearn.Algebra
>import HLearn.Models.Distributions

Next, we’ll create data type to represent Futurama characters.  There are a lot of characters, so we’ll need to keep things pretty organized.  The data type will have a record for everything we might want to know about a character.  Each of these records will be one of the variables in our multivariate distribution, and all of our data points will have this type.

>data Character = Character
>   { _name      :: String
>   , _species   :: String
>   , _job       :: Job
>   , _isGood    :: Maybe Bool
>   , _age       :: Double -- in years
>   , _height    :: Double -- in feet
>   , _weight    :: Double -- in pounds
>   }
>   deriving (Read,Show,Eq,Ord)
>
>data Job = Manager | Crew | Henchman | Other
>   deriving (Read,Show,Eq,Ord)

Now, in order for our library to be able to interpret the Character type, we call the template haskell function:

>makeTypeLenses ''Character

This function creates a bunch of data types and type classes for us.  These “type lenses” give us a type-safe way to reference the different variables in our multivariate distribution.  We’ll see how to use these type level lenses a bit later.  There’s no need to understand what’s going on under the hood, but if you’re curious then checkout the hackage documentation or source code.

Training a distribution

Now, we’re ready to create a data set and start training.  Here’s a list of the employees of Planet Express provided by the resident bureaucrat Hermes Conrad.  This list will be our first data set.

>planetExpress = 
>   [ Character "Philip J. Fry"         "human" Crew     (Just True) 1026   5.8 195
>   , Character "Turanga Leela"         "alien" Crew     (Just True) 43     5.9 170
>   , Character "Professor Farnsworth"  "human" Manager  (Just True) 85     5.5 160
>   , Character "Hermes Conrad"         "human" Manager  (Just True) 36     5.3 210
>   , Character "Amy Wong"              "human" Other    (Just True) 21     5.4 140
>   , Character "Zoidberg"              "alien" Other    (Just True) 212    5.8 225
>   , Character "Cubert Farnsworth"     "human" Other    (Just True) 8      4.3 135
>   ]

Let’s train a distribution from this data.  Here’s how we would train a distribution where every variable is independent of every other variable:

>dist1 = train planetExpress :: Multivariate Character
>  '[ Independent Categorical '[String,String,Job,Maybe Bool]
>   , Independent Normal '[Double,Double,Double]
>   ]
>   Double

In the HLearn library, we always use the function train to train a model from data points.  We specify which model to train in the type signature.

As you can see, the Multivariate distribution takes three type parameters.  The first parameter is the type of our data point, in this case Character.  The second parameter describes the dependency structure of our distribution.  We’ll go over the syntax for the dependency structure in a bit.  For now, just notice that it’s a type-level list of distributions.  Finally, the third parameter is the type we will use to store our probabilities.

What can we do with this distribution?  One simple task we can do is to find marginal distributions.  The marginal distribution is the distribution of a certain variable ignoring all the other variables.  For example, let’s say I want a distribution of the species that work at planet express.  I can get this by:

>dist1a = getMargin TH_species dist1

Notice that we specified which variable we’re taking the marginal of by using the type level lens TH_species.  This data constructor was automatically created for us by out template haskell function makeTypeLenses.  Every one of our records in the data type has its own unique type lens.  It’s name is the name of the record, prefixed by TH.  These lenses let us infer the types of our marginal distributions at compile time, rather than at run time.  For example, the type of the marginal distribution of species is:

ghci> :t dist1a
dist1a :: Categorical String Double

That is, a categorical distributions whose data points are Strings and which stores probabilities as a Double.  Now, if I wanted a distribution of the weights of the employees, I can get that by:

>dist1b = getMargin TH_weight dist1

And the type of this distribution is:

ghci> :t dist1b
dist1b :: Normal Double

Now, I can easily plot these marginal distributions with the plotDistribution function:

ghci> plotDistribution (plotFile "dist1a") dist1a
ghci> plotDistribution (plotFile "dist1b") dist1b

In a traditional statistics library, we would have to retrain our data from scratch.  If we had billions of elements in our data set, this would be an expensive mistake.  But in our HLearn library, we can take advantage of the model’s monoid structure.  In particular, the compiler used this structure to automatically derive a function called add1dp for us.  Let’s look at its type:

ghci> :t add1dp
add1dp :: HomTrainer model => model -> Datapoint model -> model

It’s pretty simple.  The function takes a model and adds the data point associated with that model.  It returns the model we would have gotten if the data point had been in our original data set.  This is called online training.

Again, because our distributions form monoids, the compiler derived an efficient and exact online training algorithm for us automatically.

So let’s create a new distribution that considers bender:

>bender = Character "Bender Rodriguez" "robot" Crew (Just True) 44 6.1 612
>dist1' = add1dp dist1 bender

And plot our new marginals:

ghci> plotDistribution (plotFile "dist1-withbender-species" $ PNG 250 250) $ 
                getMargin TH_species dist1'
ghci> plotDistribution (plotFile "dist1-withbender-weight"  $ PNG 250 250) $ 
                getMargin TH_weight dist1'

ghci> mean dist1b
176.42857142857142
ghci> mean $ getMargin TH_weight dist1'
230.875

Bender’s weight really changed the distribution after all!

Complicated DependencE structureS

That’s cool, but our original distribution isn’t very interesting.  What makes multivariate distributions interesting is when the variables affect each other.  This is true in our case, so we’d like to be able to model it.  For example, we’ve already seen that robots are much heavier than organic lifeforms, and are throwing off our statistics.  The HLearn library supports a small subset of Markov Networks for expressing these dependencies.

We represent Markov Networks as graphs with undirected edges.  Every attribute in our distribution is a node, and every dependence between attributes is an edge.  We can draw this graph with the plotNetwork command:

ghci> plotNetwork "dist1-network" dist1

As expected, there are no edges in our graph because everything is independent.  Let’s create a more interesting distribution and plot its Markov network.

>dist2 = train planetExpress :: Multivariate Character
>  '[ Ignore                  '[String]
>   , MultiCategorical        '[String]
>   , Independent Categorical '[Job,Maybe Bool]
>   , Independent Normal      '[Double,Double,Double]
>   ]
>   Double

ghci> plotNetwork "dist2-network" dist2

Okay, so what just happened?

The syntax for representing the dependence structure is a little confusing, so let’s go step by step.  We represent the dependence information in the graph as a list of types.  Each element in the list describes both the marginal distribution and the dependence structure for one or more records in our data type.  We must list these elements in the same order as the original data type.

Notice that we’ve made two changes to the list.  First, our list now starts with the type Ignore ‘[String].  This means that the first string in our data type—the name—will be ignored.  Notice that TH_name is no longer in the Markov Network.  This makes sense because we expect that a character’s name should not tell us too much about any of their other attributes.

Second, we’ve added a dependence.  The MultiCategorical distribution makes everything afterward in the list dependent on that item, but not the things before it.  This means that the exact types of dependencies it can specify are dependent on the order of the records in our data type.  Let’s see what happens if we change the location of the MultiCategorical:

>dist3 = train planetExpress :: Multivariate Character
>  '[ Ignore '[String]
>   , Independent Categorical '[String]
>   , MultiCategorical '[Job]
>   , Independent Categorical '[Maybe Bool]
>   , Independent Normal '[Double,Double,Double]
>   ]
>   Double

ghci> plotNetwork "dist3-network" dist3

As you can see, our species no longer have any relation to anything else.  Unfortunately, using this syntax, the order of list elements is important, and so the order we specify our data records is important.

Finally, we can substitute any valid univariate distribution for our Normal and Categorical distributions.  The HLearn library currently supports Binomial, Exponential, Geometric, LogNormal, and Poisson distributions.  These just don’t make much sense for modelling Futurama characters, so we’re not using them.

Now, we might be tempted to specify that every variable is fully dependent on every other variable.  In order to do this, we have to introduce the “Dependent” type.  Any valid multivariate distribution can follow Dependent, but only those records specified in the type-list will actually be dependent on each other.  For example:

>dist4 = train planetExpress :: Multivariate Character
>  '[ Ignore '[String]
>   , MultiCategorical '[String,Job,Maybe Bool]
>   , Dependent MultiNormal '[Double,Double,Double]
>   ]
>   Double

ghci> plotNetwork "dist4-network" dist4

Undoubtably, this is in always going to be the case—everything always has a slight influence on everything else.  Unfortunately, it is not easy in practice to model these fully dependent distributions.  We need roughly data points to accurately train a distribution, where n is the number of nodes in our graph and e is the number of edges in our network.  Thus, by selecting that two attributes are independent of each other, we can greatly reduce the amount of data we need to train an accurate distribution.

I realize that this syntax is a little awkward.  I chose it because it was relatively easy to implement.  Future versions of the library should support a more intuitive syntax.  I also plan to use copulas to greatly expand the expressiveness of these distributions.  In the mean time, the best way to figure out the dependencies in a Markov Network are just to plot it and see visually.

Okay.  So what distribution makes the most sense for Futurama characters?  We’ll say that everything depends on both the characters’ species and job, and that their weight depends on their height.

>planetExpress = train planetExpress :: Multivariate Character
>  '[ Ignore '[String]
>   , MultiCategorical '[String,Job]
>   , Independent Categorical '[Maybe Bool]
>   , Independent Normal '[Double]
>   , Dependent MultiNormal '[Double,Double]
>   ]
>   Double

ghci> plotNetwork "planetExpress-network" planetExpress

We still don’t have enough data to to train this network, so let’s create some more.  We start by creating a type for our Markov network called FuturamaDist.  This is just for convenience so we don’t have to retype the dependence structure many times.

>type FuturamaDist = Multivariate Character
>  '[ Ignore '[String]
>   , MultiCategorical '[String,Job]
>   , Independent Categorical '[Maybe Bool]
>   , Independent Normal '[Double]
>   , Dependent MultiNormal '[Double,Double]
>   ]
>   Double

Next, we train some more distribubtions of this type on some of the characters.  We’ll start with Mom Corporation and the brave Space Forces.

  

>momCorporation = 
>   [ Character "Mom"                   "human" Manager  (Just False) 100 5.5 130
>   , Character "Walt"                  "human" Henchman (Just False) 22  6.1 170
>   , Character "Larry"                 "human" Henchman (Just False) 18  5.9 180
>   , Character "Igner"                 "human" Henchman (Just False) 15  5.8 175
>   ]
>momDist = train momCorporation :: FuturamaDist

>spaceForce = 
>   [ Character "Zapp Brannigan"        "human" Manager  (Nothing)   45  6.0 230
>   , Character "Kif Kroker"            "alien" Crew     (Just True) 113 4.5 120
>   ]
>spaceDist = train spaceForce :: FuturamaDist

And now some more robots:

 

>robots = 
>   [ bender
>   , Character "Calculon"              "robot" Other    (Nothing)    123  6.8 650
>   , Character "The Crushinator"       "robot" Other    (Nothing)    45   8.0 4500
>   , Character "Clamps"                "robot" Henchman (Just False) 134  5.8 330
>   , Character "DonBot"                "robot" Manager  (Just False) 178  5.8 520
>   , Character "Hedonismbot"           "robot" Other    (Just False) 69   4.3 1200
>   , Character "Preacherbot"           "robot" Manager  (Nothing)    45   5.8 350
>   , Character "Roberto"               "robot" Other    (Just False) 77   5.9 250
>   , Character "Robot Devil"           "robot" Other    (Just False) 895  6.0 280
>   , Character "Robot Santa"           "robot" Other    (Just False) 488  6.3 950
>   ]
>robotDist = train robots :: FuturamaDist

Now we’re going to take advantage of the monoid structure of our multivariate distributions to combine all of these distributions into one.

> futuramaDist = dist1 <> momDist <> spaceDist <> robotDist

The resulting distribution is equivalent to having trained a distribution from scratch on all of the data points:

train (planetExpress++momCorporation++spaceForces++robots) :: FuturamaDist

parallel train (planetExpress++momCorporation++spaceForces++robots) :: FuturamaDist

ghci> plotDistribution (plotFile "goodguy" $ PNG 250 250) $ getMargin TH_isGood futuramaDist

But what if I only want to pick from those characters that are humans, or those characters that are robots?  Statisticians call this conditioning.  We can do that with the condition function:

ghci> plotDistribution (plotFile "goodguy-human" $ PNG 250 250) $
             getMargin TH_isGood $ condition TH_species "human" futuramaDist
ghci> plotDistribution (plotFile "goodguy-robot" $ PNG 250 250) $
             getMargin TH_isGood $ condition TH_species "robot" futuramaDist

Now let’s ask: What’s the average age of an evil robot?

ghci> mean $ getMargin TH_age $ 
         condition TH_isGood (Just False) $ condition TH_species "robot" futuramaDist 
273.0769230769231

Notice that conditioning a distribution is a commutative operation.  That means we can condition in any order and still get the exact same results.  Let’s try it:

ghci> mean $ getMargin TH_age $ 
         condition TH_species "robot" $ condition TH_isGood (Just False) futuramaDist 
273.0769230769231

There’s one last thing for us to consider.  What does our Markov network look like after conditioning?  Let’s find out!

plotNetwork "condition-species-isGood" $ 
         condition TH_species "robot" $ condition TH_isGood (Just False) futuramaDist

Notice that conditioning against these variables caused them to go away from our Markov Network.

Finally, there’s another similar process to conditioning called “marginalizing out.” This lets us ignore the effects of a single attribute without specifically saying what that attribute must be. When we marginalize out on our Markov network, we get the same dependence structure as if we conditioned.

plotNetwork "marginalizeOut-species-isGood" $ 
         marginalizeOut TH_species $ marginalizeOut TH_isGood futuramaDist

Effectively, what the marginalizeOut function does is “forget” the extra dependencies, whereas the condition function “applies” those dependencies.  In the end, the resulting Markov network has the same structure, but different values.

Finally, at the start of the post, I mentioned that our multivariate distributions have group and vector space structure.  This gives us two more operations we can use: the inverse and scalar multiplication.  You can find more posts on how to take advantage of these structures here and here.

Next time…

The best part of all of this is still coming.  Next, we’ll take a look at full on Bayesian classification and why it forms a monoid.  Besides online and parallel trainers, this also gives us a fast cross-validation method.

There’ll also be a posts about the monoid structure of Markov chains, the Free HomTrainer, and how this whole algebraic framework applies to NP-approximation algorithms as well.

Subscribe to the RSS feed to stay tuned.

You see, in the United States, roughly half of our tax dollars go to financing war.  (You can find a detailed breakdown here.)  This is ridiculous and unacceptable.  I would gladly pay more taxes to finance roads, schools, or public health care.  But I will no longer pay other people to kill America’s enemies on my behalf.

I deeply regret the need for tax resistance because it contradicts a number of Biblical commands.  For example, in Romans 13:7 Paul tell us that “if you owe taxes, pay taxes” and in Mathew 22:21 Jesus commands us to “give unto Caesar the things that are Caesar’s.”  I wish I could obey these commands at face value.  But obeying the commands to pay taxes would result in me breaking the greatest commandment of them all: to love my neighbor as myself.  Jesus calls everyone my neighbor, even my enemies.  Even people who kill Americans, like Osama bin Laden.  I’m deeply ashamed that my tax dollars helped finance his assassination.  Not to mention the near-daily drone strikes that continue to happen, the torture at gitmo, and the DOD’s research into newer and deadlier weapons systems.  I payed for it all.

I could say a lot more about why I feel morally compelled to not pay war taxes, but I won’t.  I’ll skip right to the part where I’m making a public statement that I will not finance war, and I will accept whatever consequences that entails.  I also acknowledge that by taking this stand, I am sinning.  But this is the least sinful option my limited wisdom can find.  So I will continue on, “sinning boldly” as ever.

Below I describe the exact mechanics of how I’m refusing to pay war taxes.  I’m following advice provided mainly by the War Resistor League’s War Tax Resistance book.

How I’m Resisting

Today I filed my taxes just like everyone else.  I filled out my form 1040, and found out that I owed 48 dollars.  It’s not very much, but it’s something.  I did my best to be as honest and complete as possible in the paperwork.  But instead of including a check, I wrote them the following letter:

To whom it may concern:

After careful consideration, I have decided not to pay my 2012 taxes to the Federal government. I cannot in good conscience provide any financial support for our ongoing wars and excessive military spending.

I do, however, want to be a good citizen and contribute my fair share to society. Therefore, I am paying the taxes I owe to the federal government ($48) to my local state government (CA) instead. I have scanned a copy of my contribution check below.

Sincerely,

Michael Izbicki

I was happy to do my California taxes in addition to giving them this extra money.  It’s only war that I’m against, not taxes in general.  Here’s a copy of the actual check I wrote:

Also, for anyone interested, I’ve posted my form 1040:

Finally, just before mailing my envelope, I said the St Francis prayer:

Lord, make me an instrument of your peace.  Where there is hatred, let me sow love; where there is injury, pardon; where there is doubt, faith; where there is despair, hope; where there is darkness, light; and where there is sadness, joy.

O Divine Master, grant that I may not so much seek to be consoled, as to console; to be understood, as to understand; to be loved, as to love.  For it is in giving that we receive; it is in pardoning that we are pardoned, and it is in dying that we are born to Eternal Life.

In this article, we’ll go over the math behind the categorical distribution, the algebraic structure of the distribution, and how to manipulate it within Haskell’s HLearn library.  We’ll also see some examples of how this focus on algebra makes HLearn’s interface more powerful than other common statistical packages.  Everything that we’re going to see is in a certain sense very “obvious” to a statistician, but this algebraic framework also makes it convenient.  And since programmers are inherently lazy, this is a Very Good Thing.

Before delving into the “cool stuff,” we have to look at some of the mechanics of the HLearn library.

Preliminaries

The HLearn-distributions package contains all the functions we need to manipulate categorical distributions. Let’s install it:

$ cabal install HLearn-distributions

We import our libraries:

>import Control.DeepSeq
>import HLearn.Algebra
>import HLearn.Gnuplot.Distributions
>import HLearn.Models.Distributions

We create a data type for Simon’s marbles:

>data Marble = Red | Green | Blue | White
>    deriving (Read,Show,Eq,Ord)

The easiest way to represent Simon’s bag of marbles is with a list:

>simonBag :: [Marble]
>simonBag = [Red, Red, Red, Green, Blue, Green, Red, Blue, Green, Green, Red, Red, Blue, Red, Red, Red, White]

And now we’re ready to train a categorical distribution of the marbles in Simon’s bag:

>simonDist = train simonBag :: Categorical Marble Double

We can load up ghci and plot the distribution with the conveniently named function plotDistribution:

ghci> plotDistribution (plotFile "simonDist") simonDist

This gives us a histogram of probabilities:

In the HLearn library, every statistical model is generated from data using either train or train’.  Because these functions are overloaded, we must specify the type of simonDist so that the compiler knows which model to generate. Categorical takes two parameters. The first is the type of the discrete data (Marble). The second is the type of the probability (Double). We could easily create Categorical distributions with different types depending on the requirements for our application. For example:

>stringDist = train (map show simonBag) :: Categorical String Float

This is the first “cool thing” about Categorical:  We can make distributions over any user-defined type.  This makes programming with probabilities easier, more intuitive, and more convenient.  Most other statistical libraries would require you to assign numbers corresponding to each color of marble, and then create a distribution over those numbers.

Now that we have a distribution, we can find some probabilities. If Simon pulls a marble from the bag, what’s the probability that it would Red?

We can use the pdf function to do this calculation for us:

ghci> pdf simonDist Red
0.5626
ghci> pdf simonDist Blue
0.1876
ghci> pdf simonDist Green
0.1876
ghci> pdf simonDist White
6.26e-2

If we sum all the probabilities, as expected we would get 1:

ghci> sum $ map (pdf simonDist) [Red,Green,Blue,White]
1.0

Due to rounding errors, you may not always get 1. If you absolutely, positively, have to avoid rounding errors, you should use Rational probabilities:

>simonDistRational = train simonBag :: Categorical Marble Rational

Rationals are slower, but won’t be subject to floating point errors.

This is just about all the functionality you would get in a “normal” stats package like R or NumPy. But using Haskell’s nice support for algebra, we can get some extra cool features.

Semigroup

First, let’s talk about semigroups. A semigroup is any data structure that has a binary operation (<>) that joins two of those data structures together. The categorical distribution is a semigroup.

Don wants to play marbles with Simon, and he has his own bag. Don’s bag contains only red and blue marbles:

>donBag = [Red,Blue,Red,Blue,Red,Blue,Blue,Red,Blue,Blue]

We can train a categorical distribution on Don’s bag in the same way we did earlier:

>donDist = train donBag :: Categorical Marble Double

In order to play marbles together, Don and Simon will have to add their bags together.

>bothBag = simonBag ++ donBag

Now, we have two options for training our distribution. First is the naive way, we can train the distribution directly on the combined bag:

>bothDist = train bothBag :: Categorical Marble Double

This is the way we would have to approach this problem in most statistical libraries. But with HLearn, we have a more efficient alternative. We can combine the trained distributions using the semigroup operation:

>bothDist' = simonDist <> donDist

Under the hood, the categorical distribution stores the number of times each possibility occurred in the training data.  The <> operator just adds the corresponding counts from each distribution together:

This method is more efficient because it avoids repeating work we’ve already done. Categorical’s semigroup operation runs in time O(1), so no matter how big the bags are, we can calculate the distribution very quickly. The naive method, in contrast, requires time O(n). If our bags had millions or billions of marbles inside them, this would be a considerable savings!

We get another cool performance trick “for free” based on the fact that Categorical is a semigroup: The function train can be automatically parallelized using the higher order function parallel. I won’t go into the details about how this works, but here’s how you do it in practice.

First, we must show the compiler how to resolve the Marble data type down to “normal form.” This basically means we must show the compiler how to fully compute the data type. (We only have to do this because Marble is a type we created.  If we were using a built in type, like a String, we could skip this step.) This is fairly easy for a type as simple as Marble:

>instance NFData Marble where
>    rnf Red   = ()
>    rnf Blue  = ()
>    rnf Green = ()
>    rnf White = ()

Then, we can perform the parallel computation by:

>simonDist_par = parallel train simonBag :: Categorical Marble Double

Other languages require a programmer to manually create parallel versions of their functions. But in Haskell with the HLearn library, we get these parallel versions for free! All we have to do is ask for it!

Monoid

A monoid is a semigroup with an empty element, which is called mempty in Haskell. It obeys the law that:

M <> mempty == mempty <> M == M

And it is easy to show that Categorical is also a monoid. We get this empty element by training on an empty data set:

mempty = train ([] :: [Marble]) :: Categorical Marble Double

The HomTrainer type class requires that all its instances also be instances of Monoid. This lets the compiler automatically derive “online trainers” for us. An online trainer can add new data points to our statistical model without retraining it from scratch.

For example, we could use the function add1dp (stands for: add one data point) to add another white marble into Simon’s bag:

>simonDistWhite = add1dp simonDist White

This also gives us another approach for our earlier problem of combining Simon and Don’s bags. We could use the function addBatch:

>bothDist'' = addBatch simonDist donBag

Because Categorical is a monoid, we maintain the property that:

bothDist == bothDist' == bothDist''

Again, statisticians have always known that you could add new points into a categorical distribution without training from scratch.  The cool thing here is that the compiler is deriving all of these functions for us, and it’s giving us a consistent interface for use with different data structures.  All we had to do to get these benefits was tell the compiler that Categorical is a monoid.  This makes designing and programming libraries much easier, quicker, and less error prone.

Group

A group is a monoid with the additional property that all elements have an inverse. This lets us perform subtraction on groups.  And Categorical is a group.

Ed wants to play marbles too, but he doesn’t have any of his own. So Simon offers to give Ed some of from his own bag. He gives Ed one of each color:

>edBag = [Red,Green,Blue,White]

Now, if Simon draws a marble from his bag, what’s the probability it will be blue?

To answer this question without algebra, we’d have to go back to the original data set, remove the marbles Simon gave Ed, then retrain the distribution. This is awkward and computationally expensive. But if we take advantage of Categorical’s group structure, we can just subtract directly from the distribution itself. This makes more sense intuitively and is easier computationally.

>simonDist2 = subBatch simonDist edBag

This is a shorthand notation for using the group operations directly:

>edDist = train edBag :: Categorical Marble Double
>simonDist2' = simonDist <> (inverse edDist)

The way the inverse operation works is it multiplies the counts for each category by -1. In picture form, this flips the distribution upside down:

Then, adding an upside down distribution to a normal one is just subtracting the histogram columns and renormalizing:

Notice that the green bar in edDist looks really big—much bigger than the green bar in simonDist.  But when we subtract it away from simonDist, we still have some green marbles left over in simonDist2.  This is because the histogram is only showing the probability of a green marble, and not the actual number of marbles.

Finally, there’s one more crazy trick we can perform with the Categorical group.  It’s perfectly okay to have both positive and negative marbles in the same distribution.  For example:

ghci> plotDistribution (plotFile "mixedDist") (edDist <> (inverse donDist))

results in:

Most statisticians would probably say that these upside down Categoricals are not “real distributions.” But at the very least, they are a convenient mathematical trick that makes working with distributions much more pleasant.

Module

Finally, an R-Module is a group with two additional properties. First, it is abelian. That means <> is commutative. So, for all a, b:

a <> b == b <> a

Second, the data type supports multiplication by any element in the ring R. In Haskell, you can think of a ring as any member of the Num type class.

How is this useful?  It let’s “retrain” our distribution on the data points it has already seen.  Back to the example…

Well, Ed—being the clever guy that he is—recently developed a marble copying machine. That’s right! You just stick some marbles in on one end, and on the other end out pop 10 exact duplicates. Ed’s not just clever, but pretty nice too. He duplicates his new marbles and gives all of them back to Simon. What’s Simon’s new distribution look like?

Again, the naive way to answer this question would be to retrain from scratch:

>duplicateBag = simonBag ++ (concat $ replicate 10 edBag)
>duplicateDist = train duplicateBag :: Categorical Marble Double

Slightly better is to take advantage of the Semigroup property, and just apply that over and over again:

>duplicateDist' = simonDist2 <> (foldl1 (<>) $ replicate 10 edDist)

But even better is to take advantage of the fact that Categorical is a module and the (.*) operator:

>duplicateDist'' = simonDist2 <> 10 .* edDist

In picture form:

Also notice that without the scalar multiplication, we would get back our original distribution:

Another way to think about the module’s scalar multiplication is that it allows us to weight our distributions.

Ed just realized that he still needs a marble, and has decided to take one.  Someone has left their Marble bag sitting nearby, but he’s not sure whose it is.  He thinks that Simon is more forgetful than Don is, so he assigns a 60% probability that the bag is Simon’s and a 40% probability that it is Don’s.  When he takes a marble, what’s the probability that it is red?

We create a weighted distribution using module multiplication:

>weightedDist = 0.6 .* simonDist <> 0.4 .* donDist

Then in ghci:

ghci> pdf weightedDist Red
0.4929577464788732

We can also train directly on weighted data:

>weightedDataDist = train [(0.4,Red),(0.5,Green),(0.2,Green),(3.7,White)] :: Categorical Marble Double

which gives us:

The Takeaway and next posts

Talking about the categorical distribution in algebraic terms let’s us do some cool new stuff with our distributions that we can’t easily do in other libraries.  None of this is statistically ground breaking. The cool thing is that algebra just makes everything so convenient to work with.

I think I’ll do another post on some cool tricks with the kernel density estimator that are not possible at all in other libraries, then do a post about the category (formal category-theoretic sense) of statistical training methods.  At that point, we’ll be ready to jump into machine learning tasks.  Depending on my mood we might take a pit stop to discuss the computational aspects of free groups and modules and how these relate to machine learning applications.

Sign up for the RSS feed to stay tuned!

Before we get into the math, we’ll need to review the basics of nuclear politics.

The nuclear Non-Proliferation Treaty (NPT) is the main treaty governing nuclear weapons. Basically, it says that there are five countries that are “allowed” to have nukes: the USA, UK, France, Russia, and China. “Allowed” is in quotes because the treaty specifies that these countries must eventually get rid of their nuclear weapons at some future, unspecified date. When another country, for example Iran, signs the NPT, they are agreeing to not develop nuclear weapons. What they get in exchange is help from the 5 nuclear weapons states in developing their own civilian nuclear power programs. (Iran has the legitimate complaint that Western countries are actively trying to stop its civilian nuclear program when they’re supposed to be helping it, but that’s a whole ‘nother can of worms.)

The Nuclear Notebook tracks the nuclear capabilities of all these countries.  The most-current estimates are from mid-2012.  Here’s a summary (click the warhead type for more info):

I’ve consolidated all this data into the file nukes-list.csv, which we will analyze in this post.  If you want to try out this code for yourself (or the homework question at the end), you’ll need to download it.  Every line in the file corresponds to a single nuclear warhead, not delivery method.  Warheads are the parts that go boom!  Bombers, ICBMs, and SSBN/SLBMs are the delivery method.

There are three things to note about this data.  First, it’s only estimates based on public sources.  In particular, it probably overestimates the Russian nuclear forces. Other estimates are considerably lower.  Second, we will only be considering deployed, strategic warheads.  Basically, this means the “really big nukes that are currently aimed at another country.”  There are thousands more tactical warheads, and warheads in reserve stockpiles waiting to be disassembled.  For simplicity—and because these nukes don’t significantly affect strategic planning—we won’t be considering them here.   Finally, there are 4 countries who are not members of the NPT but have nuclear weapons: Israel, Pakistan, India, and North Korea.  We will be ignoring them here because their inventories are relatively small, and most of their weapons would not be considered strategic.

Programming preliminaries

Now we’re ready to start programming. First, let’s import our libraries:

>import Control.Lens
>import Data.Csv
>import qualified Data.Vector as V
>import qualified Data.ByteString.Lazy.Char8  as BS
> 
>import HLearn.Algebra
>import HLearn.Models.Distributions
>import HLearn.Gnuplot.Distributions

Next, we load our data using the Cassava package.  (You don’t need to understand how this works.)

>main = do
>    Right rawdata <- fmap (fmap V.toList . decode True) $ BS.readFile "nukes-list.csv"
>        :: IO (Either String [(String, String, String, Int)])

And we’ll use the Lens package to parse the CSV file into a series of variables containing just the values we want.  (You also don’t need to understand this.)

>   let list_usa    = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="USA"   ) rawdata
>   let list_uk     = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="UK"    ) rawdata 
>   let list_france = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="France") rawdata 
>   let list_russia = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="Russia") rawdata 
>   let list_china  = fmap (\row -> row^._4) $ filter (\row -> (row^._1)=="China" ) rawdata

NOTE: All you need to understand about the above code is what these list_country variables look like. So let’s print one:

>   putStrLn $ "List of American nuclear weapon sizes = " ++ show list_usa

gives us the output:

List of American nuclear weapon sizes = fromList [335,335,335,335,335,335,335,335,335,335  ...  1200,1200,1200,1200,1200]

If we want to know how many weapons are in the American arsenal, we can take the length of the list:

>   putStrLn $ "Number of American weapons = " ++ show (length list_usa)

We get that there are 1951 American deployed, strategic nuclear weapons.  If we want to know the total “blowing up” power, we take the sum of the list:

>   putStrLn $ "Explosive power of American weapons = " ++ show (sum list_usa)

We get that the US has  516 megatons of deployed, strategic nuclear weapons.  That’s the equivalent of 1,033,870,000,000 pounds of TNT.

To get the total number of weapons in the world, we concatenate every country’s list of weapons and find the length:

>   let list_all = list_usa ++ list_uk ++ list_france ++ list_russia ++ list_china
>   putStrLn $ "Number of nukes in the whole world = " ++ show (length list_all)

Doing this for every country gives us the table:

Now let’s do some algebra!

Monoids and groups

In a previous post, we saw that the Gaussian distribution forms a group. This means that it has all the properties of a monoid—an empty element (mempty) that represents the distribution trained on no data, and a binary operation (mappend) that merges two distributions together—plus an inverse. This inverse lets us “subtract” two Gaussians from each other.

It turns out that many other distributions also have this group property. For example, the categorical distribution.  This distribution is used for measuring discrete data. Essentially, it assigns some probability to each “label.”  In our case, the labels are the size of the nuclear weapon, and the probability is the chance that a randomly chosen nuke will be exactly that destructive.  We train our categorical distribution using the train function:

> let cat_usa = train list_usa :: Categorical Int Double

If we plot this distribution, we’ll get a graph that looks something like:

A distribution like this is useful to war planners from other countries.  It can help them statistically determine the amount of casualties their infrastructure will take from a nuclear exchange.

Now, let’s train equivalent distributions for our other countries.

> let cat_uk = train list_uk :: Categorical Int Double
> let cat_france = train list_france :: Categorical Int Double
> let cat_russia = train list_russia :: Categorical Int Double
> let cat_china = train list_china :: Categorical Int Double

Because training the categorical distribution is a group homomorphism, we can train a distribution over all nukes by either training directly on the data:

>   let cat_allA = train list_all :: Categorical Int Double

or we can merge the already generated categorical distributions:

>   let cat_allB = cat_usa <> cat_uk <> cat_france <> cat_russia <> cat_china

Because of the homomorphism property, we will get the same result both ways. Since we’ve already done the calculations for each of the the countries already, method B will be more efficient—it won’t have to repeat work we’ve already done.  If we plot either of these distributions, we get:

The thing to notice in this plot is that most countries have a nuclear arsenal that is distributed similarly to the United States—except for China.  These Chinese ICBMs will become much more important when we discuss nuclear strategy in the last section.

But nuclear war planners don’t particularly care about this complete list of nuclear weapons.  What war planners care about is the survivable nuclear weapons—that is, weapons that won’t be blown up by a surprise nuclear attack.  Our distributions above contain nukes dropped from bombers, but these are not survivable.  They are easy to destroy.  For our purposes, we’ll call anything that’s not a bomber a survivable weapon.

We’ll use the group property of the categorical distribution to calculate the survivable weapons.  First, we create a distribution of just the unsurvivable bombers:

>   let list_bomber = fmap (\row -> row^._4) $ filter (\row -> (row^._2)=="Bomber") rawdata
>   let cat_bomber = train list_bomber :: Categorical Int Double

Then, we use our group inverse to subtract these unsurvivable weapons away:

>   let cat_survivable = cat_allB <> (inverse cat_bomber)

Notice that we calculated this distribution indirectly—there was no possible way to combine our variables above to generate this value without using the inverse! This is the power of groups in statistics.

More distributions

The categorical distribution is not sufficient to accurately describe the distribution of nuclear weapons. This is because we don’t actually know the yield of a given warhead. Like all things, it has some manufacturing tolerances that we must consider. For example, if we detonate a 300 kt warhead, the actual explosion might be 275 kt, 350 kt, or the bomb might even “fizzle out” and have almost a 0kt explosion.

We’ll model this by using a kernel density estimator (KDE).  The KDE basically takes all our data points, assigns each one a probability distribution called a “kernel,” then sums these kernels together.  It is a very powerful and general technique for modelling distributions… and it also happens to form a group!

First, let’s create the parameters for our KDE.  The bandwidth controls how wide each of the kernels is.  Bigger means wider.  I selected 20 because it made a reasonable looking density function.  The sample points are exactly what they sounds like: they are where we will sample the density from.  We can generate them using the function genSamplePoints.  Finally, the kernel is the shape of the distributions we will be summing up.  There are many supported kernels.

>   let kdeparams = KDEParams
>        { bandwidth    = Constant 20
>        , samplePoints = genSamplePoints
>               0       -- minimum
>               4000    -- maximum
>               4000    -- number of samples
>        , kernel       = KernelBox Gaussian
>        } :: KDEParams Double

Now, we’ll train kernel density estimates on our data.  Notice that because the KDE takes parameters, we must use the train’ function instead of just train.

>   let kde_usa     = train' kdeparams list_usa      :: KDE Double

Again, plotting just the American weapons gives:

And we train the corresponding distributions for the other countries.

>   let kde_uk      = train' kdeparams list_uk       :: KDE Double
>   let kde_france  = train' kdeparams list_france   :: KDE Double
>   let kde_russia  = train' kdeparams list_russia   :: KDE Double
>   let kde_china   = train' kdeparams list_china    :: KDE Double
>
>   let kde_all = kde_usa <> kde_uk <> kde_france <> kde_russia <> kde_china

The KDE is a powerful technique, but the draw back is that it is computationally expensive—especially when a large number of sample points are used. Fortunately, all computations in the HLearn library are easily parallelizable by applying the higher order function parallel.

We can calculate the full KDE from scratch in parallel like this:

>   let list_double_all = map fromIntegral list_all :: [Double]
>   let kde_all_parA = (parallel (train' kdeparams)) list_double_all :: KDE Double

or we can perform a parallel reduction on the KDEs for each country like this:

>   let kde_all_parB = (parallel reduce) [kde_usa, kde_uk, kde_france, kde_russia, kde_china]

And because the KDE is a homomorphism, we get the same exact thing either way.  Let’s plot the parallel version:

>   plotDistribution (genPlotParams "kde_all" kde_all_parA) kde_all_parA

The parallel computation takes about 16 seconds on my Core2 Duo laptop running on 2 processors, whereas the serial computation takes about 28 seconds.

This is a considerable speedup, but we can still do better. It turns out that there is a homomorphism from the Categorical distribution to the KDE:

>   let kde_fromcat_all = cat_allB $> kdeparams
>   plotDistribution (genPlotParams "kde_fromcat_all" kde_fromcat_all) kde_fromcat_all

(For more information about the morphism chaining operator $>, see the Hlearn documentation.) This computation takes less than a second and gets the exact same result as the much more expensive computations above.

We can express this relationship with a commutative diagram:

No matter which path we take to get to a KDE, we will get the exact same answer.  So we should always take the path that will be least computationally expensive for the data set we’re working on.

Why does this work? Well, the categorical distribution is a structure called the “free module” in disguise.

Modules and the Free Module

R-Modules (like groups, but unlike monoids) have not seen much love from functional programmers. This is a shame, because they’re quite handy. It turns out they will increase our performance dramatically in this case.

It’s not super important to know the formal definition of an R-module, but here it is anyways: An R-module is a group with an additional property: it can be “multiplied” by any element of the ring R. This is a generalization of vector spaces because R need only be a ring instead of a field. (Rings do not necessarily have multiplicative inverses.)  It’s probably easier to see what this means by an example.

Vectors are modules.  Let’s say I have a vector:

>   let vec = [1,2,3,4,5] :: [Int]

I can perform scalar multiplication on that vector like this:

>   let vec2 = 3 .* vec

which as you might expect results in:

[3,6,9,12,15]

Our next example is the free R-module. A “free” structure is one that obeys only the axioms of the structure and nothing else. Functional programmers are very familiar with the free monoid—it’s the list data type. The free Z-module is like a beefed up list. Instead of just storing the elements in a list, it also stores the number of times that element occurred.  (Z is shorthand for the set of integers, which form a ring but not a field.) This lets us greatly reduce the memory required to store a repetitive data set.

In HLearn, we represent the free module over a ring r with the data type:

:: FreeMod r a

where a is the type of elements to be stored in the free module. We can convert our lists into free modules using the function list2module like this:

>   let module_usa = list2module list_usa

But what does the free module actually look like? Let’s print it to find out:

>   print module_usa

gives us:

FreeMod (fromList [(100,768),(150,200),(300,250),(335,249),(340,50),(455,384),(1200,50)])

This is much more compact! So this is the take away: The free module makes repetitive data sets easier to work with. Now, let’s convert all our country data into module form:

>   let module_uk       = list2module list_uk
>   let module_france   = list2module list_france
>   let module_russia   = list2module list_russia
>   let module_china    = list2module list_china

Because modules are also groups, we can combine them like so:

>   let module_allA = module_usa <> module_uk <> module_france <> module_russia <> module_china

or, we could train them from scratch:

>   let module_allB = list2module list_all

Again, because generating a free module is a homomorphism, both methods are equivalent.

Module distributions

The categorical distribution and the KDE both have this module structure. This gives us two cool properties for free.

First, we can train these distributions directly from the free module.  Because the free module is potentially much more compact than a list is, this can save both memory and time. If we run:

>   let cat_module_all = train module_allB :: Categorical Int Double
>   let kde_module_all = train' kdeparams module_allB :: KDE Double

Then we get the properties:

cat_mod_all == cat_all
kde_mod_all == kde_all == kde_cat_all

Extending our commutative diagram above gives:

Again, no matter which path we take to train our KDE, we still get the same result because each of these arrows is a homomorphism.

Second, if a distribution is a module, we can weight the importance of our data points.  Let’s say we’re a general from North Korea (DPRK), and we’re planning our nuclear strategy. The US and North Korea have a very strained relationship in the nuclear department. It is much more likely that the US will try to nuke the DPRK than China will. And modules let us model this!  We can weight each country’s influence on our “nuclear threat profile” distribution like this:

>   let threats_dprk = 20 .* kde_usa
>                   <> 10 .* kde_uk
>                   <> 5  .* kde_france
>                   <> 2  .* kde_russia
>                   <> 1  .* kde_china
>
>   plotDistribution (genPlotParams "threats_dprk" threats_dprk) threats_dprk

Basically, we’re saying that the USA is 20x more likely to attack the DPRK than China is.  Graphically, our threat distribution is:

The maximum threat that we have to worry about is about 1300 kt, so we need to design all our nuclear bunkers to withstand this level of blast.  Nuclear war planners would use the above distribution to figure out how much infrastructure would survive a nuclear exchange.  To see how this is done, you’ll have to click the link.

On the other hand, if we’re an American general, then we might say that China is our biggest threat… who knows what they’ll do when we can’t pay all the debt we owe them!?

>   let threats_usa = 1 .* kde_russia 
>                  <> 5 .* kde_china
>
>   plotDistribution (genPlotParams "threats_usa" threats_usa) threats_usa

Graphically:

So now Chinese ICBMs are a real threat.  For American infrastructure to be secure, most of it needs to be able to withstand ~3500 kt blast.  (Actually, Chinese nuclear policy is called the “minimum means of reprisal”—these nukes are not targeted at military installations, but major cities.  Unlike the other nuclear powers, China doesn’t hope to win a nuclear war.  Instead, its nuclear posture is designed to prevent nuclear war in the first place.  This is why China has the fewest weapons of any of these countries.  For a detailed analysis, see the book Minimum Means of Reprisal.  This means that American military infrastructure isn’t threatened by these large Chinese nukes, and really only needs to be able to withstand an 800kt explosion to be survivable.)

By the way, since we’ve already calculated all of the kde_country variables before, these computations take virtually no time at all to compute.  Again, this is all made possible thanks to our friend abstract algebra.

Homework + next Post

If you want to try out the HLearn library for yourself, here’s a question you can try to answer: Create the DPRK and US threat distributions above, but only use survivable weapons.  Don’t include bombers in the analysis.

In our next post, we’ll go into more detail about the mathematical plumbing that makes all this possible. Then we’ll start talking about Bayesian classification and full-on machine learning. Subscribe to the RSS feed so you don’t miss out!

Why don’t you listen to Tom Lehrer’s “Song for WWIII” while you wait?

We don’t know what God wants, and we wouldn’t know how to do it even if we did.  Therefore (as Gandhi put it) we must “experiment with truth.”  We must discover truth for ourselves, and how to achieve it.

These are my experiments from 2012.  I didn’t try these experiments because they are somehow the “most Christ-like” thing to do.  I tried them because I don’t know what the most Christ-like thing is, but I want to learn.  I want to train myself to do it at all times.  Some of these experiments succeeded and some failed.  But all of them made me a better Christian.

1. Fasting: SUCCESS

This year, I decided not to eat any food on Mondays. During lent, I didn’t eat on either Monday or Thursday. I had never fasted before this. Growing up, I thought fasting was a “stupid Catholic thing.” But now I see it is immensely valuable.

From an earthly standpoint, fasting helped me exercise my willpower and discipline. From a spiritual standpoint, being hungry was a constant reminder of people in need around the world—I need to help them just like Jesus would, and I need to give thanks to God for what food and blessings I do have. Fasting also helped me grow closer to God. Every hunger pang was a reminder to say a prayer to God asking for His endurance and grace.

2. Vegetarianism: SUCCESS

From January 1st until Easter Sunday (~3 months), I didn’t eat any meat. I did this experiment for two reasons. First, I feel bad about factory farming and my involvement in the abuse of animals. Second, I want to stand in solidarity with those who choose vegetarianism for moral reasons. I want to understand what it feels like to go into a restaurant or a party and leave hungry because there’s no vegetarian food. I want to share their pain.

This experiment was much easier than I expected. I never had any cravings for meat, and learned to cook quite a variety of different foods. My new favorite food is “buffalo broccoli,” a stir fry with buffalo sauce added at the end.  It’s even better on broccoli than it is on chicken because the broccoli has so much surface area for the sauce to cling to.  It’s amazing.

Ultimately, however, I decided not to remain strictly vegetarian because it was harming my relationships. Sharing meals is an important part of friendship, and I was unable to participate in several church and family dinners because of my strict vegetarianism. So now, I try to eat “mostly vegetarian.” If I’m by myself, I won’t eat meat. If I’m at a party and there’s a vegetarian option available, I’ll eat only that. But, if I’m with other people and the only food available is meat, then I’ll eat meat with them. To me, strengthening these relationships with humans seems more important than reducing the suffering of factory farmed animals.

3. Food not Bombs: SUCCESS & FAILURE

Ever since leaving the Navy, I’ve struggled with how to express my pacifism. I want to do it in a way that promotes peace rather than just condemns war. So I helped start a Food not Bombs group on the UCR campus with a few other students. We served free vegetarian meals every Friday during the spring 2012 quarter. In total, we served 1000 meals.

Unfortunately, we were unable to continue serving food after that quarter. Some administrators decided they didn’t want Food not Bombs on campus, despite the fact that we were following all the proper regulations. All of our requests for food permits were denied for various stupid “problems,” and when we fixed those problems, the administrators found new problems.  It became clear that they would never approve a permit with our names on it, so our FNB group gradually dissolved.

One thing I learned from the experience is that I suck at organizing and motivating people behind a vision. I just don’t enjoy doing it. So in the future, rather than trying to start my own group, I’m going to help somebody else start theirs.  Finding a good peacemaking group to be a part of is my top priority for 2013.

4. Beer: SUCCESS

I grew up thinking that the way to be a good Christian was to be a good Puritan and Nationalist.  Now, I believe the way to be a good Christian is to be Christ to everyone you meet.  So since Jesus turned water into wine, I decided to try my hand at turning it into beer.  (I personally can’t stand wine.)

This has turned out better than I could ever have imagined.  Not only is making beer a ton of fun, but it really helps build relationships.  Sharing alcohol with strangers is the fastest way I know to turn them into friends.  Plus, it can lead into great conversations about God.  Mostly what I’ve been brewing is what I affectionately call “Monk Beer.”  It’s based on recipes from Trappist monasteries from Belgian.  In my experience, even the most die-hard anti-theists have good things to say about Christians after we share some monk beer together.

Oh, and the beer tastes great too.  We got 2nd place out of 550 at the Mayfaire beer competition.

5. Memorizing the sermon on the mount: FAILURE

I’ve read through the sermon more times than I can count. Many verses, like “turn the other cheek” and “go the extra mile” are burned into my heart. But I feel there is still so much more for me to learn here. I still don’t fully understand what Jesus is saying, and I’m certainly not yet living it out.  

So I decided to memorize it.  I made this goal for myself at the beginning of summer, and 6 months later I still haven’t accomplished it. I am normally good at memorizing, I just haven’t put in the time it takes to accomplish this task.  I am getting closer, however.  Finishing this task is one of my goals for 2013.

6. Respecting the Sabbath: FAILURE

I made a commitment in 2012 not to open my computer on Sunday. Instead of wasting my time working or browsing reddit, I wanted to spend this same time building relationships with members of my church and family. Growing up, I thought respecting the sabbath was a “stupid Jewish thing,” but now I realize it is immensely important.

Sadly, I didn’t keep the sabbath as well as I would have liked. Often, I would read academic papers on Sunday, grade homework, or do other things that failed to build relationships.  I did this because of poor scheduling throughout the week and misplaced priorities about what is important.

Another goal for 2013 is to change this. I want my every Sunday to be the most holy day of the year. I want to dedicate Sunday to building important relationships with God and men.

7. Walking everywhere: SUCCESS

Normally I ride my bike everywhere, but when I got a flat tire in early November I decided to try walking everywhere. Bikes are already a much slower lifestyle than cars, but walking is even slower. Every day since my tire popped until today, I did a 40 minute walk to work.  It was amazing.

As I walked to work, I got to enjoy the sights of the snowy San Gabriel mountains to the north, the Box Spring mountains to the east and Saddleback Mountain to the southwest. I got to watch and listen to the birds sing. I got to pick up trash along the side of the road, making the community just a little bit nicer, and I even got to wish the occasional homeless man a “Merry Christmas” and talk about how they’re getting along.

8. Talking with the homeless: FAILURE

Sadly, I failed to help many of the “least of these” in Riverside. Many times this year I drove passed a homeless man despite a gut wrenching conviction that I need to stop and help this person. I failed to help Christ in his greatest need, and I am ashamed.

In fall of 2011, I did a lot of talking with the homeless.  I did most of my grocery shopping at a nearby Stater Brothers, and there were always homeless men and women outside begging for money.  I would ask them to come shopping with me, buy them some food, and have a good conversation about their life and problems.  But this year my roommates and I have been shopping exclusively at Costco, and I don’t get to meet any of these homeless while going about my daily business.  I think this turned out to be a Very Bad Thing for my spiritual health.

I’m still not sure how to fix this problem for 2013.

Bonus 9th: Sharing my experiments on the internet

I’m sharing this for two reasons.  First, these experiments have been a tremendous growth to me spiritually.  Maybe they will help you as well if you try them.  But the second reason is much more important to me personally: I want public accountability for my actions.  I want to change the world for the better—I want to be like Christ—but I can’t do that without help.  I need others to lift me up, just as I need to lift others up.

So if you have any suggestions for cool religious experiments I can perform in 2013, please tell me!

This is the first in a series of posts about the HLearn library for haskell that I’ve been working on for the past few months. The idea of the library is to show that abstract algebra—specifically monoids, groups, and homomorphisms—are useful not just in esoteric functional programming, but also in real world machine learning problems.  In particular, by framing a learning algorithm according to these algebraic properties, we get three things for free: (1) an online version of the algorithm; (2) a parallel version of the algorithm; and (3) a procedure for cross-validation that runs asymptotically faster than the standard version.

We’ll start with the example of a Gaussian distribution. Gaussians are ubiquitous in learning algorithms because they accurately describe most data.  But more importantly, they are easy to work with.  They are fully determined by their mean and variance, and these parameters are easy to calculate.

In this post we’ll start with examples of why the monoid and group properties of Gaussians are useful in practice, then we’ll look at the math underlying these examples, and finally we’ll see that this technique is extremely fast in practice and results in near perfect parallelization.

HLearn by Example

Install the libraries from a shell:

$ cabal install HLearn-distributions

Then import the HLearn libraries into a literate haskell file:

> import HLearn.Algebra
> import HLearn.Models.Distributions.Gaussian

And some libraries for comparing our performance:

> import Criterion.Main
> import Statistics.Distribution.Normal
> import qualified Data.Vector.Unboxed as VU

Now let’s create some data to work with. For simplicity’s sake, we’ll use a made up data set of how much money people make. Every entry represents one person making that salary. (We use a small data set here for ease of explanation.  When we stress test this library at the end of the post we use much larger data sets.)

> gradstudents = [15e3,25e3,18e3,17e3,9e3]        :: [Double]
> teachers     = [40e3,35e3,89e3,50e3,52e3,97e3]  :: [Double]
> doctors      = [130e3,105e3,250e3]              :: [Double]

In order to train a Gaussian distribution from the data, we simply use the train function, like so:

> gradstudents_gaussian = train gradstudents      :: Gaussian Double
> teachers_gaussian     = train teachers          :: Gaussian Double
> doctors_gaussian      = train doctors           :: Gaussian Double

The train function is a member of the HomTrainer type class, which we’ll talk more about later.  Also, now that we’ve trained some Gaussian distributions, we can perform all the normal calculations we might want to do on a distribution.  For example, taking the mean, standard deviation, pdf, and cdf.

Now for the interesting bits. We start by showing that the Gaussian is a semigroup. A semigroup is any data structure that has an associative binary operation called (<>). Basically, we can think of (<>) as “adding” or “merging” the two structures together. (Semigroups are monoids with only a mappend function.)

So how do we use this? Well, what if we decide we want a Gaussian over everyone’s salaries? Using the traditional approach, we’d have to recompute this from scratch.

> all_salaries = concat [gradstudents,teachers,doctors]
> traditional_all_gaussian = train all_salaries :: Gaussian Double

But this repeats work we’ve already done. On a real world data set with millions or billions of samples, this would be very slow. Better would be to merge the Gaussians we’ve already trained into one final Gaussian. We can do that with the semigroup operation (<>):

> semigroup_all_gaussian = gradstudents_gaussian <> teachers_gaussian <> doctors_gaussian

Now,

traditional_all_gaussian == semigroup_all_gaussian

The coolest part about this is that the semigroup operation takes time O(1), no matter how much data we’ve trained the Gaussians on. The naive approach takes time O(n), so we’ve got a pretty big speed up!

Next, a monoid is a semigroup with an identity. The identity for a Gaussian is easy to define—simply train on the empty data set!

> gaussian_identity = train ([]::[Double]) :: Gaussian Double

Now,

gaussian_identity == mempty

But we’ve still got one more trick up our sleeves.  The Gaussian distribution is not just a monoid, but also a group. Groups appear all the time in abstract algebra, but they haven’t seen much attention in functional programming for some reason. Well groups are simple: they’re just monoids with an inverse. This inverse lets us do “subtraction” on our data structures.

So back to our salary example. Lets say we’ve calculated all our salaries, but we’ve realized that including grad students in the salary calculations was a mistake. (They’re not real people after all.) In a normal library, we would have to recalculate everything from scratch again, excluding the grad students:

> nograds = concat [teachers,doctors]
> traditional_nograds_gaussian = train nograds :: Gaussian Double

But as we’ve already discussed, this takes a lot of time. We can use the inverse function to do this same operation in constant time:

> group_nograds_gaussian = semigroup_all_gaussian <> (inverse gradstudents_gaussian)

And now,

traditional_nograds_gaussian == group_nograds_gaussian

Again, we’ve converted an operation that would have taken time O(n) into one that takes time O(1). Can’t get much better than that!

The HomTrainer Type Class

As I’ve already mentioned, the HomTrainer type class is the basis of the HLearn library.  Basically, any learning algorithm that is also a semigroup homomorphism can be made an instance of HomTrainer.  This means that if xs and ys are lists of data points, the class obeys the following law:

train (xs ++ ys) == (train xs) <> (train ys)

It might be easier to see what this means in picture form:

On the left hand side, we have some data sets, and on the right hand side, we have the corresponding Gaussian distributions and their parameters.  Because training the Gaussian is a homomorphism, it doesn’t matter whether we follow the orange or green paths to get to our final answer.  We get the exact same answer either way.

Based on this property alone, we get the three “free” properties I mentioned in the introduction.  (1) We get an online algorithm for free.  The function add1dp can be used to add a single new point to an existing Gaussian distribution.  Let’s say I forgot about one of the graduate students—I’m sure this would never happen in real life—I can add their salary like this:

> gradstudents_updated_gaussian = add1dp gradstudents_gaussian (10e3::Double)

This updated Gaussian is exactly what we would get if we had included the new data point in the original data set.

(2) We get a parallel algorithm.  We can use the higher order function parallel to parallelize any application of train.  For example,

> gradstudents_parallel_gaussian = (parallel train) gradstudents :: Gaussian Double

The function parallel automatically detects the number of processors your computer has and evenly distributes the work load over them.  As we’ll see in the performance section, this results in perfect parallelization of the training function.  Parallelization literally could not be any simpler!

(3) We get asymptotically faster cross-validation; but that’s not really applicable to a Gaussian distribution so we’ll ignore it here.

One last note about the HomTrainer class: we never actually have to define the train function for our learning algorithm explicitly.  All we have to do is define the semigroup operation, and the compiler will derive our training function for us!  We’ll save a discussion of why this homomorphism property gives us these results for another post.  Instead, we’ll just take a look at what the Gaussian distribution’s semigroup operation looks like.

The Semigroup operation

Our Gaussian data type is defined as:

data Gaussian datapoint = Gaussian
    { n  :: !Int         -- The number of samples trained on
    , m1 :: !datapoint   -- The mean (first moment) of the trained distribution
    , m2 :: !datapoint   -- The variance (second moment) times (n-1)
    , dc :: !Int         -- The number of "dummy points" that have been added
    }

In order to estimate a Gaussian from a sample, we must find the total number of samples (n), the mean (m1), and the variance (calculated from m2).  (We’ll explain what dc means a little later.)  Therefore, we must figure out an appropriate definition for our semigroup operation below:

(Gaussian na m1a m2a dca) <> (Gaussian nb m1b m2b dcb) = Gaussian n' m1' m2' dc'

First, we calculate the number of samples n’. The number of samples in the resulting distribution is simply the sum of the number of samples in both the input distributions:

Second, we calculate the new average m1′. We start with the definition that the final mean is:

Then we split the summation according to whether the input element was from the left Gaussian a or right Gaussian b, and substitute with the definition of the mean above:

Notice that this is simply the weighted average of the two means. This makes intuitive sense. But there is a slight problem with this definition: When implemented on a computer with floating point arithmetic, we will get infinity whenever n’ is 0.  We solve this problem by adding a “dummy” element into the Gaussian whenever n’ would be zero.  This increases n’ from 0 to 1, preventing the division by 0.  The variable dc counts how many dummy variables have been added, so that we can remove them before performing calculations (e.g. finding the pdf) that would be affected by an incorrect number of samples.

Finally, we must calculate the new m2′. We start with the definition that the variance times (n-1) is:

(Note that the second half of the equation is a property of variance, and its derivation can be found on wikipedia.)

Then, we do some algebra, split the summations according to which input Gaussian the data point came from, and resubstitute the definition of m2 to get:

Notice that this equation has no divisions in it.  This is why we are storing m2 as the variance times (n-1) rather than simply the variance.  Adding in the extra divisions causes training our Gaussian distribution to run about 4x slower.  I’d say haskell is getting pretty fast if the number of floating point divisions we perform is impacting our code’s performance that much!

Performance

This algebraic interpretation of the Gaussian distribution has excellent time and space performance.  To show this, we’ll compare performance to the excellent Haskell package called “statistics” that also has support for Gaussian distributions.  We use the criterion package to create three tests:

> size = 10^8
> main = defaultMain
>     [ bench "statistics-Gaussian" $ whnf (normalFromSample . VU.enumFromN 0) (size)
>     , bench "HLearn-Gaussian" $ whnf
>         (train :: VU.Vector Double -> Gaussian Double)
>         (VU.enumFromN (0::Double) size)
>     , bench "HLearn-Gaussian-Parallel" $ whnf
>         (parallel $ (train :: VU.Vector Double -> Gaussian Double))
>         (VU.enumFromN (0::Double) size)
>     ]

In these test, we time three different methods of constructing Gaussian distributions given 100,000,000 data points.  On my laptop with 2 cores, I get these results:

Pretty nice!  The algebraic method managed to outperform the traditional method for training a Gaussian by a handy margin.  Plus, our parallel algorithm runs exactly twice as fast on two processors.  Theoretically, this should scale to an arbitrary number of processors, but I don’t have a bigger machine to try it out on.

Another interesting advantage of the HLearn library is that we can trade off time and space performance by changing which data structures store our data set.  Specifically, we can use the same functions to train on a list or an unboxed vector.  We do this by using the ConstraintKinds package on hackage that extends the base type classes like Functor and Foldable to work on classes that require constraints.  Thus, we have a Functor instance of Vector.Unboxed. This is not possible without ConstraintKinds.

Using this benchmark code:

main = do
    print $ (train [0..fromIntegral size::Double] :: Gaussian Double)
    print $ (train (VU.enumFromN (0::Double) size) :: Gaussian Double)

We generate the following heap profile:

Processing the data as a vector requires that we allocate all the memory in advance.  This lets the program run faster, but prevents us from loading data sets larger than the amount of memory we have.  Processing the data as a list, however, allows us to allocate the memory only as we use it.  But because lists are boxed and lazy data structures, we must accept that our program will run about 10x slower.  Lucky for us, GHC takes care of all the boring details of making this happen seamlessly.  We only have to write our train function once.

Future Posts

There’s still at least four more major topics to cover in the HLearn library:  (1) We can extend this discussion to show how the Naive Bayes learning algorithm has a similar monoid and group structure.  (2) There are many more learning algorithms with group structures we can look into.  (3) We can look at exactly how all these higher order functions, like batch and parallel work under the hood.  And (4) we can see how the fast cross-validation I briefly mentioned works and why it’s important.

Subscribe to the RSS feed and stay tuned!

Anyone who wants to be first must be the very last, and the servant of all.

– Mark 9:35

Being the servant of all is a hard task that we often forget to do.  We need to remind ourselves constantly that we are here to serve everyone.  One easy way to do this is to call everyone else “Sir” or “Ma’am.”  This language serves as a reminder to ourselves, and at the same time uplifts the person we’re talking to.

I discovered this trick when I was an officer in the Navy.  All the enlisted sailors had to call me “Sir,” and that frankly made me feel good about myself.  It made me feel important.  I had twenty people I was in charge of, and their sole purpose in life was to do what I told them to do.  To drive this point home, all midshipmen have to memorize this quote in their first year at the Naval Academy:

Sir, sir is subservient word surviving from the surly days of old Serbia, when certain serfs, too ignorant to remember their lord’s names, yet too servile to blaspheme them, circumvented the situation by surrogating the subservient word sir, by which I now belatedly address a certain senior cirroped who correctly surmised that I was syrupy enough to say sir after every word I said, sir.

But one of the reasons I left the Navy was the realization that the military’s power structure looked nothing like what Jesus wanted from his followers.  Jesus came to invert this power structure.  He came to make the first last and the last first—so we need to do that in our own lives.  As an officer, I should have been calling my sailors sir, because it should have been me serving them.  That’s the example that Jesus gave.

So now when I’m walking down the street and see a homeless women, I greet her with “Good morning, ma’am!”  God put me on that path in order to be a Jesus for her—to be her servant.  I need to remind myself that I was made just for this moment: to accept this woman’s sins as my own and serve her as Christ would.  And she needs someone to treat her with the dignity of a human created in God’s image.  Being called ma’am may just be the only dignity she receives for the rest of her life.