E-Mail Merge in Go

Mail merge is a funny thing.  Once a year, I use "mail merge" in Microsoft Office to produce envelopes that are physically mailed.  Mail merge is really good for that... you can make a single word doc that is easy to print, and then you've got all the physical documents you need, ready to be taken to a physical post office.

As a professor, there are many, many times that I need to do a mail merge that results in an email being sent.  Partly because I do a lot of work in Linux environments, and partly because of other oddities of how I like to work, I usually have a hybrid Excel-then-text workflow for this task.

The first step is to produce a spreadsheet, where each column corresponds to the content I want placed into an email.  Ultimately, I save this as a '.csv' file.  Importantly, I make sure that each column corresponds to text that requires no further edits.  If I'm sending out grades, I'll store three columns: your sum, the total, and your average.  You could imagine something like the following:

bob@phonyemail, 15, 20, 75% 

sue@phonyemail, 19, 20, 95%

...

The third step (yes, I know this sounds like Underpants Gnomes) is that I have one file per email, saved with a name corresponding to the email address, ready to be sent, and I use a quick shell script like this to send the files:

for f in *; do mutt -s "Grade Report" -c myemail@phony.net $f < $f; done

That is "for each file, where the name happens to be the same as the full email address of the recipient, send an email with the subject 'Grade Report', cc'd to me, to the person, and use the content of the corresponding file as the content of the email".

So far, so good, right?  What about phase two?  I'm pretty good with recording emacs macros on the fly, so I used to just record a macro of me turning a single line of csv into a file, and then replay that macro for each line of the csv.  It worked, it took about 10 minutes, but it was ugly and error-prone.

I recently decided to start learning Google Go (in part because one of the founders of a really cool startup called Loopd pointed out that native code performance can make a huge difference when you're doing real-time analytics on your web server).  Since I've simplified my problem tremendously (remember: the csv has text that's ready to dump straight into the final email), the Go code to make this work is pretty simple.  Unfortunately, it wasn't as simple to write as I would have hoped, because the documentation for templates is lacking.

Here's the code:

package main

import ("encoding/csv"; "flag"; "io"; "os"; "text/template")

/// Wrap an array of strings as a struct, so we can pass it to a template
type TWrap struct { Fields *[]string }

/// Parse a CSV so that each line becomes an array of strings, and then use
/// the array of strings with a template to generate one file per csv line
func main() {
 // parse command line options
 csvname := flag.String("c", "a.csv", "The csv file to parse")
 tplname := flag.String("t", "a.tpl", "The template to use")
 fnameidx := flag.Int("i", 0, "Column of csv to use as output file basename")
 fnamesuffix := flag.String("s", "out", "Output file suffix")
 flag.Parse()

 // load the text template
 tpl, err := template.ParseFiles(*tplname)
 if err != nil { panic(err) }

 // load the csv file
 file, err := os.Open(*csvname)
 if err != nil { panic(err) }
 defer file.Close()

 // parse the csv, one record at a time
 reader := csv.NewReader(file)
 reader.Comma = ','
 for {
  // get next row... exit on EOF
  row, err := reader.Read()
  if err == io.EOF { break } else if err != nil { panic(err) }
  // create output file for row
  f, err := os.Create("./" + row[*fnameidx] + "." + *fnamesuffix)
  if err != nil { panic(err) }
  defer f.Close()
  // apply template to row, dump to file
  tpl.Execute(f, TWrap{Fields:&row})
 }
}

This lets me make a "template" file, merge the csv with it, and output one file per csv row.  Furthermore, I can use a specific row of the csv to dictate the filename, and I can provide extensions (which makes the shell script above a tad trickier, but it's worth it).

The code pulls in the csv row as an array of strings.  That being the case, I can wrap the array in a struct, and then access any array entry via {{index .Fields X}}, where x is the array index.

To make it a tad more concrete, here's a sample template:

Programming Assignment #1 Grade Report

Student Name:  {{index .Fields 2}} {{index .Fields 1}}
User ID:       {{index .Fields 0}}
Overall Score: {{index .Fields 3}}/100

The script uses command line arguments, so it's 100% reusable.  Just provide the template, the csv, the column to use as the output, and the output file extension.

The code isn't really all that impressive, except that (a) it's short, and (b) it is almost as flexible as code in a scripting language, yet it runs natively.  The hardest part was finding good examples online for how to get a template to write to a file.  It's possible I'm doing it entirely wrong, but it seems to work.  If any Go expert wants to chime in and advise on how to use text templates or the csv reader in a more idiomatic way, please leave a comment.

Saving time with VBA and Outlook

Over the years, I've used Visual Basic for Applications in a lot of ways.  I've never really thought of myself as an expert, but I have written a fair bit of VB scripts, even though I'm mostly a Unix/C++/Java programmer.

One thing I've always appreciated about the VB community is that there is a lot of code sharing.  One script I stumbled on a while back is for downloading all attachments, from all selected files, in Outlook.

Our department scanner will send me a separate email for each file that I scan, which means that I can scan all of my students' assignments, one at a time, and have a digital copy of each.  But forwarding those on to the students is usually a pain.

Enter VBA... I used this script to download all the attachments at once.  Then I used the preview pane in Windows to quickly check that the file names were time-ordered in the same sequence as the students user IDs.  A few lines of bash later, and all 89 pdfs were mailed.  Hooray!

Getting Started with JUnit

I had some fun learning about JUnit recently.  I've always believed that it's important to develop incrementally.  The neat thing (to me) about unit testing is that it encourages incremental development -- if you have lots of tests that don't pass, then the natural thing to do is to pick them off, one at a time, and fix them.  In grad school, and now as a professor, I've had a fair number of occasions where someone said "I'm almost done writing it up, I should be ready to compile in a day or two".  Perhaps encouraging students to develop their tests first will discourage them from falling into that pattern of behavior.

Anyhow, I built a tutorial about JUnit, for use in my CSE398 class.  Feel free to share your thoughts on the tutorial, JUnit, and test-driven development in the comments!

Callbacks and Scribble Mode

Last week, we had a mobiLEHIGH tutorial session, and a student asked about how to make Fruit Ninja with LibLOL.  It turns out that getting the right behavior isn't all that hard... you can use the "scribble" feature to detect a "slice" movement on the screen, and configure the obstacles that are scribbled to have the power to defeat enemies.

There was just one problem... configuring the obstacles requires changing the LibLOL code.  I don't discourage people from changing LibLOL, but it's better to have an orthogonal way of getting the same behavior.  In this case, it's easy: let the scribble mode code take a callback, and use that callback to modify an obstacle immediately after it is created.

This is one of those changes that I can't help but love... there's less code in LibLOL, and more power is exposed to the programmer.  But it's not really any harder, and there's less "magic" going on behind the scenes now.

Here's an example of how to provide a callback to scribble mode:

    // turn on 'scribble mode'... this says "draw a purple ball that is 1.5x1.5 at the
    // location where the scribble happened, but only do it if we haven't drawn anything in
    // 10 milliseconds."  It also says "when an obstacle is drawn, do some stuff to the
    // obstacle".  If you don't want any of this functionality, you can replace the whole
    // "new LolCallback..." region of code with "null".
    Level.setScribbleMode("purpleball.png", 1.5f, 1.5f, 10, new LolCallback(){
        @Override
        public void onEvent() {
            // each time we draw an obstacle, it will be visible to this code as the
            // callback's "attached Actor".  We'll change its elasticity, make it disappear
            // after 10 seconds, and make it so that the obstacles aren't stationary
            mAttachedActor.setPhysics(0, 2, 0);
            mAttachedActor.setDisappearDelay(10, true);
            mAttachedActor.setCanFall();
        }
    });

I'm starting to think that I should redesign more of the LibLOL interfaces to use callbacks... what do you think?

Getting Started with OpenCV

OpenCV is a great library for doing all sorts of computer vision.  It's also easy enough that you can use it for all sorts of menial tasks, like resizing images, cropping, adjusting colors, etc.  And best of all, it's FAST.  I have used ImageMagick in the past, and OpenCV feels a ton faster.  If you want some data to support that claim, see this link.

Given that OpenCV is fast, general-purpose, and not too hard to use, I thought it would be a good topic for CSE398.  Here's the tutorial I wrote.

I opted to do everything in Java, because that seemed easier than worrying about memory management.  Of course, that also means it's harder to find good documentation.  I ended up transliterating a lot of C++ code to show how various features work.

Since I'm not expert in Computer Vision, I wouldn't be surprised if I got some things wrong in the tutorial.  Please don't hesitate to send a note if you find any errors!

An Introduction to Extending Code

Here's yet another CSE398 tutorial.  This time it's about how to dynamically load code, or to otherwise create extensions that expand the behavior of a program.  I realized that while students have heard of "DLL Hell", they don't know how to make DLLs (or their Unix equivalent, "shared objects").

DLLs are like lambdas (did you like yesterday's post?): if you don't know how to use them, you end up doing an outrageous amount of work to get an effect that would otherwise be simple.  The tutorial starts by showing how to load and use a shared object in C++, which lets me introduce my basic Makefile template, function pointers, and C++ name mangling.  Then it shows how to do dynamic class loading in Java.  We move from there to using exec or spawn to do inter-process communication (IPC) between Node.js and Java or C++ programs.

I tried to make the tutorial a little bit more fun, by hiding the code for a Java Pig Latin converter.  This was also an attempt to get people to look at the HTML code for my tutorials.  I don't think students realize how easy it is to do clean web design by hand, once you know a little CSS and jQuery.

As with all of my CSE398 tutorials, I wrap up with next steps, this time involving the use of existing Node.js packages to load C++ and Java code directly into a running node server.

A Few Great C++11 Features

A few years back, I heard a talk by John Spicer of Edison Design Group on new features in C++11. After hearing John's talk, I realized that my students and I all fell into his category of programmers who knew the "FORTRAN subset of C++".  Sure, I've written lots of code over the years, and a C++ compiler will understand that code.  But was it really C++?  Was it idiomatic?  And even if it was, would it still be considered idiomatic in the face of C++11?

C++11 adds tons of cool features.  The concurrency support is great.  The STL is even better than before.  But the best part, in my opinion, is lambdas.  With lambdas, data structure design gets a lot easier.  One "map" function can replace all of those one-off functions that otherwise go into a data structure (min, max, average, selectIf, countIf, etc.).

Here's a program I wrote to demonstrate some of the features of C++11.

/// This is a demonstration of a few nice C++11 features: auto, the ':'
/// iterator, initializer lists, std::function, and lambda expressions.  In
/// many cases, it's possible to achieve the same effect as I present using
/// even less code, but by being explicit I hope I have made the code and
/// comments easy to follow

#include <functional>
#include <iostream>
#include <map>
#include <string>
#include <vector>

using namespace std; // I'm being lazy here... should explicitly use cout,
                     // endl, function, map, pair, string, and vector


/// wrapped_map_t extends map<string, string>.  We start with a map of
/// key/value pairs, where both the key and value are strings, and we add a
/// new method to it.  The new method, apply_to_all, lets us apply a lambda
/// to (a copy of) every key/value pair that is in the map.
///
/// my claim is that lambdas fundamentally change the art of data structure
/// design.  To support my claim, I will show how this one simple function
/// obviates many public functions that one might want int a data structure
/// (count, print, extract_keys, find_matching_keys, etc).
struct wrapped_map_t : public map<string, string>
{
    /// the apply_to_all function simply iterates through the key/value pairs
    /// of the map, calling lambda(key, value) for each pair.  Strictly
    /// speaking, std::map is a red/black tree, and this will do a pre-order
    /// traversal
    void apply_to_all(function<void(const string, const string)>& lambda)
    {
        // note the use of C++11 'auto' keyword to avoid having to specify
        // the type of the iterator
        for (auto i = begin(), e = end(); i != e; ++i)
            lambda(i->first, i->second);
    }
};

int main()
{
    // declare an instance of the extended map
    wrapped_map_t kvpairs;

    // populate the map with some keys and values.  We're going to use some
    // great C++ magic here: initializer lists.  We don't even have to know
    // the type of the list, but the compiler will deduce that it's something
    // like pair<string,string>[]
    kvpairs.insert({{"donald", "duck"}, {"mickey", "mouse"},
                    {"minnie", "mouse"}, {"bat", "man"},
                    {"super", "man"}, {"milo", "otis"},
                    {"mighty", "mouse"}, {"rocky", "squirrel"},
                    {"bullwinkle", "moose"}});

    // 'func' is a reference to a function that takes two strings as its
    // parameters, and returns nothing.  This is a lot nicer than the old C
    // syntax in use before C++11 (i.e., void (*func)(string, string)).
    function<void(string, string)> func;

    // let's assign a new lambda to func.  the lambda will simply print its
    // parameters
    func = [](string k, string v) { cout << "{" << k << ", " << v << "} "; };
    // now let's apply func to every element in the set.  No need for a
    // "print_in_sorted_key_order" function in our class.  It would be
    // trivial to add "print keys" and "print values" features without
    // changing the data structure... we'd just need to pass different
    // lambdas
    cout << "KVPairs = ";
    kvpairs.apply_to_all(func);
    cout << endl;

    // The last example was pretty boring.  A function pointer could have
    // done that easily.  Now let's pass in a function that has a captured
    // reference to a variable that is local to this function.  In that way,
    // each time the lambda is called, it can access the function 'my_count'
    // that we declare *right here*.  The net result is that we can count the
    // elements in the map, without requiring map to provide a "count()"
    // function.  Note that the parameters to the lambda aren't used, and
    // that's OK.
    int my_count = 0;
    func = [&my_count](string k, string v) { my_count++; };
    kvpairs.apply_to_all(func);
    cout << "Elements in map = " << my_count << endl;

    // OK, so we could have achieved that effect with an apply_to_all that
    // took three parameters: string, string, int&.  But would we really want
    // to do that for every possible pass-by-reference type?  For example,
    // here we pass a vector of strings so that we can copy out all keys from
    // the map
    vector<string> keys;
    func = [&keys](string k, string v) {keys.push_back(k);};
    kvpairs.apply_to_all(func);
    // print the keys, to show that it worked
    cout << "Keys in map = {";
    for (auto i : keys)
        cout << i << ",";
    cout << "}" << endl;

    // note, too, that we can use the lambda to capture multiple local
    // variables, and that some can be pass by value while others are pass by
    // reference.  Here 'pre' is pass by value, 'keys' is pass by reference,
    // and our function copies into 'keys' all keys in the map whose values
    // begin with 'mo'
    string pre = "mo";
    keys.clear();
    func = [&keys, pre](string k, string v) { if (v.find(pre) == 0) keys.push_back(k);};
    kvpairs.apply_to_all(func);
    cout << "Keys whose values start with 'mo' = {";
    for (auto i : keys)
        cout << i << ",";
    cout << "}" << endl;

    // this example is much like the prior one.  Here we extract all
    // key/value pairs where the key begins with 'mi'
    pre = "mi";
    vector<pair<string, string>> pairs;
    func = [&pairs, pre] (string k, string v)
        { if (k.find(pre) == 0) pairs.push_back(make_pair(k, v)); };
    kvpairs.apply_to_all(func);
    cout << "Elements whose key begins with 'mi' = {";
    for (auto i : pairs)
        cout << "{"<<i.first<<", "<<i.second<<"} ";
    cout << "}" << endl;

    // the point of this example is to illustrate that when the lambda is
    // called with the key and value, the function we are executing *cannot*
    // modify the internals of the map.  Put another way, the lambda does not
    // become a member function of the class.  So, for example, if we wanted
    // to add a suffix to every key, then the only way we could do it would
    // be by explicitly updating the k/v pair in the map.
    string suf = " <3";
    // note: we need to capture *kvpairs itself* by reference
    func = [&kvpairs, suf] (string k, string v) { kvpairs[k] = v+suf; };
    kvpairs.apply_to_all(func);
    // use a lambda to print it out
    func = [](string k, string v) { cout << "{" << k << ", " << v << "} "; };
    cout << "Modified kvpairs = ";
    kvpairs.apply_to_all(func);
    cout << endl;
}

Don't forget that you need to use the -std=c++11 flag when you compile this with g++.