The transaction_wrap feature in GCC

When using transactional memory, a common challenge is that functions in standard libraries cannot be called from transactions.  Sometimes, the incompatibility is unavoidable (for example, because the library does something that cannot be rolled back).  But in other cases, the implementation is not transaction safe, when other (less performant) implementations could be.

The "transaction_wrap" attribute allows you to specify that function w() should be called in place of o() whenever o() is called from within a transaction.  The syntax is not too cumbersome, but there are a few gotchas.  Here's an example to show how it all works:

// This is a quick demonstration of how to use transaction_wrap in GCC

#include <iostream>
using namespace std;

// We have a function called orig(), which we assume is implemented in a
// manner that is not transaction-safe.
int orig(int);

// This line says that there is a function called wrapper(), which is
// transaction-safe, which has the same signature as orig().  When a
// transaction calls orig(), we would like wrapper() to be called instead.
int wrapper(int) __attribute__((transaction_safe, transaction_wrap (orig)));

// Here is our original function.  It does two things:
//
// 1 - saves its operand to a volatile variable iii.  This is not
//     transaction-safe!
// 2 - adds one to its operand and returns the sum
volatile int iii;
int orig(int x) {
    iii = x;
    return x + 1;
}

// Here is our wrapper function.  It adds two to its operand and returns the
// sum.  Note that we have explicitly implemented this in a manner that
// differs from orig(), so that we can easily see which is called
int wrapper(int x) {
    return x + 2;
}

// Our driver function calls orig(1) from three contexts: nontransactional,
// atomic transaction, and relaxed transaction, and prints the result of each
// call
//
// Be warned: the behavior is not what you expect, because it depends on the
// TM algorithm that is used.  For serial-irrevocable (serialirr), the result
// is (2,2,2).  For serial, ml_wt, and gl_wt, the result is (2, 3, 3).  For
// htm, the result is (2,3,2).
int main() {
    int x = orig(1);
    cout << "orig(1) (raw) == " << x << endl;
    __transaction_atomic { x = orig(1); }
    cout << "orig(1) (atomic) == " << x << endl;
    __transaction_relaxed { x = orig(1); }
    cout << "orig(1) (relaxed) == " << x << endl;
    return 0;
}

Compile the code like this:

g++ -fgnu-tm -std=c++11 -O3 test.cc -o test

And as suggested in the comments, the output will depend on the ITM_DEFAULT_METHOD you choose:

ITM_DEFAULT_METHOD=serialirr ./test
orig(1) (raw) == 2
orig(1) (atomic) == 2
orig(1) (relaxed) == 2
ITM_DEFAULT_METHOD=serial ./test
orig(1) (raw) == 2
orig(1) (atomic) == 3
orig(1) (relaxed) == 3
ITM_DEFAULT_METHOD=ml_wt ./test
orig(1) (raw) == 2
orig(1) (atomic) == 3
orig(1) (relaxed) == 3
ITM_DEFAULT_METHOD=gl_wt ./test
orig(1) (raw) == 2
orig(1) (atomic) == 3
orig(1) (relaxed) == 3
ITM_DEFAULT_METHOD=htm ./test
orig(1) (raw) == 2
orig(1) (atomic) == 3
orig(1) (relaxed) == 2

E-Mail Merge in Go

Mail merge is a funny thing.  Once a year, I use "mail merge" in Microsoft Office to produce envelopes that are physically mailed.  Mail merge is really good for that... you can make a single word doc that is easy to print, and then you've got all the physical documents you need, ready to be taken to a physical post office.

As a professor, there are many, many times that I need to do a mail merge that results in an email being sent.  Partly because I do a lot of work in Linux environments, and partly because of other oddities of how I like to work, I usually have a hybrid Excel-then-text workflow for this task.

The first step is to produce a spreadsheet, where each column corresponds to the content I want placed into an email.  Ultimately, I save this as a '.csv' file.  Importantly, I make sure that each column corresponds to text that requires no further edits.  If I'm sending out grades, I'll store three columns: your sum, the total, and your average.  You could imagine something like the following:

bob@phonyemail, 15, 20, 75% 

sue@phonyemail, 19, 20, 95%

...

The third step (yes, I know this sounds like Underpants Gnomes) is that I have one file per email, saved with a name corresponding to the email address, ready to be sent, and I use a quick shell script like this to send the files:

for f in *; do mutt -s "Grade Report" -c myemail@phony.net $f < $f; done

That is "for each file, where the name happens to be the same as the full email address of the recipient, send an email with the subject 'Grade Report', cc'd to me, to the person, and use the content of the corresponding file as the content of the email".

So far, so good, right?  What about phase two?  I'm pretty good with recording emacs macros on the fly, so I used to just record a macro of me turning a single line of csv into a file, and then replay that macro for each line of the csv.  It worked, it took about 10 minutes, but it was ugly and error-prone.

I recently decided to start learning Google Go (in part because one of the founders of a really cool startup called Loopd pointed out that native code performance can make a huge difference when you're doing real-time analytics on your web server).  Since I've simplified my problem tremendously (remember: the csv has text that's ready to dump straight into the final email), the Go code to make this work is pretty simple.  Unfortunately, it wasn't as simple to write as I would have hoped, because the documentation for templates is lacking.

Here's the code:

package main

import ("encoding/csv"; "flag"; "io"; "os"; "text/template")

/// Wrap an array of strings as a struct, so we can pass it to a template
type TWrap struct { Fields *[]string }

/// Parse a CSV so that each line becomes an array of strings, and then use
/// the array of strings with a template to generate one file per csv line
func main() {
 // parse command line options
 csvname := flag.String("c", "a.csv", "The csv file to parse")
 tplname := flag.String("t", "a.tpl", "The template to use")
 fnameidx := flag.Int("i", 0, "Column of csv to use as output file basename")
 fnamesuffix := flag.String("s", "out", "Output file suffix")
 flag.Parse()

 // load the text template
 tpl, err := template.ParseFiles(*tplname)
 if err != nil { panic(err) }

 // load the csv file
 file, err := os.Open(*csvname)
 if err != nil { panic(err) }
 defer file.Close()

 // parse the csv, one record at a time
 reader := csv.NewReader(file)
 reader.Comma = ','
 for {
  // get next row... exit on EOF
  row, err := reader.Read()
  if err == io.EOF { break } else if err != nil { panic(err) }
  // create output file for row
  f, err := os.Create("./" + row[*fnameidx] + "." + *fnamesuffix)
  if err != nil { panic(err) }
  defer f.Close()
  // apply template to row, dump to file
  tpl.Execute(f, TWrap{Fields:&row})
 }
}

This lets me make a "template" file, merge the csv with it, and output one file per csv row.  Furthermore, I can use a specific row of the csv to dictate the filename, and I can provide extensions (which makes the shell script above a tad trickier, but it's worth it).

The code pulls in the csv row as an array of strings.  That being the case, I can wrap the array in a struct, and then access any array entry via {{index .Fields X}}, where x is the array index.

To make it a tad more concrete, here's a sample template:

Programming Assignment #1 Grade Report

Student Name:  {{index .Fields 2}} {{index .Fields 1}}
User ID:       {{index .Fields 0}}
Overall Score: {{index .Fields 3}}/100

The script uses command line arguments, so it's 100% reusable.  Just provide the template, the csv, the column to use as the output, and the output file extension.

The code isn't really all that impressive, except that (a) it's short, and (b) it is almost as flexible as code in a scripting language, yet it runs natively.  The hardest part was finding good examples online for how to get a template to write to a file.  It's possible I'm doing it entirely wrong, but it seems to work.  If any Go expert wants to chime in and advise on how to use text templates or the csv reader in a more idiomatic way, please leave a comment.

Saving time with VBA and Outlook

Over the years, I've used Visual Basic for Applications in a lot of ways.  I've never really thought of myself as an expert, but I have written a fair bit of VB scripts, even though I'm mostly a Unix/C++/Java programmer.

One thing I've always appreciated about the VB community is that there is a lot of code sharing.  One script I stumbled on a while back is for downloading all attachments, from all selected files, in Outlook.

Our department scanner will send me a separate email for each file that I scan, which means that I can scan all of my students' assignments, one at a time, and have a digital copy of each.  But forwarding those on to the students is usually a pain.

Enter VBA... I used this script to download all the attachments at once.  Then I used the preview pane in Windows to quickly check that the file names were time-ordered in the same sequence as the students user IDs.  A few lines of bash later, and all 89 pdfs were mailed.  Hooray!

Getting Started with JUnit

I had some fun learning about JUnit recently.  I've always believed that it's important to develop incrementally.  The neat thing (to me) about unit testing is that it encourages incremental development -- if you have lots of tests that don't pass, then the natural thing to do is to pick them off, one at a time, and fix them.  In grad school, and now as a professor, I've had a fair number of occasions where someone said "I'm almost done writing it up, I should be ready to compile in a day or two".  Perhaps encouraging students to develop their tests first will discourage them from falling into that pattern of behavior.

Anyhow, I built a tutorial about JUnit, for use in my CSE398 class.  Feel free to share your thoughts on the tutorial, JUnit, and test-driven development in the comments!

Callbacks and Scribble Mode

Last week, we had a mobiLEHIGH tutorial session, and a student asked about how to make Fruit Ninja with LibLOL.  It turns out that getting the right behavior isn't all that hard... you can use the "scribble" feature to detect a "slice" movement on the screen, and configure the obstacles that are scribbled to have the power to defeat enemies.

There was just one problem... configuring the obstacles requires changing the LibLOL code.  I don't discourage people from changing LibLOL, but it's better to have an orthogonal way of getting the same behavior.  In this case, it's easy: let the scribble mode code take a callback, and use that callback to modify an obstacle immediately after it is created.

This is one of those changes that I can't help but love... there's less code in LibLOL, and more power is exposed to the programmer.  But it's not really any harder, and there's less "magic" going on behind the scenes now.

Here's an example of how to provide a callback to scribble mode:

    // turn on 'scribble mode'... this says "draw a purple ball that is 1.5x1.5 at the
    // location where the scribble happened, but only do it if we haven't drawn anything in
    // 10 milliseconds."  It also says "when an obstacle is drawn, do some stuff to the
    // obstacle".  If you don't want any of this functionality, you can replace the whole
    // "new LolCallback..." region of code with "null".
    Level.setScribbleMode("purpleball.png", 1.5f, 1.5f, 10, new LolCallback(){
        @Override
        public void onEvent() {
            // each time we draw an obstacle, it will be visible to this code as the
            // callback's "attached Actor".  We'll change its elasticity, make it disappear
            // after 10 seconds, and make it so that the obstacles aren't stationary
            mAttachedActor.setPhysics(0, 2, 0);
            mAttachedActor.setDisappearDelay(10, true);
            mAttachedActor.setCanFall();
        }
    });

I'm starting to think that I should redesign more of the LibLOL interfaces to use callbacks... what do you think?

Getting Started with OpenCV

OpenCV is a great library for doing all sorts of computer vision.  It's also easy enough that you can use it for all sorts of menial tasks, like resizing images, cropping, adjusting colors, etc.  And best of all, it's FAST.  I have used ImageMagick in the past, and OpenCV feels a ton faster.  If you want some data to support that claim, see this link.

Given that OpenCV is fast, general-purpose, and not too hard to use, I thought it would be a good topic for CSE398.  Here's the tutorial I wrote.

I opted to do everything in Java, because that seemed easier than worrying about memory management.  Of course, that also means it's harder to find good documentation.  I ended up transliterating a lot of C++ code to show how various features work.

Since I'm not expert in Computer Vision, I wouldn't be surprised if I got some things wrong in the tutorial.  Please don't hesitate to send a note if you find any errors!

An Introduction to Extending Code

Here's yet another CSE398 tutorial.  This time it's about how to dynamically load code, or to otherwise create extensions that expand the behavior of a program.  I realized that while students have heard of "DLL Hell", they don't know how to make DLLs (or their Unix equivalent, "shared objects").

DLLs are like lambdas (did you like yesterday's post?): if you don't know how to use them, you end up doing an outrageous amount of work to get an effect that would otherwise be simple.  The tutorial starts by showing how to load and use a shared object in C++, which lets me introduce my basic Makefile template, function pointers, and C++ name mangling.  Then it shows how to do dynamic class loading in Java.  We move from there to using exec or spawn to do inter-process communication (IPC) between Node.js and Java or C++ programs.

I tried to make the tutorial a little bit more fun, by hiding the code for a Java Pig Latin converter.  This was also an attempt to get people to look at the HTML code for my tutorials.  I don't think students realize how easy it is to do clean web design by hand, once you know a little CSS and jQuery.

As with all of my CSE398 tutorials, I wrap up with next steps, this time involving the use of existing Node.js packages to load C++ and Java code directly into a running node server.