Launching a Go App on Heroku

Lehigh is part of the KEEN network, an organization that promotes more entrepreneurial-minded learning in engineering curriculum.  This summer, as a KEEN project, Corey Caplan and I are designing some fun new courseware for our Software Engineering course.

Our intention is to do everything in Java within the course.  But when I need to figure out something about web backends in a hurry, I'd rather use Go.  Today was one of those times.

Without going into too much detail, I have a web app that I wanted to stop running via localhost, and start running on Heroku.  (If you're thinking this means that our Software Engineering students are going to start learning how to deploy their apps on Heroku's PaaS, you're right!).  Below is something of a recipe for how I got it to work.

Confession: this turned out to be a lot harder than I expected, and it was probably my fault.

Caveat: the recipe below is possibly a good bit more complex than it needs to be... but it works, and seems to be repeatable.

Background: I had an app that looked like this:

• /src/admin/*.go -- a simple admin program
• /src/appserver/*.go -- the code for the server
• /web/... -- the entire web frontend, designed as a single-page webapp
• /env -- a script to set the environment variable when running the app locally
• /setgopath.sh -- a script to set the GOPATH to the root of this project

There were a few more things in the folder, like a .gitignore, but they aren't important to this discussion.

Note, too, that I like to have a different GOPATH for each GO project, instead of checking them all into the same place.  I organize my work in folders: teaching, research, etc.  Using Visual Studio Code, I can just open a bash prompt, source my setgopath.sh script, type "code &", and I've got an IDE, a shell, and everything else I need.

Dependencies: Here's the first reason why this app was interesting: it uses Google's OAuth provider for authentication, and it connects to a MongoDB instance.  There are four dependencies that I usually had to 'go get':

• go get golang.org/x/oauth2
• go get golang.org/x/oauth2/google
• go get gopkg.in/mgo.v2
• go get gopkg.in/mgo.v2/bson

And my code is in a bitbucket repository.  Let's say it's bitbucket.org/me/myapp.  When I started, I had a checkout of myapp on the desktop.  So there was a folder ~/Desktop/myapp, in which was a .git/ folder and all the stuff mentioned above.

Restructuring:  This was probably overkill, but it worked.  I started by creating a new folder on the desktop called myapp_heroku.  In it, I made a src/bitbucket/me folder, and I moved myapp/ from the Desktop to that place.  I also changed my setgopath.sh script, so that Desktop/myapp_heroku is the new GOPATH.

Note: now when I'm working on this project, I traverse all the way into the src/bitbucket.org/me/myapp folder, and I work there, but when I do a 'go install' or a 'go get', things are placed a few levels up in the directory tree.

After restructuring, I removed some cruft from the build folder.  Previously, there were bin/ and pkg/ folders in myapp... I got rid of them.  I also removed any source folders that were fetched via 'go get', because dependent files go elsewhere now.

Using godep:  Our goal in this step is to get all the code we depend on, in a manner that will ensure that Heroku grabs the same code when it builds updated versions of the app.

This is where things became un-intuitive.  Go, of course, doesn't have any built-in mechanism for managing dependencies.  Godep essentially just vendors everything into the source tree, which I don't particularly like, but it suffices.

Naturally, we need to get godep first, and add it to our path:

• go get github.com/tools/godep
• export PATH=$PATH:$GOPATH/bin

With that in order, we should restructure our repository ever so slightly:

• git mv src cmd
• git mv src/appserver src/myapp

I don't know why these steps were necessary.  But stuff really didn't work until I made both of those changes.  The Heroku docs obliquely state the first requirement, without any explanation, and the second requirement (which is that the main program you want to run should have the same name as your repository) was just a fact of how the tutorials I read all were done.  None of those tutorials had multiple executables in their projects.

(Update: I might not have needed to rename src/appserver.)

We can use godep to fetch the packages on which we depend:

• godep get golang.org/x/oauth2
• godep get golang.org/x/oauth2/google
• godep get google.golang.org/appengine
• godep get gopkg.in/mgo.v2
• godep get gopkg.in/mgo.v2/bson

Oddly, when fetching oauth2, we get an error that appengine isn't available.  For me, doing a recursive get (godep get golang.org/x/oauth2/...) didn't work.  So I manually got one more package.

Now we can take the 'vendoring' step:

• godep save ./...

And voila!  There's a folder called 'vendor', with all of the code we depend upon, and there's also a Godep folder.  Too bad it won't work.

The problem is that we're going to push our code to a Heroku "dyno" (think "container") and it's going to build the code.  But the mgo.v2 library's optional sasl support will be built when we push to Heroku.  That support depends on libsasl-dev being available on the host machine at build time.  The image for the Heroku dyno I'm using doesn't have libsasl-dev.  So if we were to push this repository to Heroku, it wouldn't build, and the code would be rejected.

The fix is easy: just delete the sasl folder from the vendored mgo.v2:

• rm -rf vendor/gopkg.in/mgo.v2/internal/sasl/

Ugly, but it works.  And indeed, we're close to having everything work at this point.  To test that our vendoring is good, try to locally build the project:

• godep go install -v bitbucket.org/me/myapp/cmd/myapp

The code should build... and it should use the vendored versions of the libraries.

Heroku Stuff: Heroku has a few more requirements that we need to satisfy.  First, we need a file called Procfile.  Its contents will just be "web: myapp".  Second, we need an app.json file.  Its contents are a bit more complex, though still straightforward:

{
  "name": "myapp",
  "description": "MyApp App Server",
  "keywords": [
    "go",
    "MyApp"
  ],
  "image": "heroku/go:1.6",
  "mount_dir": "src/bitbucket.org/me/myapp",
  "website": "https://bitbucket.org/me/myapp",
  "repository": "https://bitbucket.org/me/myapp"
}

Now we can actually create the heroku app.  I was working in Git Bash for Windows, which isn't supported by the Heroku toolbelt.  So I had to switch to the command prompt, and log in:

• cd \Users\Me\Desktop\myapp_heroku\bitbucket.org\me\myapp
• heroku login
• heroku app:create myapp

At this point 'heroku local' should work.  To push to Heroku, we first 'git add' the vendor folder and all of our other recent additions, and then 'git commit'.  Then we can 'git push heroku master'.  This takes longer than a usual git push, because it doesn't finish until Heroku is done building and verifying our program.

Are We Done Yet?  Not really.  If you 'heroku run bash', you can see that bin/admin is present in the dyno, as is bin/myapp.  That's a good sign.  But our app isn't running yet.  One issue I had was that I needed to manually start the app:

• heroku ps:scale web=1

The other issue is that we didn't yet set up the environment variables on Heroku.  We need to 'heroku config:set DBCONNECTSTRING=...' in order to let our app know how to find our cloud-hosted MongoDB instance, we need to set some OAUTH secrets, and we need to set environment variables for whatever else the app is expecting.  But that depends on the app, not on Heroku, so I'm not going to discuss it here.

Wrap-Up:  It took longer than I expected to get this to work.  Since I'll probably have to do it again, I thought it would be worth writing up the steps I took.  If this is helpful to you, too, please leave a comment and let me know.

Fun with Google Forms

Here's a link to another of my tutorials for CSE398.  I figured this out in the course of making a registration page for an elementary school's after-school enrichment program.  We wanted the convenience of a Google Form, with all results going to a shared spreadsheet, but we also wanted to be able to keep people from registering for activities that were either (a) not available based on their child's grade level, or (b) already at capacity.

The trick is to make a separate worksheet in the Google Spreadsheet that is attached to the form, set it up to summarize the data in the main sheet of the spreadsheet, publish the second worksheet, and only the second worksheet, and then consume it as JSON.  Then it's just a matter of using JavaScript on the client side to fetch the JSON, parse it, and use the resulting data to drive the UI.

Somewhere along the way, I realized that the Google Form is really two components with a straightforward API connecting them.  We can't change the API, but we can have a different front-end that interacts with the same back-end, as long as we respect the API.  Once I realized that, it seemed worthwhile to make a tutorial about it.

The link to the tutorial is here.  Hopefully it is easy to follow.  As with my other CSE398 tutorials, there are "next steps" that I leave up to the student.

Disclaimer: there are ways to hack the technique I present, so don't use it for anything mission-critical.  You'll still get the safety that Google Forms provides, but people can get around the JavaScript-based validation.

Getting started with Node.js

This semester I'm teaching a new class, CSE398: Large Scale Software Development.  The goal of the class is to provide students with a hands-on experience designing and implementing bigger programs than they see in a typical class.  My informal informal motto is "If you need to have a database and a web site, that's not two projects... it's half a project, because you don't have a desktop or mobile component."

That may seem trite, but my feeling is that it's important to use other people's APIs, to connect software components that are implemented in different languages, and to understand that real-world software must be thought of as a large set of loosely connected components.

I'm also hopeful that this will be a great laboratory for learning about technical debt.  In particular, I'm hoping that students will cut corners early in the semester, and then have nobody but themselves to blame when they return to that code later in the semester.  We shall see how it goes.

Anyway, with all of that exposition out of the way, I'll note that a few weeks back, I put together a tutorial on how to do all the basic web things (forms, REST, connect to MongoDB, push notifications) that people do with Node.js.  The tutorial is publicly available through this link.

There are a few bugs in the tutorial, so if you notice anything incorrect, please let me know in the comments, so I can make corrections.

You'll also note that each tutorial has some "next steps", which I expect the students to do without my close guidance.  This is meant to encourage learning about how to teach oneself from on-line tutorials, and also how to evaluate on-line resources.  For what it's worth, I think the scotch.io tutorial series on passport.js is pretty good.