A day in the life

I seem to have a career as a professional software engineer.  Don't get me wrong, I love doing it, but somehow my avocation became my vocation.  C'est la guerre.

Last night I was thinking about what my job really entails and it is a bit different from what I usually think and tell people.  The exciting part is taking a poorly formed concept, refining it to a very detailed and well-formed formal concept, translating the ideas to software, and making a tool or service that people can use.  There is a tedious part, too.  Frankly, I don't find it too monotonous because it involves a lot of puzzle solving and finding creative solutions, but it is necessary work that doesn't seem logically necessary, and that tends to dampen my enthusiasm for doing it.

I spend a lot of time just “plumbing”.  There is a lot of infrastructure where I work.  In all the projects I've worked on, the vast majority of the code has already been written.  So the very first phase of software development is identifying the infrastructure subcomponents that the project should depend upon.  This is pretty easy to do.  Any basic project will always have these components:

• A web server front-end with the following properties:
  • Automated load balancing.
  • Automated failover.
  • High availability.
  • Resistance against DOS attacks.
• Persistent storage or a database with these properties:
  • Geographically separate replicas.
  • Regular backups.
  • Tools for manual editing and revision of stored data.
• Software attendants to keep it running ‘unattended’.

My current project involves selling things (like all businesses need to do on occasion). There is infrastructure to handle purchases, refunds, deal with fraud, etc. We'll also need some more infrastructure to handle user authorization and authentication. And I didn't mention the infrastructure needed to deliver the purchased items. When it all comes down to it, there isn't a whole lot of logic involved in the project itself. Almost all the functionality comes from the underlying infrastructure. Our team spends a lot of time plugging it together. That doesn't mean that it is easy or that it doesn't involve any creativity.

I've found on most projects I work on that there is always some piece of software that I intend to use but I have only very vague ideas about how it might work. My task might be to hook it up.

Of course I spend several days thoroughly reading and understanding the copious documentation.

This is what I really do: I know that the end user is going to be sitting in front of his web browser and at some point click on something. This will eventually trigger a method call that performs the requested operation. Let's assume the user wants to buy something and that the mouse click the user made was the last one before the purchase (the one that asks “Ok to charge your credit card?”). What can I deduce?

1. There ought to be a method that ultimately handles the ‘purchase’.
2. Since buying something must be a very common operation, I expect there to be some easy way to charge the user's credit card.
   1. I first look for the ‘quick start’ guide or the FAQ.
   2. If I can't find an answer there, I look at the table of contents of the detailed documentation.
   3. A lot of the time there isn't any documentation or if there is, I can't find it. The end result is the same. If there is no documentation, then I think whether there are similar projects that might want to use the same infrastructure in the same way. I'll eventually find some code that actually works.
   4. Now comes a trick. I sketch out in my head a few possible protocol designs. There should be a way, given a user and an item, to “begin” the process of purchasing. There should be a way to determine if the process has completed. There should be a way to determine the outcome of the process. Each of these mechanisms should exist as separate calls because we might have to re-do any part of the process if the computer crashes during the purchase. But since the usual case will involve rapid turnaround and no crashes, there should be an abstraction that handles the entire process from end to end. I poke around in the docs and in the source code to see if these things exist.
   5. Now I analyze what I've found and make a guess at the sophistication of the design and the programmer. If I see an API that involves continuation-passing style or a variant (Java callbacks are poor-man's CPS), then I expect a high level of sophistication from the API. I expect there to be a ‘standard’ way of purchasing something and I expect that all the edge cases are handled in a reasonable way. Why? Anyone that successfully uses a complex concept like continuation-passing style has most likely applied a lot of serious thought to the API.
   6. On the other hand, if I see an API that involves a lot of methods that return void and must be called in the right order or they won't work, or an API with ambiguous methods like checkPurchase() and validatePurchase() (which one do I use?), or one where the arguments must be strings that will then be parsed to something that makes more sense, or APIs that throw only a single catch-all error (or just return null on success or maybe null on failure), then I expect that I'm going to have to think hard about all the edge cases myself and I'm going to have to wade through a lot of examples to determine the right ‘magic’ Why? People that write APIs like this aren't thinking. They code up the first solution that comes into their head and when they are done, they call it an API. It is likely that they either didn't think there were edge cases, or it didn't occur to them that they should abstract them away.
3. So I finally determine what methods need to be called. I look at the argument lists of the methods and I look at the arguments passed in to the servlet. These are generally isomorphic in some way, but it is typically the case that the servlet is being passed a string representation and the underlying method needs an object. If the API isn't completely idiotic, it will provide parsers and unparsers for the data that represent major concepts. So the servlet might get a string that represents a user and a string that represents what he wants to purchase. Now I have to figure out how to call the method.
   1. If the API is well designed, there might even be a method that takes the strings directly from the servlet and does all the work including checking inventory and scheduling order fulfillment. This is the rare case.
   2. If the API is pretty good, then I'll probably have to make the appropriate calls to parse the servlet arguments to actual data, handle errors, and validate the input. Then, with the parsed arguments in hand I can call the purchase method.
   3. If the API is lame, then after the argument parsing I'll have to check the database to see if there is inventory and somehow reserve it. Then I'll run the purchase methods (which will not be unified into one abstraction, but consist of five methods that need to be called in the right order). Then I'll have to see if the purchase succeeded, and if not, return the reserved item to inventory, or if it did, arrange for the item to be delivered.

So how do I know how to use the API if it is undocumented? The simple answer is this: look for methods that have the same type signature as the objects that are at hand in the servlet. There probably won't be very many. Now look at the names of the methods and see if any make sense. The easyPurchase(User user, Item item) seems a lot more likely than the getReceipt(User user, Collection items). A really good API uses names that practically shout out their purpose.

Finally I try out the code and see if it works. If it does, I call it day. If not, then it is time to twiddle, tweak, and frob things. If the API is good, then I'll get informative error messages that tell me what I did wrong. Something along the lines of “Error: user id is an email address, did you forget to call parseUser()?” instantly tell me what I'm doing wrong. Something along the lines of “null pointer exception” followed by a stack trace filled with methods I never heard of before now tell me nothing whatsoever.

---

This is work. It isn't extremely difficult, but it does require practice, skill, and imagination. It also isn't really engineering. It is more like plumbing. Get this stuff here and put it over there, and you might need an adapter fitting.