Rail Road Diagrams

I've been wanting to promote the usage of the Lua programming language, so I thought it would be nice to provide visual diagrams of the syntax to help new people who would not take the time to read BNF descriptions of the language. When I looked online for a solution to this problem, I found this description of how sqlite generates railroad diagrams by using a Tcl/Tk script. I also found these nice CSS diagrams which are "generated" with power point.

The Tcl/Tk solution is neat, but trying to describe the railroad diagrams in Tcl/Tk was difficult, and I don't have easy access to Tcl/Tk. The powerpoint solution was also non-optimal since power point does a poor job with these sorts of tasks.

So, I figured this might be a great way to showcase the power of Lua (with the LuaCairo binding). I ended up writing this railroad diagram generator. Note that this railroad diagram "generator" requires you to fully specify your "railroads", which direction they are flowing and all the "entrances" and "exits", so it isn't particularly easy to use at the moment (ideally it would just parse BNF to generate the diagrams). It also has a fair number of hacks that I still need to work out before I really "publish" this, but the above picture is my first working rail road diagram.

This diagram is for the "chunk" production described in the Lua Manual. Here is the equivalent BNF production:

chunk ::= {stat [`;´]} [laststat [`;´]]

Note that the yellow in the diagram is to demarcate "terminals" in the syntax. I hope to later on do the complete syntax with links in the grammar to the other diagrams (and perhaps have the "terminals" link to the description of what they mean in the manual?

Lua Web Debugger

Lately I've been doing a lot of dabbling in Lua in my spare time. One of the ideas swirling around in my head is to have a debugger that you could run on a production webserver. Running any debugger in production is a difficult thing since your webservers are normally behind some kind of load balancer, so you don't know what machine your request will be routed to, there is also the possibility of blocking out users of your website, opening up your website to denial of service attacks, and security vulnerabilities.

To overcome these problems, there could be a central "debug service" website that is only accessible to the staff. Once you login to that website, you generate a cookie with a unique debug session id. On this website you could then set break points in production, and the production webservers could check for this debug session cookie. If the debug session cookie is valid, then appropriate break points get set on the request. If a break point is reached, then the current request thread (which must be unique to the request) does a blocking service call to the central "debug service" with information about local variables and the state of the stack, which then pushes this information back to the debugger window, which shows the source code.

Ideally you would be able to set break points in the C code and in the Lua code, and be able to step into a Lua function which happens to be implemented in C. You should also be alerted of break points that are triggered when you are in the middle of stepping through code. These break point alerts could happen if your web browser makes multiple requests to your production webserver.

Another super useful tool would be a profiler that is activated by a debug session cookie. After sending the response back to the client, a call graph that contains information about how much time is spent in each function could be sent back to the debug service which would then display it.

Similarly, a memory profiler could be created that records a snapshot of the stack whenever garbage collection is triggered along with the ability to tune how often the garbage collection happens. This would allow you to determine hot spots in memory usage that you could then set break points on. You could then use the memory inspector feature of the debugger to see what types of objects are in memory based on the metatable for that object. Ideally you would be able to sort on size of largest object(s) and by size of all objects of a given type.

Now I just need to find the time to write this tool ;-).

An HTTP "rules engine".

I'm seriously contemplating the creation of yet another web server... actually, this isn't entirely true. What I really want to do is make it easy to connect a few components together in lua so that anyone can create the web server of their choice. Today it is pretty easy to use lua-ev in combination with luasocket and lua-http-parser in order to write a web server. However, lua-ev is still a little rough around the edges. In particular, I would like to do these improvements when time allows it:

• Support for coroutines so that the event loop can implement "green threads".
  
• Support for "error backs" so that an event loop callback that throws an error can recover from that error (like send a 400 bad request back to the client if there is an error during HTTP parsing, or an error happens in the content handler such that we need to create a 500 internal server error).
  
• Support the "proactor" design pattern such that it is trivial to register a callback that will be ran with the results of a read operation (and if the read fails or if a time limit is exceeded, the "error back" chain would be ran with an appropriate error message).
  

However, the biggest missing piece in my mind is a way to enable people to "plug-in" to your web server. In essence, what I really need is an HTTP rules engine. For example, Apache mod_rewrite is really just a rules engine that toggles various pieces of the request object based on some regular expressions. What if this idea was extended so that the response can also be manipulated, and that you could have rules that depend on other rules? This rules engine could then be the primary point of extension for people that want to write "plug-in"s.

An HTTP Rules Engine

At a high level, the rules engine API allows you to define a set of rules that runs arbitrary lua code which can modify the "request object" or "response object". The rule can be triggered by request or response meta-data, and rules can define a set of dependencies (only run this rule if these other rules have already been ran). All incoming requests are pumped into the rules engine which then triggers a bunch of arbitrary lua code to be called which then assembles the response. To prevent infinite loops, a rule can only be executed N times per request-response cycle (where N is configurable, and this failure is non-fatal, but is logged).

The "request object" consists of: the request method, the requested url, the query string, the HTTP version, the http headers, the "cookies", the remote ip, and the input filter chain. The input filter chain allows anybody to register a lua function to filter the incoming request body through a function or to register one or more input handlers. If no input handler is registered, the request body is simply ignored. Once the input filter chain is put to use on the body of the request, the filter chain is immutable (any attempt to change it cause errors).

The "response object" consists of: the HTTP version, status code, reason phrase, http headers, and the output filter chain. The output filter chain allows anybody to register a lua function to filter the outbound response body through a function or to reserve one or more "chunks" of output (where the response meta-data represents the first chunk of output).

The "reservation system" makes it possible to generate the response while making parallel service calls to the back-end. For example, you can write a rule to generate a footer by reserving the last chunk, then immediately "emitting" the footer. Yet the body of the response can still be generated by another rule that reserves the "next chunk", and may depend on some back-end service call results. This reservation system also allows for AJAX push notifications.

So, what do you think of this HTTP "rules engine"? Do you think this API is complete? Are you wishing this API already existed?

Content Aggregation Platform

I've been doing web programming for more than 10 years, and it is always the same thing: grab data from disparate data sources and assemble them into a single webpage (or image) that is sent back to the browser.

Although that sounds simple, there are a lot of things that will make or break your web application. Things such as:

• Keeping accurate account of page latency (how long it takes to return the first byte, and how long it takes to return the last byte). [Performance]
  
• How many requests per second can your application handle? [Performance]
  
• How do you make requests for the disparate data sources in parallel? [Performance]
  
• How do you handle errors? [Administrable]
  
• Is there a debugger available? How do you debug your web application? [Development]
  
• How do you test your web application? [Development]
  
• If there is a programming problem, how do you capture stack-traces, and how do you go about reproducing the problem? [Administrable]
  
• How do you update your web application software with minimal impact? [Administrable]
  

Basically it comes down to three things: How well can your web application perform? How easy it is to administrate when problems happen? How easily can you evolve/develop your application?

Ruby on rails is a great example of a "platform" that makes it easy to evolve/develop your application, but performance is not very spectacular. I'm also not very keen on the idea of "batteries included" that the Ruby community seems to be embracing (the term was coined from Python).

One "platform" that looks promising is node.js. At the core is an event loop that has the ability to deliver performance on par with the lighttpd webserver. Also, since it is using V8 Javascript, web developers can re-use their skills and libraries on both the server and client side. The V8 engine is also very fast in comparison to Ruby. However, it is still a pretty young project, so it will be interesting to see how things evolve.

Before I knew about node.js, I wrote a binding to libev for lua so that I could learn more about both libev and lua. The two things that really attract me to Lua is the ability to use coroutines to adapt "threaded programming" to "event loop" programming, and the ability to easily create confined "sandboxes". The sandbox can be done by "creating a separate interpreter", or it can be done by making it so that only a small set of functions are defined when new code is evaluated in the interpreter.

To this end, I would like to eventually (don't have a lot of free time at the moment) create a "platform" for content aggregation that uses Lua to "glue" together a lot of well tested C libraries such as bsd sockets API, libev event loop, zlib compression, etc. Some attributes of this content aggregation that I would love to have are:

• Built-in support for "listening" on a debug port that allows you to step through the program using gdb and the Lua debug API.
  
• A web application that can connect to the debug port and present a web-based debugger. Ideally this application would be "just another application" on the "platform".
  
• All web applications would be ran in a "sandbox" so that:
  
  
  
  • All input/output can be "recorded" in order to diagnose problems. For example, if we get a SIGSEGV, we could store the "recording", then play it back in order to reproduce the problem. In this way, bugs could be fixed faster.
    
  • Module loading could be based on a git SHA1 tree identifier, making it so that different versions of your web application can be ran at the same time. This facility could be used to do A/B testing, or to make it so that software updates are done on a "session" by "session" basis.
    
  • Memory allocated "per transaction" can be limited.
    
  • Time spent executing a callback (before yielding back to the event loop) can be limited.
    
  
  
• The ability to intermix both "green threads" and "event loop" programming paradigms so that users of the platform can choose the style that works best for their needs.
  
• Support for "templates":
  
  
  
  • Templates can be edited on-line wiki style.
    
  • Templates only take an arbitrary data-structure as input.
    
  • Templates are heavily restricted with respect to what operations can be performed.
    
  
  

Why I Dabble In Lua

From time to time, people ask my what my motivation is to maintain the various lua-* modules on github (lua-yajl, lua-ev, lua-zlib, and lua-archive at the moment). However, before I answer that question, let me give a little bit of history.

Back in 2008 I was reading Josh Haberman's blog about brilliant programmers. One of the highlighted programmers was Mike Pall and the work he has done to improve Lua's performance by providing a JIT compiler for Lua. I also noticed that Josh was testing Gazelle with Lua, so I started to dabble a little.

One of the things that I liked the most about Lua was the C API. At that time I was also really interested in libev, so I decided to make lua-ev (at that time I was calling it evlua). This task proved to be a lot harder than I expected since I needed a way to callback to lua from an arbitrary C callback, and I really wanted to be sure that everything worked correctly with the garbage collector.

After this first experience, I started thinking of other uses for Lua which got me started in writing lua-zlib and lua-archive. In particular, I was thinking that Lua would be an awesome platform for solving the "deployment problem". In particular, I wanted to use lua to make it easy to run remote commands to install software, something similar to apt-get.

Another place where I think Lua would be awesome is to create a "content aggregater" and template rendering system. I'll describe this more in another post.

So, in conclusion, I don't have a lot of motivation to maintain the lua-* modules. It is just something that I do for fun. It is also my first attempt at trying to do open source... I just still need to figure out how to monetize on open source if I ever want it to be a full time job.

A Better Make

I love these features of GNU Make:

• Variables localized to a dependency graph.
• Automatic mtime checking to see if a target is out of date.
• Easy to append new prerequisites to a target.
• Macro expansion (although making this a default for "=" has probably confused a lot of people, and there are a lot of 'gotcha's).
• Order-only prerequisites (ability to assert ordering of targets, but not force a target to be rebuilt because the prerequisite was rebuilt).
• Ability to run targets in parallel rocks!
• Ability to model targets that generate makefiles.
  

Here are some things that are just plain frustrating about GNU Make:

• Dealing with special characters in filenames, anything from spaces to "%" to ":" is a royal PITA!
• Make needs a "real" syntax, not just a glorified lexer (similar as above).
• No easy way to model rules that generate multiple build artifacts.
• Challenging to make it portable.
• No support for local variables.
  

I've looked for alternatives, but most of the alternatives are missing out on at least a few of the things I love about GNU Make. Rake seems to have a lot of strong potential, if only it wasn't slow and had more of the features I love about GNU Make.

I've been contemplating writing yet another make system that uses Lua to describe the build process, and has a lot of stuff just "builtin" so that you can more easily write a cross platform makefile. I'm thinking, that the build phase will use libev so that multiple targets can be built at the same time without the complications of multi-threading.

In Appreciation of Wedding Videographers

I've been fascinated with video since high school (which is over 10 years ago for me). I even got the opportunity to shoot my first wedding when I was in high school. Back then SVHS was the "format of choice", and all editing was done manually by playing on one tape deck and recording from a different tape deck (although NLE's where just starting to make in entrance into the market).

Video has come a long way since then SVHS was replaced with DV which is now being replaced with high definition (go AVCHD!). We also now have decent computers that can store and play these higher bit-rate videos.

Recently I got an opportunity to shoot another wedding as a favor for my mother in law. At the wedding one of the guests thought that wedding videography was a lucrative business since it costs so much. This got me thinking, what would it be like if I went pro and shot wedding for ~$2,000 per wedding.

Weddings tend to be in the summer and fall, and the probability of finding a gig every week is pretty low. So, let us say that on average I was to land a job every other week. That would be 26 (52/2) per year. That means I would be raking in $52,000 per year, better than working fast food, but not even half of my current salary.

Now let us look at the expenses of this business:

• Day of wedding requires at least one other cameraman. I'm guessing that a minimum wage for this job is $50/hour because you need to hire someone skilled. Why do you need another cameraman? Equipment fails, weddings happen only once. If for no other reason, you must use two cameras (and have someone man them) to reduce your liability. So there is about $10,400 subtracted for hired help.
  
• Equiptment costs:
  
  
  
  • Digital Audio Recorder: minimum of $350
    
  • Lavaliere Microphone: minimum of $600 x 2 = $1,200. Need two for the same reason you need to cameras. Why external microphone? The most important element of good audio acquisition is proximity. Being able to hear the wedding vows is probably the most important part of the entire production, and the onboard camera microphones will never be close enough to the bride or groom to be effective.
    
  • Prosumer Quality HD Cameras: minimum of $2000 x 2 = $4,000. Consumer quality cameras simply don't have the manual controls that allow you to control basic things like focus and depth of field. They also don't do good in the low light "mood light" that people like to have at their weddings. They also tend to have lower dynamic range which makes it difficult to capture any details in the white wedding dress next to the black suites.
    
  • Editing Software: $1,500 (Final Cut Pro)
    
  • Editing Hardware: $2,500 (Mac Pro) I've been using my laptop, for my project, but my render times for a 45 minute long wedding is on the order of 20 hours.
    
  • Accessories: ~$1,000 (2x tripods each at least $250, Media for the cameras, external capture card, external monitor(s), etc)
    
  
  

So, initial capital expenditure is ~$10,550, let us round that up to $11,000. I'm guessing that this equipment has a depreciation schedule of 3 years, so that is about $3,500 per year in equipment expenses. There is also the option of renting, but then you would potentially be working with unfamiliar equipment. Note that there is also probably some repair expenses to.

$52,000 - $10,400 - $3,500 = $38,100

Now let us look at all the other work involved. There is the work on the day of wedding, then there is the time needed to edit it all together. I'm guessing once you got this process down you could do the editing in a 40 hour work week, there is also the time needed to make plan and negotiate with each client (minimum of 8 hours work?), the time to actually be at the wedding (average of 8 hours, so we can do the pre-wedding pictures/video, actual wedding, reception, time to drive to and from wedding site), there is also the time and money needed to drum up business and advertise. I'm guessing at minimum this would take up 4 weeks per year (160 hours), and cost a few thousand (get a booth at the yearly wedding expo, flyers printed, website, etc).

With these additional expenses taken into account we are down to $36,100 for working 160 (advertising per year) + 26 (weddings per year) x 56 (average time per wedding) = 1616 hours per year. This means I would be getting a wage of about $22.33 per hour.

My extra camera man would be making more money per hour than I would be!

Note that this analysis also didn't take into account the fact that people often want popular songs in their wedding videos, which if properly licensed costs $1,000's of dollars per song per video! I could not properly license them, but at the added risk of being sued. How do professional videographers deal with this problem?

One other aspect of this business that I don't like is that I'm gone away from my family every other weekend.

So, if confronted with paying $2,000 for having a wedding video, be thankful! Also note that it is a worthwhile investment if shot and edited by a pro.