Designing Applications in a Vacuum (don’t do it)

Throughout my career as a software engineer/consultant, I’ve run across many passionate arguments for software development methodologies. I can’t count the times I’ve been in a room with very loud discussions about placing business logic in code or in stored procedures. I think it’s safe to say that the coders won this argument and I don’t hear these discussions anymore.

There have been others….Java vs. .NET, Fat Client vs. Web, SQL Server vs. MySQL, OO vs. Procedural and so on.

There is one I’ve hear recently that’s relatively old (10 years) that has some traction, but I don’t think it’s by any means a standard. This is Eric Evans’ Domain Driven Design philosophy (there’s a book on it). Admittedly, I’ve built many systems starting from the data model and more recently looked at API First. I’ve often developer object models and domain models as a part of my process, but I never really focused on that aspect as a required first step.

With the prevalence of web applications and specifically REST, Web API, and AJAX oriented implementations, I think I’ve come to a working conclusion. None of these are starting points (API, Domain, Data). They are deliverables.

You may work between each of these deliverables as you design an application, but you will learn something different and come up with different questions when you touch each area of design.

When you touch the domain, you’ll be asking resource and behavior questions. When you touch the API, you’ll be asking extensibility and shared service questions. When you touch the data model, you’ll be asking questions about reporting, atomic transactions, and performance.

These are all very important questions and by starting at a single point, you could potentially ignore important questions and thus, important design decisions.

Corporate Technical Architecture and the Missing Link

I’ve had an obsession with technical architecture from the moment I started typing on a computer terminal back in the late 70’s. Over the course of my 27 year professional career I’ve had many opportunities to be involved in architectural discussions. In recent years I have led these discussions or been a part of the leadership.

I shouldn’t say discussion because my viewpoint almost always garners derision and fuels heated battles over the makeup and purpose of a new technical architecture. I’ve thought long and hard about the things I value in a new architecture over what other (most) architects value. I’ve struggled to articulate my vision and because I’m not as strong technically as others, I generally lose battles quickly and badly.

But I do know what I’m talking about and the other architects keep proving me right. Here’s why…

Almost every architecture team focuses on technical vision and business requirements. So if a company needs to be able to share services across a number of silo’d departments or divisions, this will require a certain type of disconnected and service-oriented architecture. At a high level, this is good and right. But everything goes haywire in the implementation because these same technologists decide to find the most technically elegant solution available.

What most architects fail to take into consideration, or simply deride any attempt to “dumb-down” their goals, is that the results of their work have an associated future-cost and could potentially damage the business. This future-cost/business risk is defined by the level of talent required to implement and maintain the proposed architecture.

The missing piece is the HR department and the market for available technical talent. When implementing a new technical architecture, the architecture team should be forced to sit down with HR personnel and look at current and projected levels of talent in the marketplace and their associated costs. If a proposed implementation requires senior level talent, then HR should be able to offer insight into current and future costs associated with that proposed implementation from a building and maintenance perspective.

“This year, the proposed implementation will cost the company $2,000,000. Maintenance costs will increase to $4,000,000 because the level of talent required to maintain the proposed architecture will likely come from the contractor world and not from full-time employees. We propose you lower the complexity of the implementation so that a mid-level developer can maintain the system. This will open availability and lower future costs to $2,500.000.”

This is the sort of conversation that is missing from every architecture discussion and needs to change. I’ve sort of realized that if I were to carve out a position in a firm, this is where I would excel. I would be able to work between HR and the technical team to monitor proposed implementations and their future costs and complexity.

It’s about time this position become standard in all mid to large corporations.

JavaScript, jQuery, and UI Frameworking

In my professional consulting career, I’ve been avoiding the UI area of software development for years, much like I tend to avoid reporting tools like Crystal Reports or Business Objects. I like poking around in the guts of systems. Everything below the surface is so much more interesting, or so I thought. Instead of working on UI layers, I tried to hire other people to do what I thought was fairly simple work. This wasn’t working. It turns out, in order to direct hired hands, you need to know what the hell you’re asking them to do. So I backed off having other people do UI work and realized I just need to strap in and learn it on my own. This includes iOS, JavaScript, jQuery, and anything else to do with building user interfaces. It turns out, there’s a lot to learn.

I’ll talk about iOS in a future blog entry, once I get comfortable with iOS, Objective C, and all the related tools. Today I’m going to talk about JavaScript and jQuery.

In a side-project called Zifmia, I’m developing a service/cloud based game engine for Interactive Fiction. I’ve been working on this in pieces for a long time. It’s morphed from WPF, to Silverlight, to Java, to iOS, and I finally realized that if I could create a service, I don’t need to port it anywhere anymore. Well, except maybe the Kindle since the Kindle doesn’t support connected services. (This makes sense. Most people use the Kindle for reading and are only concerned about connectivity when purchasing and downloading a new book or other content.) But for the web, smartphones, the iPad, iPod, and other tablets, a service/cloud engine is really the right way to go.

I start developing the service layer using RESTful web services. I used the Microsoft .NET WCF REST 4.0 template to provide the external layer for the service side functionality. This was great since WCF 4.0 REST works nearly perfectly out of the box. There’s very little “tuning” to do to make it work locally or on a server. The one major issue I had was with cross-domain ajax calls, but that’s solved by adding a header on all return calls from the server. The service allows a consuming application to register a user, login, start a session (game), send commands to a session, and various other management tasks. I plan to implement oAuth so that the service can be embedded in Facebook properly.

I then start playing with various platforms to consume the service. I played round with Windows Phone 7 (Silverlight) and that was drop dead simple. It takes an object definition and two lines of code to consume the service. I didn’t implement a full Application, but it helped work through some of the service prototyping issues.

I played around with iOS, but stalled out because I wanted the web interface completed first. I had a very committed developer working on it, but our communication wasn’t working well, and so I set that relationship aside for the time being. (I haven’t forgotten about you Dan…hope to catch up soon). I then started to work on the client side JavaScript code and realized there was a LOT of work that needed to be done before anyone could use Zifmia properly. The result of that work is the essence of this article.

I started developing the interfaces in jQuery and JavaScript, but soon realized the code (HTML, CSS, JavaScript, jQuery) was becoming quite convoluted and unreadable. I also realized it would only get worse if I continued. So I started asking around and researching how JavaScript/jQuery based client user interfaces should be organized. The surprising answer is that although there are a handful of serious solutions, there isn’t a lot of open thinking about how this should be done. The people that understand the need and have the ability to work through the problems have created tools, but haven’t really shared the process with the public. It’s a shame, since that’s the really important part to share, not just the code.

I looked at a few of these tools (like JavascriptMVC, Backbone.js, SproutCore) but found all of them to have aspects I dislike. Complexity, dependencies, and high learning curves. Instead of adopting, I chose to develop my own, trying to work through the issues. Here is what I came up with. This is a very simple methodology for separation of concerns in a javascript UI framework.

The first order of business is to identify assumptions:

• We will work on the concept of a single page interface.
• The page will have its own class/namespace.
• The page will have a master HTML arrangement (header, body, footer).
• The body will be a View container.
• Each view will be based on an HTML template separate from the main html page. The HTML template will use normal HTML, CSS, and JavaScript.
• Each View will have its own Controller or Presenter class.
• All AJAX calls will be contained in a single Service class. (This is arguable, since a more complex web application could have highly distinct groups of service calls. More than one Service class could be adopted).

The result is a very simple and effective way to separate code in a single page framework jQuery driven website.

Create a web page called index.html similar to the one below:

<html>
<head>
    <title>Zifmia></title>
    ....
</head>
<body onload="javascript:$MyWeb.onLoad();">
    <div id="outerPanel">
        <div id="ajaxLoading"><img src="images/loading.gif" alt="Loading" /></div>
        <div id="headerPanel"><p>Textfyre Presents, A Zifmia+Web Portal (Beta)</p></div>
        <div id="viewContainer"></div>
        <div id="footerPanel">Copyright @ 2011 Textfyre, Inc. - All Rights Reserved</div>
    </div>
</body>
</html>

In the head section of the html page, create a class and create an instance of the class as MyWeb:

var $MyWeb = new mySite();

function mySite() {

    this.controller = null;

    this.username = getCookie("username");

    this.onLoad() = function() {
        // Initialize page here.

        // based on your page logic, cookies, etc, determine the initial state of the page
        // this is where you would determine which view to show, or to show the default
        // view. Based on whatever view is needed, we set that controller to the page
        // controller and call init(), which should be a standard function on all
        // controllers.

        if (username != "") {
            this.controller = new sessionController();
        } else {
            this.controller = new defaultController();
        }

        this.controller.init();
    };

}

Since I assume you have a design in mind, you should be able to separate the various views on your own into html templates. Each template might look something like this:

<div id="loginPanel">
    <div style="font-size: 14pt; font-weight: bold;">
        Zifmia Login Form<span id="loginMessage" style="font-size: 10pt; font-weight: normal;
            margin-left: 20px;"></span></div>
    <hr />
    <table>
        <tr>
            <td>
                <span style="white-space: nowrap;">
                    Username:
                    <input class="login" id="zLogUsername" type="text" onkeyup="javascript:validateRequiredField(this);" /><span
                    id="zLogUsernameError"></span></span>
                <span id="nickNameField" style="margin-right:10px;font-weight:bold;">
                </span>
            </td>
            <td style="white-space: nowrap;">
                Password:
                <input class="login" id="zLogPassword" type="password" onkeyup="javascript:validateRequiredField(this);" /><span
                    id="zLogPasswordError"></span>
            </td>
            <td style="white-space: nowrap;">
                <a id="zLogSubmit" href="#login" onclick="javascript:$MyWeb.controller.login();return false;">login</a>
            </td>
            <td style="text-align: right; padding-left: 20px;">
                <span id="registerNotice">If you don't have an account, please <a href="#showRegistration"
                    onclick="javascript:$MyWeb.controller.showRegistration();return false;">register</a>.</span>
            </td>
        </tr>
    </table>
</div>

This html should be stored in a subdirectory called templates with the name loginTemplate.html. We can use a simple AJAX call to retrieve it:

    var loginTemplate = getTemplate("templates/loginTemplate.html");

    function getTemplate(templateURI) {
        return $.ajax({
            url: templateURI,
            global: false,
            type: "GET",
            async: false
        }).responseText;
    }

The getTemplate function should be placed in a utility class or in the page controller for easy reuse. From here’s simple to create a controller class to display a view template in the correct place in our html page:

function introController() {

    var REG_USERNAME_FIELD = "zRegUsername";
    var REG_PASSWORD_FIELD = "zRegPassword";
    var REG_NICKNAME_FIELD = "zRegNickname";
    var REG_EMAIL_ADDRESS_FIELD = "zRegEmailAddress";

    var LOG_USERNAME_FIELD = "zLogUsername";
    var LOG_PASSWORD_FIELD = "zLogPassword";

    var viewContainer = "#viewContainer";
    var usernameField = "#zLogUsername";
    var nicknameField = "#nickNameField";
    var regErrorPanel = "#regErrorPanel";
    var loginMessage = "#loginMessage";

    this.introTemplate = $Z.getTemplate("templates/introTemplate.html");
    this.loginTemplate = $Z.getTemplate("templates/loginTemplate.html");
    this.registrationTemplate = $Z.getTemplate("templates/registrationTemplate.html");

    this.init = function (message) {

        $(viewContainer).hide();
        $(viewContainer).html($(this.loginTemplate).html());

        var zLogUsername = document.getElementById(LOG_USERNAME_FIELD);
        var zLogPassword = document.getElementById(LOG_PASSWORD_FIELD);

        setRequiredValidator(zLogPassword);

        if (message != null && message != "") {
            $(loginMessage).text("(" + message + ")");
        } else {
            $(loginMessage).text("");
        }

        if ($Z.username != "") {
            //
            // We know the username and nickname, but the user is logged out.
            //
            $(usernameField).hide();
            zLogUsername.isValid = true;
            $(nicknameField).show();
            if ($Z.nickname != "") {
                $(nickNameField).html('<a href="#changeUser" onclick="javascript:$MyWeb.controller.changeUser();">' + $Z.nickname + '</a>');
            } else {
                $(nickNameField).html('<a href="#changeUser" onclick="javascript:$MyWeb.controller.changeUser();">' + $Z.username + '</a>');
            }
        } else {
            //
            // We don't know anything about the user.
            //
            setRequiredValidator(zLogUsername);
        }

        $(viewContainer).append($(this.introTemplate).html());
        $(viewContainer).show();

    };

    this.changeUser = function () {
        $Z.username = "";
        $Z.saveSessionData();
        this.init();
    };

    this.login = function () {
        var zLogUsername = document.getElementById(LOG_USERNAME_FIELD);
        var zLogPassword = document.getElementById(LOG_PASSWORD_FIELD);

        $Z.login(zLogUsername.value, zLogPassword.value,
                function (zifmiaLoginViewModel) {
                    if (zifmiaLoginViewModel.Status == "0") {
                        $Z.username = zRegUsername.value;
                        $Z.authKey = zifmiaLoginViewModel.AuthKey;
                        $Z.nickname = zifmiaLoginViewModel.Nickname;
                        $Z.saveSessionData();
                        $ZWeb.onLoad();
                    } else {
                        $(regErrorPanel).text(zifmiaLoginViewModel.Message);
                        $(zLogUsername).focus();
                    }
                },
                function (xhr, textStatus, errorThrown) {
                    $(regErrorPanel).html(xhr.responseText);
                }
            );
    };

    this.showRegistration = function () {
        $(viewContainer).hide();
        $(viewContainer).html($(this.registrationTemplate).html());

        var zRegUsername = document.getElementById(REG_USERNAME_FIELD);
        var zRegPassword = document.getElementById(REG_PASSWORD_FIELD);
        var zRegNickname = document.getElementById(REG_NICKNAME_FIELD);
        var zRegEmailAddress = document.getElementById(REG_EMAIL_ADDRESS_FIELD);

        setRequiredValidator(zRegUsername);
        setRequiredValidator(zRegPassword);
        setRequiredValidator(zRegNickname);
        setRequiredValidator(zRegEmailAddress);

        $(viewContainer).show();
    };

    this.register = function () {
        $(regErrorPanel).text("");
        var zRegUsername = document.getElementById(REG_USERNAME_FIELD);
        var zRegPassword = document.getElementById(REG_PASSWORD_FIELD);
        var zRegNickname = document.getElementById(REG_NICKNAME_FIELD);
        var zRegEmailAddress = document.getElementById(REG_EMAIL_ADDRESS_FIELD);

        $Z.register(zRegUsername.value, zRegPassword.value, zRegNickname.value, zRegEmailAddress.value,
                function (zifmiaRegistrationViewModel) {
                    if (zifmiaRegistrationViewModel.Status == "0") {
                        $Z.username = zRegUsername.value;
                        $Z.nickname = zRegNickname.value;
                        $Z.saveSessionData();
                        $ZWeb.controller.init(zifmiaRegistrationViewModel.Message);
                    } else {
                        $(regErrorPanel).text(zifmiaRegistrationViewModel.Message);
                        $(zRegUsername).focus();
                    }
                },
                function (xhr, textStatus, errorThrown) {
                    $(regErrorPanel).html(xhr.responseText);
                }
            );
    };
}

Note there are many other things happening in this controller including ajax calls and handling user interactions. The view is shown to the user through a jQuery call (which can be done with simple DOM manipulation):

        $(viewContainer).hide();
        $(viewContainer).html($(this.loginTemplate).html());
        $(viewContainer).show();

So that’s it. You have a main controller in your index.html page that can be access by using $MyWeb at any time, you have varied controllers that are at $MyWeb.controller that have methods that can be called from your html. All of your HTML is in separate files for easy editing and reuse. This is a simple, yet effective way to separate and manage your AJAX/jQuery/JavaScript enabled website.

Moving Files from XAP to Isolated Storage for Local HTML Content on Windows Phone 7

For the past few months I’ve been working on porting The Shadow in the Cathedral to Windows Phone 7. Chris Cavanagh has been doing the bulk of the UI framework implementation while I’ve been working on adding all of the extras like help, hints, and other “nice touches”.

Adding documentation to a WP7 app is tricky. You can use PDF’s, but that seems shoddy to me. You can lay out tons of pages of large text for the user to page through in XAML, but this is painful and you don’t get the multi-touch and zoom capabilities by default.

Then you discover the WebBrowser control and you think you’re home free. Just make some web pages for your documentation and plug them into the phone application, right? Wrong. It’s not that simple quite yet. WP7’s SDK is still a work in progress and only a ton of usage will refine its design to a point where developers can do most things easily.

The WebBrowser control is currently meant for reaching the Internet. It has the capability of serving local content, but only from IsolatedStorage, not from content or resources in your XAP file. At least not yet. I’m not sure what the reasoning is behind this, but it is what it is.

So in order to serve your fancy html documentation, you need to move it to IsolatedStorage from your XAP file sometime when your application executes. I spent a little time this past week developing a couple of extension methods to help do exactly this.

    public static class ISExtensions
    {
        public static void CopyTextFile(this IsolatedStorageFile isf, string filename, bool replace = false)
        {
            if (!isf.FileExists(filename) || replace == true)
            {
                StreamReader stream = new StreamReader(TitleContainer.OpenStream(filename));
                IsolatedStorageFileStream outFile = isf.CreateFile(filename);

                string fileAsString = stream.ReadToEnd();
                byte[] fileBytes = System.Text.Encoding.UTF8.GetBytes(fileAsString);

                outFile.Write(fileBytes, 0, fileBytes.Length);

                stream.Close();
                outFile.Close();
            }
        }

        public static void CopyBinaryFile(this IsolatedStorageFile isf, string filename, bool replace = false)
        {
            if (!isf.FileExists(filename) || replace == true)
            {
                BinaryReader fileReader = new BinaryReader(TitleContainer.OpenStream(filename));
                IsolatedStorageFileStream outFile = isf.CreateFile(filename);

                bool eof = false;
                long fileLength = fileReader.BaseStream.Length;
                int writeLength = 512;
                while (!eof)
                {
                    if (fileLength < 512)
                    {
                        writeLength = Convert.ToInt32(fileLength);
                        outFile.Write(fileReader.ReadBytes(writeLength), 0, writeLength);
                    }
                    else
                    {
                        outFile.Write(fileReader.ReadBytes(writeLength), 0, writeLength);
                    }

                    fileLength = fileLength - 512;

                    if (fileLength <= 0) eof = true;
                }
                fileReader.Close();
                outFile.Close();
            }

        }
    }

The next step is to add all of your content (html, images, etc) to your project and set the Build Action on every file to Content.

Then execute steps to create your directories and copy your files.

            storageFile.CreateDirectory("HTDocs\\images");
            storageFile.CopyTextFile("HTDocs\\index.html", true);
            storageFile.CopyTextFile("HTDocs\\about.html", true);
            storageFile.CopyTextFile("HTDocs\\games.html", true);
            storageFile.CopyTextFile("HTDocs\\support.html", true);
            storageFile.CopyBinaryFile("HTDocs\\images\\image1.png", true);
            storageFile.CopyBinaryFile("HTDocs\\images\\image2.png", true);
            storageFile.CopyBinaryFile("HTDocs\\images\\image3.png", true);
            storageFile.CopyBinaryFile("HTDocs\\images\\image4.png", true);
            storageFile.CopyBinaryFile("HTDocs\\images\\image5.jpg", true);

The first command “CreateDirectory” is one of the existing methods in the IsolatedStorageFile object.

The CopyTextFile and CopyBinaryFile are the extension methods I’ve added to simplify moving Content files to IS.

In order to serve your content in a WP7 WebBrowser control, you simply set the Base and call the first page.

        private void webBrowser1_Loaded(object sender, RoutedEventArgs e)
        {
            webBrowser1.Base = "HTDocs";
            webBrowser1.Navigate(new Uri("index.html", UriKind.Relative));
        }

In your html, images and other files are relative to the Base. I have not tried JavaScript or CSS files, but I assume they will work the same as images.

In order to make your HTML compliant with the WP7 browser, make sure you have a DOCTYPE tag.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

Also make sure you set the viewport in a meta tag, otherwise your HTML will shrink. I think the default viewport is 1000px wide and it’s just easier to set it to 460px wide and work from there. Make sure you manage your margins too.

    <meta name="viewport" content="width=460, user-scalable=no"/>

Now you have everything you need to develop HTML based local content in your Windows Phone 7 application.

Programming in Grades

(this post is a work in progress, but I’m sharing it now to think about the grading more)

As an architect and a developer I have often been frustrated by the gap in understanding between what I call “rock star” programmers and “average joe” developers. To start, let’s define who’s who:

Rock Star Developer
The rock star developer is the guy that eats code in all forms at any level of complexity. He can read it, decompile it, reconstruct it, and communicate it to others. What he may not be able to do is understand the big picture and how code relates to a business. If he does, then he’s a developer God and there are an infinitely small number of those people.

Average Joe Developer
The average joe developer is the guy that writes code for a living, but would very likely rather be doing something else. He has a life that doesn’t include much time on the computer beyond his work day. He’s not going to seek new knowledge. He only grows when internal systems and standards change and he’s forced to learn new things, which he will do happily, but not ambitously.

Having been in this business for 25 years (many as a consultant), I have witnessed many IT environments. It’s my belief that the number of Rock Star developers is a relatively small one. By far, the vast majority of developers are Average Joe’s.

This presents a very serious problem in most IT departments in that the minority is in control of the standards and architecture for the majority. I can’t tell you how many times I’ve witnessed the development of a “framework” that is great on paper, sells well to the business, but no one ever asks the Average Joe’s what they think. Everyone just expects that they will learn it and move on.

The problem is that this will stress out most Average Joe’s and they will become less productive. If they can’t grasp the new “framework”, they’re likely to take months if not an entire year to learn the environment.

So then you’ll hear the rock star’s say, “This isn’t that hard. All good developers should know this stuff.” and will convince management to hire consultants or more senior developers. This creates an imbalance where too many cooks are in the kitchen and egos will start to come into play. IT departments are no different than a carefully developed fish tank and its ecosystem. If you have too many rock starts, things will get unmanagable. If you don’t have any, standards will be tossed aside and you’ll have as many styles as you do programmers.

So there are a couple of problems here. One is to balance out the number of rock star developers with average joe developers. The other is, how do you determine who’s a rock star and who isn’t. So I’ve come up with a grading system to define what I think they are and you can chime in with your thoughts. We’re going to categorize various development tasks with a grade level (as in 1st grade, 12th grade, associates degree, bachelor’s degree, masters, doctorate. We’ll keep the basics at 4th grade. These are the most basic tasks a programmer needs to be productive. The scope of this grading system is strictly business application development. There are other areas of development, such as programming computer chips, that are so different that they need their own grading system. This also ignores communication skills like speaking/writing English.

4th Grade Developer Tasks

• G4-01: Ability to use a Windows, Linux, or Apple OS X computer, including log in, run programs, use a command line, print documents, e-mail, instant message, and use a text editor.
• G4-02:Ability to use the file system to locate, list, and manage files.
• G4-03:Ability to write a Hello, World! program in at least one programming language without a book.

6th Grade Development Tasks

• G6-01: Ability to use a platform specific Integrated Development Environment such as Visual Studio or Eclipse.
• G6-02: Ability to create a basic web page with a form that posts data to a server.
• G6-03: Ability to handle server side logic of a given web page in at least one programming langauge.
• G6-04: Ability to handle querystrings and form variables.
• G6-05: Ability to access a database using at least one programming language.
• G6-06: Ability to use data from a database to output to a web page.
• G6-07: Ability to write simple CRUD SQL statements.

8th Grade Development Tasks

• G8-01: Ability to create a class with private and public members.
• G8-02: Ability to create SQL Statements with one JOIN.
• G8-03: Ability to create Stored Procedures and Views.
• G8-04: Ability to write a Functional Requirements document.

10th Grade Development Tasks

• G10-01: Ability to create protected and static members on a class.
• G10-02: Ability to create an abstract or base class with virtual members.
• G10-03: Ability to create a sub class with overriding members.
• G10-04: Ability to develop a complete CRUD application in at least one programming language, including the data model, the object model, a data access layer, screens, web pages, with validation, exception handling, logging,
• G10-05: Ability to create SQL statements with multiple JOIN’s.
• G10-06: Ability to create database Functions, Indexes, and constraints.
• G10-07: Ability to develop a simple data access layer.
• G10-08: Ability to designa a simple domain model.
• G10-09: Ability to design a simple data model.
• G10-10: Ability to write a Detail Design Document.
• G10-11: Ability to write a unit test class.

12th Grade Development Tasks

• G12-01: Ability to develop polymorphic class structures.
• G12-02: Ability to develop custom generic types.
• G12-03: Ability to diagnose database performance issues.
• G12-04: Ability to design a complex domain model.
• G12-05: Ability to design a complex data model.
• G12-06: Ability to develop by writing unit tests first.

BS Level Development Tasks

• GBS-01: Ability to create and use complex data structures.
• GBS-02: Understand and use big O notation.

Masters Level Development Tasks

• GM-01: Ability to develop a compiler for an existing programming language.
• GM-02: Ability to develop a compiler for a new programming language.

PHD Level Development Tasks

Architecting From the Bottom Up

I’m sure I’m going to steal someone’s cheese with this one, but I have very strong feelings about how systems are developed and who is involved in the process.

It’s been my experience that most companies employ someone in the Chief Architect role. This person interacts with the stakeholders to determine short and long-term technical needs, works with managers to integrate architectural change through internal product development, and above all else, let no man enter the Ivory Tower.

For whatever reason, many really smart (I mean really really smart) developers make their way into the role of Chief Architect without ever understanding who their audience is. They assume their audience is the IT Director, the Board, the principals, and all managers. It’s my contention that this is entirely wrong. The main audience for architecture is the developer. And not just any developer. I’m talking about Joe Average Developer. I’m not talking about the clock-watching consultant who tries to get his straight-eights in. I’m not talking about the workaholic senior developer who, if he (or her – no gender slights meant) doesn’t understand something, will draw his own blood in nightly rituals of self-abuse in order to gain an understanding. I’m talking about the developer that comes to work, gets handed an assignment, and wants to do a good job. They know they’re not likely to be a senior developer. They’re probably going to fork off into business analysis or project management at some point. But for now, they’re a developer. Heck, they may actually be an actor or singer who happens to know how to code. Coding pays the rent.

Back to our architect. The architect also assumes that they know what’s best for the company. They may claim they have no prejudice but believe me, they do. We all do. Out of the gate, all architects are suspect zero. We all have an agenda, we all like things a certain way. In most cases, the architect will arrange the pieces on the board that makes us feel the most comfortable. We will set out on a relentless quest to see a technical environment come to life that bears our heart and soul. We do this because we know that we can be successful in such a world. Well, actually success isn’t the motivator. That’s an illusion. We want to control our environment. That is our primary goal and in many cases, the architect doesn’t even understand their own primary goal.

However, there is another way. I see the lead architect role with four primary responsibilities.

First, there has to be strong communication with all levels. The architect should know the stakeholders, the analysts, the developers, the testers, the infrastructure staff, and the users. They should have strong communication networks with all of these people (and probably more). If you have an architect that likes to hide, boy do you have problems. I’m not talking about someone who’s digging in to get work done. I’m talking about someone that you’re never quite sure what they’re doing.

Second, there has to be a strong set of standards written down, introduced at all levels, discussed at all levels, and monitored at all levels. This may include code reviews, static code analysis, check-in policies. Developers like the freedom to write code, but they don’t like everyone doing it differently. In this respect, developers are a lot like soldiers. If you give them strict guidelines, they will find the art of their skill in being productive. They will be better for it and so will your company. If standards are given short-shrift or treated as an aside, your developers will be unhappy, less-productive, and very likely to switch jobs after the next disappointing pay-raise.

Third, all new technical requirements should be developed with all hands on deck. That means that developers should have a say in any newly developed framework. Everyone should be given the opportunity to provide technical ideas and in fact, the entire process should be presented in such a way that developers are encouraged and rewarded for their input. The less an architect listens to the developers, the more costly and time-consuming any and all new projects will become.

Fourth, and this is probably the most important. Keep it simple. I’ll say it again. Keep it simple. I don’t care if you have three PhD’s in math, engineering, or computer science. You are not going to be writing code. The developers, Joe Average Developer, are going to be writing code. So if your plan is to implement a complex set of requirements as a framework, you have failed before one line of code is written in your new framework. If you have an architect that constantly talks about object-relation mapping, object containers, dependency injection, separation of concerns, and abstraction layers, you have an architect that will create a set of middleware that no one, and I mean no one, will understand. If he gets hit by a bus, you’re screwed.

I have a rule. It’s a pretty simple rule. Take any framework or middleware or API. Grab any Joe Average Developer off the street and ask him or her to create a small data entry application with three or four related tables or a handful of related classes. It doesn’t matter if they start at the domain level or the database level. They get the same assistance that anyone else does on staff. If there’s training or pair programming in the regimen, that’s great. Here’s the rule. If Joe Average Developer can’t deliver a simple application in less than two weeks (and that’s a very forgiving schedule in my book), then you have completely failed at creating a productive and cost-effective framework. If you can’t get them to create a screen or report or web page in a few days, you’ve really messed up.

Some of the arguments I’ve heard over the years include:

“We know there are average developers, but we’re going to segregate them off into simpler jobs, like UI development.” – This is great for new development. What happens when someone has to maintain the code and all the smart people are missing?

“This isn’t that hard. I find it hard to believe that a good programmer won’t understand the framework.” – This is hubris. Don’t smart people know they’re smart? I don’t like to be an elitist but I’ve been around this business for a long time. There are many different kinds of developers, but I can assure you, only a handful will reach architect status and only a few those actually know design patterns. You can’t expect to hire all architects and senior developers. It’s just never going to work.

“We (meaning the smart people) have researched all of the requirements thoroughly. We understand all of our strengths and weaknesses. We know what’s best.” – By your divine right, you shall (cough) succeed. In the short-term maybe….let’s see your framework and code base in two years.

In summary, developers are the meat of any company that relies on computer systems for its products and services. Instead of putting one person in charge of the function of your business, doesn’t it make more sense to have everyone involved? Doesn’t it make more sense to rely on simplicity? Doesn’t it make more sense to create a technical environment that can be handed over to twenty Joe Average Developers and sustain productivity?

Or maybe you like your Ivory Tower.

Why Packaged ORM Frameworks are Bad for your Business

There are as many ORM or middleware or data access layer frameworks as grains of sand on the beach in Miami. Most or all do a lot of work for you and can speed up development tremendously. They can reduce code, increase quality, and centralize solutions that solve known problems in software development.

Most software architects love frameworks. For every architect, there’s at least one home grown framework. For every architect there may also be one or more favorite packaged framework. Some make a living pushing their framework by using it as a ramp up to consulting projects. Some write books about all the great patterns they’ve implemented in their framework.

Managers and directors and chief architects will all do powerpoint presentations showing how the chosen framework will reduce costs, save time, and improve results.

What they won’t tell you is that frameworks can be a terrible thing for your business.

Now that I’ve thrown down the gauntlet, let me explain.

I’m going to talk about Technical Business Debt today. This is from my own observations of using various frameworks in the last decade and I believe it’s something very measurable and very important to understand.

You have an existing codebase in any established business. At some point the code, the database, the reports, the systems…they all seem old. Somewhere along the line someone decides changes need to be made to improve one or more aspects of the codebase. You need a new data model. You need a web based data entry system. You need web services to share logic with other departments or external businesses. Whatever the catalyst is, something needs to change.

The first thing most architects look for is an easy win. How can I take the existing codebase and replace it or fix it with a cool new framework. Let’s implement an ORM system or Dependency Injection or Code Generation. I can have that new system developed in no time. Trust me.

Sound too good to be true? It is, but you won’t realize it until it’s too late. The problem with these pre-baked easy to implement frameworks is that you lose an essential piece of the upgrade process. You lose your business. How’s that?

It’s simple. If you slap a coat of paint on an old house, fix the trim, get a landscape crew to spruce up the yard, have some aromatherapy going on inside the house when people visit…hey, it looks like a nice house. People will be impressed. But they won’t see the cracked foundation that draws a 4 inch flood every spring. They won’t see the 15 year old furnace that costs 60% more than a new one would. They don’t see dust in the carpet that’s making all occupants suffer terribly from allergies.

If you really wanted to fix the house, you need to live in it. Then you need to take the time to fix things properly. You need to note the cost of heat and replace the furnace. You need to replace the old carpet with new carpet or hardwood floors. You need to put new windows in.

The same holds true of your business computer systems. In order to truly upgrade, you need to do it step by step. You need to implement each element of your business systems again, from scratch. The reason you need to do this is because your business has changed. Your customers have changed. The world has changed. As you rebuild your systems from scratch, the developers will ask questions. These will get passed on to a business analyst or stakeholder. The stakeholder will listen to the question and the proposed or assumed solution and stop everyone in their tracks. They’ll do this because the question is bringing up old business practices as well as revealing new practices. This process actually shows the stakeholders how their business has changed, much of which would have never been examined if it weren’t for the process of rebuilding each element.

It’s a bit like the Socratic Method. You learn by questioning and testing the foundation of each question. If you look at your systems and question their purpose as you rebuild them, you will discover how your business used to work, but more importantly, you will determine how your business should work. If you’re smart and lucky, you’ll also see trends.

Using frameworks that simply put a new face on old systems is fine for short term solutions. But don’t kid yourself that you’ve moved your business forward. You haven’t. You’ve just increased your Technical Business Debt. And you’ve also lost a priceless opportunity to examine your business at a low level. Something all businesses should do on a regular basis. You may not think you need to rebuild from scratch, but trust me, if your business has changed, the only way you’ll know is by doing just that.

The Perils of Consulting

In November of 2008, I was winding up a project at Insurance Auto Auctions. The project had gone extremely well and I was walking out with a letter from a director on how satisfied they were with my work. Over the course of ten months, I made increment changes to their development practices. They already had a very mature and strong development environment that included all the necessary hardware and software. No architect or developer could ever say that they weren’t given the tools at Insurance Auto Auctions.

After convincing management that we needed a dedicated build server, the first thing I implemented was a set of middle-ware templates that acted as a framework for accessing their databases. Not as robust as nHibernate or any of the common frameworks, but tooled for their environment without all of the extra features that either confuse developers or entangle projects with needless work. For instance, the templates I use do not have relations built-in. You have to wire them up on your own. For most traditional CRUD development projects, you rarely need more than Fowler’s Table Module enterprise design pattern anyway.

The second step was to implement continuous integration using Cruise Control .NET. A fairly common tool in the development world, created by Martin Fowler’s Thoughtworks consulting firm. This gave us immediate feedback on checked in code.

Thirdly, we went round and round about how to build the new system. Did we want to use traditional WebForms? We’re we okay with using the beta of ASP.NET MVC? Did we want to role our own MVP (Model View Presenter) framework or possibly do something else? In the end, we decided the simplest implementation, for various reasons, was to create a home grown MVP framework. Of course MVP is considered obsolete now and we saw what people were recommending, but we moved forward anyway. We felt the ability to cleanly write unit tests was a critical aspect. The downside of course was that we had to rewrite a number of HTML controls that come out of the box in ASP.NET. The out of box controls use ViewState and with ViewState off, we had to go it alone, so to speak. Luckily we had Cliff on the team. Any team that has some has brilliant as Cliff should feel blessed. I’m a pretty smart guy, but more suited to larger problem solving. Cliff has a knack for needling through really fiddly problems with a very elegant solution.

We of course got buy-in to implement unit tests in Visual Studio .NET 2008 Team Test. We installed this on the build server and after every check-in, all tests were executed and the build failed if any tests failed.

The business side worked on user stories, the developers had meetings to gain an understanding of each story, and the rest they say, is history. the development went very smoothly. By implementing all of the code in a single solution and by also implementing a nifty web.config connection string locator, we were able to execute deployments with the built-in Publish feature in Visual Studio. After dealing with complex and error prone deployment processes in the past, the project manager was deleriously happy with the new process.

We deployed the web application to several servers in a farm and after some fairly minor performance tuning, the system was deemed stable and in production.

In the last few weeks of the project, I was also working on securing my next position. This proved to be a very important decision and one that seemed too good to be true. Alas, that old saying is only the tip of the iceberg and will be discussed in Part 2.