Software Development Can Be Better
Programming allows us to do just about anything with computers. We can receive input, transform it, stage it and have it produce something useful on the output end. One thing that I have observed in my 15 years of writing custom software is that we re-invent the product with every project.
In a previous post I stated that programmers are like auto mechanics but mechanics have an advantage over programmers because, for the most part, all cars are the same. I am not a mechanical engineer and will admit that I don't have the background necessary to design an automobile. However, I think that if I researched enough and got the right people, welders, machinists, I could possibly come up with a way to assemble a simple automobile (nothing fancy, more like a go-cart). I could simply grab the different necessary components and assemble them into an automobile. The trick would be in making the components compatible with one another.
I think line-of-business (LOB) software has the capability to be assembled in very much the same way. There are a few things that hinder us from developing software in this manner though. I will list some of those obstacles in this post. These are just simple observations that I have seen over the past fifteen years.
Most LOB systems have many things in common. It doesn't matter what type of system we are building, there are some common components no matter what software product we are creating.
Some of these things are
These are only a few of the features that I can think of off the top of my head. There are many other types of processes within their respective industries that are common. I am willing to bet there is a lot of code duplication there as well.
The thing that comes to mind is; if we know that the typical LOB software has many things in common why must we re-invent the wheel with each new system? If we compare this to the automobile, it would be akin to developing a custom vehicle costing as much as a Ferrari every time, with the quality and feature set no better than a ordinary VW Beetle. In our case even worse, because the maintenance costs will outweigh the purchase price.
This is a real problem. I think it's time that we take a long hard look at the way we create LOB software and take a smarter approach.
I think it's time that we begin creating frameworks and establishing design patterns at the business level. Sure we have many frameworks at the technical level We have frameworks such as .Net, Spring (in Java), the entire Java set of libraries, PHP libraries, JavaScript libraries, etc. These frameworks are good and have proved their worth but they provide little in terms of providing out of the box business solutions.
In the design pattern arena we have the Gang of Four (GoF) design patterns to guide us. Taking that even further we have the patterns described in Domain Driven Design. Those are excellent tools but we need to go even further. We need an establishment of business design patterns. I am sure that many of the same business problems have been solved thousands of times.
In terms of design patterns there is little that can hold up our progress since they are re-usable from one project to the next or one technology to the next. Re-usable components are a bigger challenge. Here is a list of obstacles that hinder us from investing too much into re-usable business frameworks and components.
Creation of new stacks
Every so often new stacks are introduced. Here are a few of the stacks that have been created over the years. Some of them may be considered stacks or simply technologies. Nevertheless they provide a way to create custom software.
I'm sure there are many others but I just listed what I could come up with off the top of my head. These stacks always show up as the best thing since sliced bread. Some prove their worth more than others. Usually the latest stack is better than the last, solving problems that were not addressed in the previous. This is a good thing. However, this also proves to be a problem because all the re-usable code that we might have created in the previous stack gets thrown away with the last set of projects in that stack.
This is quite a challenge to overcome because the proliferation of new stacks seems to be never ending.
Proprietary Systems
There is no sharing of code from one organization to the next. I guarantee you that just about every organization has their own version of a role-based access system, for example. In many cases there are many different version of the role-based access systems because each project created their own.
Every organization determines that they will not share any of their own code with any other organization. As such all the investment in said codebase is used only by them. There is a lot of waste involved when each organization sets out to create their own version of common components. This would be akin to every automobile manufactures set out to invent a brake system or engine system each time they built a new car. The craziest part about all of this is that there are thousands of times many more software developer organizations than automobiles manufactures. Let that sink in for a minute.
This is a real problem also because if you have created a killer business framework, it might no longer be usable in the next version of the .Net/Java version. This means constant upgrades. It is not only a challenge for you as the developer of the framework but also a challenge for the consumers of your framework and components. Migration may require a significant amount of effort and by significant I mean tens or even hundreds of thousands of dollars.
Developer Immaturity
Most organizations management teams understand management but have little understanding of software development. Thus management usually does not make the best choice when selecting the technical members for a brand new project. A brand new project requires a solid team that is led by a solid technical lead/architect that can make the best architectural decisions.
Instead many organizations hire developers with insufficient knowledge and experience. Your average mid-level developer understands very little about software development at the architectural level. The schools teach very little in terms of real world systems. As such these developers don't utilize the best practices. I know many of you reading this will say, my organization does. That may be true, but it will also be true that your organization is one of the more advanced ones.
This lack of developer maturity will mean that the product will not be as maintainable and will not be as secure. Above all, the software will prove to be quite a challenge to understand by any new developers. This lack of maturity means that not only has the system been re-invented it is likely a poor implementation of the invention.
The best approach that a manager can take in this realm is to hire the strongest technical lead they can get their hands on and allow them to choose the team from there. But then there is another problem That problem being that the manager is not adequately prepared to choose a strong tech lead. I would suggest lots of research and some consulting to get help with that determination.
Proprietary Control
For many stacks and frameworks we are at the mercy of big corporations. Microsoft owns .Net and Oracle owns Java. The minute they decide to no longer support a particular version you will have the choice of upgrading, paying lots of money for support for an outdated system, or stay on an outdated version and leave yourself open for vulnerabilities.
In other cases you may have no choice but to abandon the framework altogether (Think Silverlight or XNA). Microsoft introduced Silverlight with a lot of fanfare but in the end it proved to be just another Flash. This is after they sold those as the future. XNA on the other hand was ditched altogether.
Programmer Resistance
Programming requires a lot of creativity. It provides a great way to create something from nothing. Programmers enjoy creating things. This is because they are inventing. Re-inventing and creating is fun but it is also very expensive. Most developers don't want to simply configure an existing framework. Certain frameworks that provide out-of-the-box solutions, require mostly configuration. Some systems like that already exist (think PEGA). These systems are very robust and can allow developers to create software many times faster than even the 10x developer. The biggest problems with these frameworks is that they are expensive and proprietary. As such they are part of a niche market. That means that it's hard to find developers with those skills. Or even worse, it is hard to find developers that won't change jobs even once you train them for that product.
These systems allow you work at a higher level of abstraction and can accelerate development. The problem is that programmers don't like to work with higher level systems in many cases. Programmers want to work with the bricks of mortar of the system.
Another reason for resistance is that many programmers want to work with the new shiny toys. Take a look at any of the hundreds of SPA frameworks that have been created over the last few years and I guarantee you that many developers dove their organizations head first into these technologies. Many reasons exist for playing with the latest and greatest, Resume Driven Development (RDD) being one of those reasons. There is nothing wrong with using the latest greatest if it suits the projects needs the best. However, what I've found is that many times developers are simply doing it for the sake of playing with the latest and greatest. All these things come at a very high price. The price is paid by the customer and the organization.
As with anything, you must choose the right tool for the right job. Be ware of developers that are only willing to play with the latest and greets for the sake of the adventure.
Summary
Perhaps I am an utopian, but I think there is much room for improvement on reducing waste in the software development industry. I believe that we can create systems that allow for more rapid development which will help us provide the customer more value for less cost. I am not sure that we can individually make a change for the reasons named above. Sure we can begin doing a better job of creating re-usable software that is shared across the organization but there is currently little in terms of creating software that is shared across organizations. You may argue that open source is a possible way. What I have noticed with open source is that usually it targets core technical problems but not the business level technical problems.
I have some ideas and ways to mitigate these problems within my team, but not for the industry as a whole. I know I just brought a problem and did not include the solution. It's like that sometimes. Before you can identify a solution you must acknowledge that you have a problem.
What are the things that you see that can help us be more efficient as an industry? What steps do you think we can take to overcome these obstacles? What can we do as an industry to make the software industry reduce this waste?
As always, thanks for reading and please remember to follow me on twitter @fernandozamoraj.
In a previous post I stated that programmers are like auto mechanics but mechanics have an advantage over programmers because, for the most part, all cars are the same. I am not a mechanical engineer and will admit that I don't have the background necessary to design an automobile. However, I think that if I researched enough and got the right people, welders, machinists, I could possibly come up with a way to assemble a simple automobile (nothing fancy, more like a go-cart). I could simply grab the different necessary components and assemble them into an automobile. The trick would be in making the components compatible with one another.
I think line-of-business (LOB) software has the capability to be assembled in very much the same way. There are a few things that hinder us from developing software in this manner though. I will list some of those obstacles in this post. These are just simple observations that I have seen over the past fifteen years.
Most LOB systems have many things in common. It doesn't matter what type of system we are building, there are some common components no matter what software product we are creating.
Some of these things are
- Role Based Access Configuration and Control - This is providing access to the users within the system based on their roles. The access must be controlled and also a graphical interface must be provided for users to configure access for other users.
- Input Validation - This is all your input validation, white lists, black lists, minimum values, maximum values, password validation, etc.
- Data Storage - Data needs to be stored persisted
- External Interfaces - At some point data must be either imported or exported to or from external systems. Other times certain triggers must take place for this communication to take place. External interfaces can be in the form of messaging systems, web services, heaven-forbid-direr database access.
- Job Staging and Processing - Some jobs are too big to take on during normal processing. Other times the concept of store and forward is necessary. Things like reconciliations, clean up and archival jobs must be performed.
- Setting Configuration - Setting and configuration of the system is necessary for the system to behave in a preferred manner.
- Transferal of Assets - Assets can be anything from real physical assets in an asset management system to digital assets such as in the case of lending books on Kindle.
- Workflows - Some processes require the work to be done in steps by different users. In this case the workflow must be controlled in a certain manner. Many times this is done through staging processes and notifications to the next user on the flow process.
These are only a few of the features that I can think of off the top of my head. There are many other types of processes within their respective industries that are common. I am willing to bet there is a lot of code duplication there as well.
The thing that comes to mind is; if we know that the typical LOB software has many things in common why must we re-invent the wheel with each new system? If we compare this to the automobile, it would be akin to developing a custom vehicle costing as much as a Ferrari every time, with the quality and feature set no better than a ordinary VW Beetle. In our case even worse, because the maintenance costs will outweigh the purchase price.
This is a real problem. I think it's time that we take a long hard look at the way we create LOB software and take a smarter approach.
I think it's time that we begin creating frameworks and establishing design patterns at the business level. Sure we have many frameworks at the technical level We have frameworks such as .Net, Spring (in Java), the entire Java set of libraries, PHP libraries, JavaScript libraries, etc. These frameworks are good and have proved their worth but they provide little in terms of providing out of the box business solutions.
In the design pattern arena we have the Gang of Four (GoF) design patterns to guide us. Taking that even further we have the patterns described in Domain Driven Design. Those are excellent tools but we need to go even further. We need an establishment of business design patterns. I am sure that many of the same business problems have been solved thousands of times.
In terms of design patterns there is little that can hold up our progress since they are re-usable from one project to the next or one technology to the next. Re-usable components are a bigger challenge. Here is a list of obstacles that hinder us from investing too much into re-usable business frameworks and components.
Creation of new stacks
Every so often new stacks are introduced. Here are a few of the stacks that have been created over the years. Some of them may be considered stacks or simply technologies. Nevertheless they provide a way to create custom software.
- C++ with the Microsoft Foundation Library
- C++ and CGIs
- Visual Basic
- Classic ASP
- JSP
- J2EE
- Asp.net
- SilverLight
- Windows Presentation Foundation
- Java Android
- Apple Object C
- Ruby on Rails
- PHP (LAMP and others)
- Node.js (MEAN, Backbone, etc)
I'm sure there are many others but I just listed what I could come up with off the top of my head. These stacks always show up as the best thing since sliced bread. Some prove their worth more than others. Usually the latest stack is better than the last, solving problems that were not addressed in the previous. This is a good thing. However, this also proves to be a problem because all the re-usable code that we might have created in the previous stack gets thrown away with the last set of projects in that stack.
This is quite a challenge to overcome because the proliferation of new stacks seems to be never ending.
Proprietary Systems
There is no sharing of code from one organization to the next. I guarantee you that just about every organization has their own version of a role-based access system, for example. In many cases there are many different version of the role-based access systems because each project created their own.
Every organization determines that they will not share any of their own code with any other organization. As such all the investment in said codebase is used only by them. There is a lot of waste involved when each organization sets out to create their own version of common components. This would be akin to every automobile manufactures set out to invent a brake system or engine system each time they built a new car. The craziest part about all of this is that there are thousands of times many more software developer organizations than automobiles manufactures. Let that sink in for a minute.
This is a real problem also because if you have created a killer business framework, it might no longer be usable in the next version of the .Net/Java version. This means constant upgrades. It is not only a challenge for you as the developer of the framework but also a challenge for the consumers of your framework and components. Migration may require a significant amount of effort and by significant I mean tens or even hundreds of thousands of dollars.
Developer Immaturity
Most organizations management teams understand management but have little understanding of software development. Thus management usually does not make the best choice when selecting the technical members for a brand new project. A brand new project requires a solid team that is led by a solid technical lead/architect that can make the best architectural decisions.
Instead many organizations hire developers with insufficient knowledge and experience. Your average mid-level developer understands very little about software development at the architectural level. The schools teach very little in terms of real world systems. As such these developers don't utilize the best practices. I know many of you reading this will say, my organization does. That may be true, but it will also be true that your organization is one of the more advanced ones.
This lack of developer maturity will mean that the product will not be as maintainable and will not be as secure. Above all, the software will prove to be quite a challenge to understand by any new developers. This lack of maturity means that not only has the system been re-invented it is likely a poor implementation of the invention.
The best approach that a manager can take in this realm is to hire the strongest technical lead they can get their hands on and allow them to choose the team from there. But then there is another problem That problem being that the manager is not adequately prepared to choose a strong tech lead. I would suggest lots of research and some consulting to get help with that determination.
Proprietary Control
For many stacks and frameworks we are at the mercy of big corporations. Microsoft owns .Net and Oracle owns Java. The minute they decide to no longer support a particular version you will have the choice of upgrading, paying lots of money for support for an outdated system, or stay on an outdated version and leave yourself open for vulnerabilities.
In other cases you may have no choice but to abandon the framework altogether (Think Silverlight or XNA). Microsoft introduced Silverlight with a lot of fanfare but in the end it proved to be just another Flash. This is after they sold those as the future. XNA on the other hand was ditched altogether.
Programmer Resistance
Programming requires a lot of creativity. It provides a great way to create something from nothing. Programmers enjoy creating things. This is because they are inventing. Re-inventing and creating is fun but it is also very expensive. Most developers don't want to simply configure an existing framework. Certain frameworks that provide out-of-the-box solutions, require mostly configuration. Some systems like that already exist (think PEGA). These systems are very robust and can allow developers to create software many times faster than even the 10x developer. The biggest problems with these frameworks is that they are expensive and proprietary. As such they are part of a niche market. That means that it's hard to find developers with those skills. Or even worse, it is hard to find developers that won't change jobs even once you train them for that product.
These systems allow you work at a higher level of abstraction and can accelerate development. The problem is that programmers don't like to work with higher level systems in many cases. Programmers want to work with the bricks of mortar of the system.
Another reason for resistance is that many programmers want to work with the new shiny toys. Take a look at any of the hundreds of SPA frameworks that have been created over the last few years and I guarantee you that many developers dove their organizations head first into these technologies. Many reasons exist for playing with the latest and greatest, Resume Driven Development (RDD) being one of those reasons. There is nothing wrong with using the latest greatest if it suits the projects needs the best. However, what I've found is that many times developers are simply doing it for the sake of playing with the latest and greatest. All these things come at a very high price. The price is paid by the customer and the organization.
As with anything, you must choose the right tool for the right job. Be ware of developers that are only willing to play with the latest and greets for the sake of the adventure.
Summary
Perhaps I am an utopian, but I think there is much room for improvement on reducing waste in the software development industry. I believe that we can create systems that allow for more rapid development which will help us provide the customer more value for less cost. I am not sure that we can individually make a change for the reasons named above. Sure we can begin doing a better job of creating re-usable software that is shared across the organization but there is currently little in terms of creating software that is shared across organizations. You may argue that open source is a possible way. What I have noticed with open source is that usually it targets core technical problems but not the business level technical problems.
I have some ideas and ways to mitigate these problems within my team, but not for the industry as a whole. I know I just brought a problem and did not include the solution. It's like that sometimes. Before you can identify a solution you must acknowledge that you have a problem.
What are the things that you see that can help us be more efficient as an industry? What steps do you think we can take to overcome these obstacles? What can we do as an industry to make the software industry reduce this waste?
As always, thanks for reading and please remember to follow me on twitter @fernandozamoraj.
Comments