How to Improve the Odds of Success in Software Development

Software development projects are notorious for having a high failure rate. In the context of this paper, “failure” is defined as, “not meeting the project sponsor’s expectation and/or stated requirements”. This would include such things as failure to function in the intended way as defined in a requirements document, not obtaining the required performance standards, going so far over budget that the project is canceled, or incurring so many bugs that the end-users view the system as unusable.

I began programming business applications twenty-nine years ago. In that time I’ve worked as a systems support engineer, developer, solution architect, director of development, consultant, trainer, and CEO of a software company. What I’ve learned from these years of experience is that projects fail repeatedly for a very short list of reasons. This paper will identify those key points of failure and offer simple guidance on how to avoid them – I say simple because to adequately cover all of the ways to solve software development problems takes volumes of books.

1 – Requirements

Many, if not most, companies have a natural history in the migration of their data storage, workflow, and reporting processes. The typical path of transformation is to go from paper, to spreadsheet, to database, to sophisticated business application. During this transformation, which often occurs over many years, the terminology and workflow process that were used when the business operated on paper often gets carried over to the spreadsheet. Business jargon and processes are established around how the business needs to operate under a paper-based system and continues after the company migrates to a spreadsheet-based system. This repeats itself again when adopting the database-based system, and so on.

The problem with this is that once a company has finally matured to using a fully capable business application for streamlining workflow processes, expanding the businesses capabilities for analyzing and reporting on business data, that system’s full capability is rarely realized. This is not due to the inability of the technology or the programmers creating it, it is typically caused by the business not being properly analyzed when preparing the requirements.

All too often, the internal sponsors of the project, end-users, business analysts, and other domain experts, are often in too much of a time constraint to meet milestones imposed by a Project Manager or Business Manager. Thusly; the project misses a truly golden opportunity to realize a much higher ROI on the system, greater productivity increases, longer life of the system, and better suitability for the way the business currently operates.

Here’s how you might resolve the problem:

  1. Advise/enlighten the PM: Let the PM and the project’s stakeholders know of the consequences of not evaluating the workflow process and domain terminology sufficiently.
  2. Document the cost of needing to rewrite a system: A rewrite in only a couple of years, or worse, never getting the system launched at all, compared to the extra time to conduct a proper analysis needs to be reviewed during the initial planning of the project. Engage the business analyst and/or architect to help with this as early in the process as possible.
  3. Question conventional terminology. Create a dictionary of the domain’s “Ubiquitous Language”. Challenge each term and its meaning to each stakeholder, sponsor, or end-user. In other words, requirements gathering is more than just collecting nouns and verbs.
  4. Work with a Domain Expert: A domain expert – versus everyday end-users – can analyze business processes that need to improve and how the system can accommodate that. Don’t just assume the data set tells the whole story about how it is used. The business analyst, or domain expert, must have a solid understanding of your business, not the technology to be used to serve it. Again, this should be done in collaboration with the architect.
  5. Create simple to understand user stories: Good user stories are short, precise, and limited to single actions. They should clearly state who, what, and why for each action the end-user or the system needs to perform. Don’t create elaborate requirements documents that obscure the intent of the requirement – it’s the old adage of, “can’t see the forest through the trees”.

2 – Translation of Requirements to Technical Specifications

The biggest “hat trick” in developing software is taking business concepts, which are often rather abstract in nature, and then converting them into very literal, concrete technical specifications. Many times the context of the business processes are either not understood by the programmers or, not accurately translated into the technical specifications and ultimately into the code of the system.

The problem with this is that you have business people producing the requirements and technical people making that translation. Unless the technical person has a true understanding of your business and, its business concepts, then the translation will most often be wrong. Unlike translating two languages with Google translate, where a person can guess at the meaning of words not translated correctly given a specific context of the conversation, a computer application cannot. Concepts, processes, actions all have to be very specific in order for the computer to process it.

Many development companies assign the task of making this translation to programmers. This is inherently flawed as programmers are dealing with the finest details of coding rather than the higher level, abstractions found in business. Bridging this gap in concepts and level of detail is nearly impossible to do well and, often time produces catastrophic failure in the project.

This is witnessed by observing the code and comparing it to the user stories. Often time the code combines multiple unrelated user stories into the same file, making it all but impossible to understand, modify, extend, verify, or maintain.

Another observation is that the code will be missing complete concepts derived from the domain experts and will be accommodated by a lengthy bit of code that works around the concept rather than articulates it. Examples of this would be where there are well used, common terms of the business, which relates to either specific data or specific processes that are real-world things in that particular business domain. When reviewing the code, it is common to see none of these terms used, but instead, replaced with technical jargon, arbitrary abbreviations, or worse, single letters. This makes it difficult to impossible to know if the code is truly matching the requirements. Even if the end-user functionality seems to be there and working, it doesn’t mean the code was constructed properly. What this can lead to – and almost always does – is that there is a high probability that while the first iteration of the system might seem to work fine, when the business wants to extend a feature’s capability or, add new features, the foundation of the code just won’t support it. I cannot count the number of times either I or other technologists have had to advise the client, “A rewrite is required”.

Most companies attempt to solve this issue by including a business analyst and/or solution architect on the team.

The responsibility of the business analyst is to be a domain expert and know how to correctly document the requirements of the system in a way that technical people can understand.

The role of the architect is to take the requirements and model a system in a way that illustrates a clear understanding of the requirements to the project’s stakeholders and a clear technical framework to work within for the programmers – thus, the “hat trick”.

The problem with this is that most companies think of a Solution Architect as a person who has been a senior programmer or technical team leader who, due to longevity (experience) needs to be promoted to architect as the next logical step. However, this just means that we are back to the start of the problem with a programmer, albeit highly experienced, making the translations. Unless the business analyst did a superb job in articulating the requirements in way that is easy for programmers to understand, there’s still likely to be the gap in concepts and understanding.

Rarely do senior technologists have a deep understanding of business concepts, and equally, business analysts a deep understanding of programming. What’s even more problematic is that programmers are almost never experienced in a domain enough to offer insight or suggestions into how to improve the current business processes or way the business data is organized. They rely solely on the details of the requirements and do not typically “think outside of the box”. This is extremely common with offhshore development companies.

The one difficulty regarding this is that it still takes a unique individual to truly excel at being equally capable in understanding business concepts versus technical concepts. Many people operate with only one type of thinking pattern, either they are very detail-oriented, methodical thinkers, the type that make good software programmers, accountants, and NASA engineers, or, they are creative, abstract thinkers who make good architects, artists, financial analysts, and marketing professionals. The latter can more easily see things from different perspectives where the detailed-oriented needs to make sure all the dots are connected. It is simply the altitude at which you view a problem.

Understanding broad business processes such as customer relationship management, or supply chain management requires a far more abstract, imaginative type of thinker than does determining how to construct complex securities trading algorithms in code.

Unfortunately this is a difficult problem to solve as there are not a lot of solution architects who have both a MSC and MBA in their education or, the hand-on business experience to augment their technical experience. However, this is really what is needed.

Here’s how you might resolve the problem:

  1. Educate architects in business concepts: Companies with internal development departments need to adjust their definition of solution architect and provide the educational and training opportunities for gaining the missing knowledge. Architects should take business courses in order to have a better understanding of how businesses operate. If the architect is working within just one specific domain, education about that domain from a business point of view is critical.
  2. Allow sufficient allocation of time and budget: Teams pressured to hit the PM’s milestones will always pull out their toolbox of “shortcut tricks”. This often leads to a serious degradation of the quality of the underlying code. Since PMs don’t usually see the code, or understand it, then, “out of sight, out of mind”.
  3. Engage the architect earlier into the process: Allow them to work side-by-side with the project sponsors and/or the business analyst from project kickoff to final release. This cross-collaboration will help each develop the skills to bridge the gap between the abstract and the specific.
  4. Make Domain Driven Design (DDD) an integral part of your development process. The DDD philosophy (not really a methodology or process) provides a set of design patterns and, an approach to acquiring requirements that can help bridge the gap for the highly technical person who needs to understand the business requirements. DDD explains the best methods for communicating with the stakeholders of a project. The style in which the technologist communicates helps them discover the deeper meaning of domain terms and concepts. This greatly aids in modeling a system that accurately reflects the real-world domain.

3 – Process

There are many processes used in software development, but for this paper, let’s simplify it to the following; Phased-Based, Agile, and RAD. Phased-based are those processes, such as the “Waterfall” methodology that puts an emphasis on a thorough completion of each phase before the next phase begins. For example, all requirements are gathered and documented before it goes to the design phase.

Agile-based are those processes, such as “Scrum”, which perform all phases on small chunks of the requirements in short iterations, typically 2-4 weeks long, and then repeating through enough iterations to produce a final product. For example, requirements, design, programming, testing for a subset of the requirements would all be performed in one 2-week iteration.

RAD (Rapid Application Development) is a methodology that focuses on end-user needs, typically articulated through user-interface mockups and data schemas. There is little need for understanding the business domain or the workflows. This methodology is intended to produce working software as fast as possible.

Many technologists will profess one method is better than another; however, it really depends on the nature of the project.

Phase-based: If you are designing a guided missile system for the U.S. Department of Defense, there is typically a much greater risk due to project failure, a bigger budget, and more time flexibility given to the project. The more important aspects of the project might be quality, capability, and assurance of completion. For this type of project, the phased-based is often more suitable as it provides more details upfront so that PMs know what milestones to establish, resources required, and budgets.

The drawback to this method is that if, you don’t get one phase correct, there is a cascading effect and magnification of the problem, thrown over the fence to the next phase. It also is not very accommodating to changes mid-stream.

Because it is very difficult to know everything upfront, before coding begins, this methodology has a very high failure rate.

Agile: The various Agile methodologies have great advantage over the Phased-based when applied to the majority of business applications since most business applications have a moderate amount of complexity compared to guided missile systems. The greatest advantage is that the Agile, iterative style of process is highly accommodating to change. Typical businesses don’t have the time to go through the extensive analysis phase that the Department of Defense has; therefore, most projects start with an incomplete, or inaccurate, set of requirements. The business sponsors and/or programmers often discover missing concepts, need for additional features, changes to features, or changes to workflow processes as coding progresses.

Agile processes embrace this reality and, due to the short-cycle nature of the process, it allows for incorporating these changes into the project without having significant impact on code already generated.

RAD: RAD has been around a long time and there have been many tool development companies attempt to create a single tool that does everything very quickly. Create the user-interface, the database, and the business logic with the least amount of hand coding.

This method of software development is fraught with problems. First, it forces business requirements to be implemented within the tight confines of how the tool generates code.

Few RAD tools, if any, accommodate good coding practices such as best practices for object-oriented programming or ease of testing code. Most tools inadvertently promote the lumping of large amounts of code together, which is doing many different things – this breaks the object-oriented nature of the code. While RAD development style is very quick and can produce a working feature almost immediately, it is far more difficult to easily add new capabilities, find bugs, or be maintained later on after the code has been deployed than properly written code.

Applications that are best suited for this style of development are those applications that are mostly just database systems where data is entered and later queried for review. These systems might have little to no real business logic other than some data validation. In addition, the simpler the data structure is the more suitable this method is. In addition, applications best suited for RAD are also applications with far less need to change significantly with new, or extended, features over its lifespan.

Software development teams, or companies, that do not have a well-defined development process, that is applicable to the nature of its projects, can easily get derailed in meeting estimated budgets and timelines.

In addition, taking a well-defined process, that has been proven effective in numerous other organizations, and making changes can compromise the effectiveness of the process.

An example of this is Scrum. Many teams or companies alter this process to fit their traditional practices. In Scrum, there is no role of Project Manager. While most all companies, prior to adopting Scrum, have always had this particular role, they seem to be unwilling to abandon it. The result of this, is it works against the Scrum process. This ultimately causes many companies, who have tried it, to abandon it saying, “it doesn’t work”. What doesn’t work is using a traditional PM in a Scrum process.

Here’s how you might resolve the problem:

  1. Understand each process type: If you chose to do an Agile process like Scrum, don’t do what is called “ScrumBut”, where you take parts out and put other parts in. These processes have been developed over long periods of time and in environments where they received a thorough adoption. Making arbitrary changes to shoe horn the process into your current conventions can completely eliminate all of the advantages of the process.
  2. Adopt the appropriate process for the particular type of project: The solution to adopting a proper process is to first understand the nature of the project. It is not a good practice to adopt one process and apply it to all projects, unless all of your projects are of the same type.
  3. Thoroughly educate your client on the process: This is particularly true with RAD and Agile as they are the least understood by business managers, project sponsors, PMs, and end-users. Make sure that the client fully embraces the process in both theory and practice. The client must be fully aware of their role within the process and, they must be prepared to exercise that role in a timely manner. For example, in Scrum, the project’s domain expert, business analyst, or project’s sponsor might be filling the role of Product Owner. It is their responsibility to prioritize the requirements that need to be coded for each Scrum Sprint. It is also their responsibility to be available for clarification of requirements. Projects can be seriously delayed or otherwise compromised should the client not fully embrace, or exercise, their role and responsibilities. Don’t be timid in setting the ground-rules for the client – internal or external.

4 – Programmer Skills

A vast amount of coding is being produced “offshore”. The term “offshore” is a bit misleading these days. Let’s just say it refers to countries that have relatively cheap labor rates for programming when compared to major European countries, Japan, Australia, or the U.S.A.

I have worked with programmers in China, India, Egypt, Thailand, Vietnam, Philippines, Lithuania, Denmark, Holland, Germany, England, New Zealand, Australia, and the U.S.A.. What is consistent with all of the countries that fit the definition of “offshore” is that the skill set of the programmers is far lower than that of their peers in the U.S.A., England, Germany, Denmark, Holland, etc. – the non-offshore countries.

I have concluded after working with hundreds of these programmers is that the problem originates in the universities of these “offshore” countries. There is very little taught in the way of advanced programming techniques, object-oriented programming, software design, design patterns, unit testing, etc. These universities primarily focus their computer science classes on the use of development tools and programming language syntax. Little is being taught on the ways to creatively and properly use the tools and programming languages.

Most of the graduates come out of the university, even with masters degrees, knowing only the RAD style of development, and a limited set of tools.

Here’s how you might resolve the problem:

  1. Establish an ongoing training program: This may take away 2-4 hours every week per programmer from billable/production coding time, but will pay enormous dividends in the long run. Focus on architecture and good coding craftsmanship.
  2. Continually improve the level of English: English is the standard language around the world and what virtually all technology books, articles, and tutorials are based on. I the programmer is weak in English, they only get a limited amount of knowledge transferred during the learning process. If the programming team does not have a solid foundation in English, including a lack of strong accent, it is problematic at best, to assure that requirements are clearly understood. High standards must be set in this area. This is especially true for European sponsors of project since in most European countries, English is also a second language and strong accents prevail. Many projects have either been difficult to get done well, or simply fail, because a German accent has great difficulty in beinbg understood by the Indian programmer, who also has a strong accent. This issue becomes critical when the development company has adopted a development process such as Scrum where the developerment team has much greater direct communication with the client, business analyst, domain expert, and end-users.
  3. Elevate your coding standards: There are many different versions of what good coding standards might be, but almost everyone would agree that few practice any of the versions well. My hot-button, and a red-flag indicating there’s a programmer skills issue, is the naming convention used in the code. If I see programmers using abbreviations in their code, then I know they have learned bad programming habits in school or on the job. I also know that they have not spent time investigating what good coding standards are. In other words, they would not be considered good craftsman of their trade. While there are many other ways to make this judgment, I have found that this one item is foolproof in telling me that many other best practices are ignored. In addition, if the naming of classes, methods, and variables related to the business domain, do not reflect its specific intent within its domain, then I know the programmer is very likely to lack a solid understanding of the requirements from the point of view of the project sponsor or domain expert.
  4. Make Test Driven Development a Religion: TDD is a development process mostly embraced by the Agile community. It is also a catch-22 situation when it comes to implementing it. With most organizations not fully on the Agile bandwagon and, still following traditional command-n-control project management conventions, testing is one of those things most often dropped from a programmer’s daily tasks when the project is pressured to meet deadlines by the PM. Development teams that can fully embrace the Agile, particularly Scrum, methodology, know that teams can achieve high standards for both quality and output when motivated with rewards on such items as test-coverage. High test-coverage usually translates to less refactoring, less bug fixing, easier extension, and easier maintenance. All of which contribute to the ROI of the project, the thing the PM should be most concerned with. It also has an indirect benefit of improving the relationship with the client should additional phases of development occur. This results from the tests making it much easier to add the new features requested. When the amount of time and expense is lower due to good coding and thorough test-coverage, that usually impresses them and motivates them to stay with you as a client.
  5. Follow the Domain Driven Design approach to fully understand the domain: DDD skills provide the programmer and architect methods to best understand the business through the eyes of the business person. This greatly increases the likelihood that key domain concepts will be accurately portrayed in the code. This provides a stable foundation of code to build upon as new requirements or changes to existing features come in. DDD improves the communication techniques between the business and the engineering sides of the project.

Conclusion

This was a short list of project failure points and some ideas on how to solve, or avoid, them. All comments to this paper are welcome. There are dozens, even hundreds, of other points of contention in a software project; however, these have been the most consistent ones experienced over the twenty-nine years of working at developmenting software.

Last bit of advice is that unless your organization is committed to solving the above mentioned problems with bold action and firm commitment, the probability of success is greatly diminished. Changing development methodology from Waterfall to Agile can be a cultural nightmare. For those who make the transformation when needed for the nature of their projects, the end result is astonishing.

To use a metaphor from rock climbing, if a rock climber’s techniques of climbing is inadequate, and he/she falls, if the process of belaying is also inadequate and only slows the rate of descent, the climber will still hit the ground with catastrophic results. Focus on the programmer’s skills and your organization’s processes and, that should reduce the risk of falling in the first place or, not arresting the fall altogether.

See Your Company Flourish With Software Development Services

Your business fledges and your business needs of streamlining your database of customers and meeting the demands or expectations of ebb and flow of business rise. To raise the bar of services and products, software business systems are recommended. These software development systems are clearly defined niche and reposed on and built for a particular service. The software system includes design, development and implementation.

The software development is specifically tailored made according to the needs of the clients and deals with the inherent and external challenge an organization faces. From a large sized business venture to small scale industries are adopting this technique and creating their own software to give themselves an ultimate business advantage and significant returns on investment. Companies traversing various spheres such as Finance, Architects, Solicitors & Notaries, Construction Groups and Media have developed their own scalable Microsoft based technologies & software to enhance and boost their business expansion.

Generally a software development services comprehends software development, system integration, maintenance and support. To tighten up the whole process, these under mentioned points should be kept in mind before partnering with the providers of software or web Development Services:

1) A proven track record of prowess in creating need based software.

2) Should be professionally involved in the business.

3) Highly skilled and expert team of software engineers

4) Should deploy personnel infrastructure of skilled project manager and quality assurance manager for your project

5) Should offer a cost effective deadlines and be strict towards deadlines

6) Can deliver quality product, on-time and on-budget

7) Use of the latest web development as per industry standards

8) Can provide upgraded version of software

Advantages of utilizing a web application development service are:

A) Enhancements and expansions of IT systems

B) Streamlining of administration and documentation

C) A unique web appearance

D) Programming Services for any new developments

To make your business a growing piece of opportunities is not a bed of roses and to continually concentrate on increasing competition of market, sales and other administration process are equally cumbersome and time consuming. With a right company to provide customized software will not only help you to boost your web presence but make an everlasting impression of a reliable company among customers

Return on Investment in Software Development

Hi I thought I would write a series of articles on a topic that I believe is becoming absolutely critical in the software development industry, that being the magical and mysterious ROI (return on investment) in software development.

As the owner of a programming company here in South Africa, I decided at the outset of the company that a rule of thumb for accepting any software development project would be that I would need to figure out how to measure the ROI in that particular project. Now you may or may not be surprised to know that in most instances my clients don’t actually think of this measure before taking on the project, they know that they have a business problem or business need that needs to be solved with technology and are willing to pay an amount of money for that. Experience over the years however has shown me that the best way to retain that client in the long term and or to make more business from that client is to prove a ROI in some way on that software development project.

I find this measure to be critical because for my companies purposes it shows exactly how successful a project is to my client in monetary terms. The bigger my impact through the software I develop, the more successful I have been at satisfying that clients need, the easier it is for me to go back to that client and offer further services. Contrary to popular belief, I don’t believe that a project coming in on time or under cost is a great measure of the success of that project at all, this is typically what IT companies do. If I really want to do a great job for my client then I always intentionally build a monitoring tool into the software I develop for them and that tool must track the ROI for the client in some way and present a report to the client whenever they want to see what this piece of software really is worth to them, be it through savings, additional income, time savings, it doesn’t matter so long as you track the value.
So how do you go about figuring out the ROI on software development project?

This is the difficult portion because it requires listening to the clients need carefully and scoping there need correctly, once you have done this, it should become fairly obvious what the client is trying to solve by having the software developed, this should then be a clear indication of the measure to be used on ROI. For example,a client of mine wished to have a system developed that automated the process of allowing shareholders in various properties to swap there owned weeks with each other. Before the client came to me they did this process manually, ie called there shareholders to find out if they were willing to exchange a week with a fellow shareholder that already gave an indication that they wanted to swap there week etc. This took a lot of time and for the administration fee that the client asked to do this, I suspect the profit was very little. Once we had developed the system and allowed the shareholders to interact with each other through the website instead of the web system, the ROI was very clear, why?

-We tracked each exchange in the system, each exchange has a monetary value associated to it.

-The clients telephone bill dropped significantly

-More swaps occur because of the ease at which shareholders can swap weeks, instead of having to go through a process of calling the company then waiting etc, this increase in sales is measurable

-Because the client offered the service electronically he could increase his administration fee.

All this information the client could get through there system thereby ensuring that he could very clearly see what his ROI was.
Conclusion

I believe without a doubt that if you truly want to complete a software development project successfully for a client, then you need to build a ROI tool into the software. Convince the client to do this, if you cant do that then build it free of charge, but make sure that you get your client to look at these figures. In the long term this will have a positive effect on your business and keeps your software development projects honest in terms of creativity and keeping the clients needs in mind constantly.

Please offer your comments and views on this topic, I am very sure that peoples opinions and views on this vary quite considerably, I am very willing to listen to other opinions in this regard.

In my next article on ROI and software, I will be describing more specifically how you can measure ROI on certain types of projects, in this case it will be how to measure ROI precisely in SMS campaigns.