Writing Effective Designs

posted 15 Jun 2010, 13:49 by Sanjeev Kumar   [ updated 30 Nov 2011, 03:43 by Sanjeev Kumar ]

Overview

As the software development is maturing, the HLDs and LLDs are widely accepted as an integrated part of software development cycle, no matter what software development methodology we use. Even though the need of having a design is accepted and 30% to 40% of the total development effort is spent on design, it is rare to see effective designs that lay strong foundation for the application development. Very often they fail miserably in providing guidance and clarity for the development teams to start with and as a result the actual implementations are miles apart from the original design. In fact, in many cases the only useful purpose design serves is to complete the list of deliverables for the project for its closure and audits. 

Writing effective design is one way to prevent such situations. This document attempts to suggest a step by step approach to develop and deliver effective designs that shall not only be followed and respected by developer community but also play key role in development of stable and robust application.

What is design?

Before we get into the details about writing effective design, it is important that we have the same understanding about what the design is that we are referring to here. Here we are essentially talking about  HLDs and LLDs for software development which is an essential part of software development and should ideally be available before the start of development face of the project.

What is an HLD?

  • Is a document or/and set of UML artifacts that defines the higher level component view of the system
  • Usually includes a high-level architecture diagram depicting the components, interfaces and networks that need to be further specified or developed.
  • Mentions every work area briefly, clearly delegating the ownership of more detailed design activity to the concerned LLDs.
  • May define high level domain model and interaction diagram which shall be elaborated in LLDs
  • Defines boundaries of the system, possible subsystems and establish communication methodologies within various sub-systems
  • HLD contains details at macro level and so it cannot be given to programmers as a document for coding
  • Documents HLD decisions stating the reason for the design decision. For example using Struts2 framework over Spring MVC.
  • Is used by designers or senior developers to create LLDs for their subsystems strictly following the boundaries, interfaces and guidelines provided in HLD

What is an LLD?

  • Is used as program specification by the developers to develop the code for the subsystem that the LLD is written for
  • Is at the lowest level of abstraction before the code itself and clearly define the methods, their parameters return types and interaction with dependencies
  • Contains low level domain model and details each and every property of all domain objects
  • May contain separate low level ER diagram representing the tables participating in the LLD
  • In most cases uses HLD as the input criteria
  • Documents LLDs decisions stating the reason for the design decision. For example using inner class rather than separate class.

Why do we need design?

Why we need HLD?

  • It serves as a communication tool that speaks to and for all stakeholders of the project. HLD is responsible for establishing vocabulary for the project that helps in project related communication across designers and developers
  • It defines overall architecture and subsystems of the application and fulfills all functional and well non-functional requirements which is most important aspect from customer’s point of view
  • It enforces standards by publishing guidelines for services and components development
  • It bridges the gap between functional and technical views of the system and brings the two view points together to discuss and close issues that are important for the success of the project.
  • Creates lower level abstractions of the requirements for the low level designers to work on by breaking up the requirements into deliverable UCs and features. However, the abstraction level is not low enough to hand it over to developer for coding based on it and that’s why we need LLD.
  • Starting point or input criteria for LLD

Why we need LLD?

  • Helps in thinking ahead and being proactive for low level development related issue. It is a know fact that the early we find an issue the lesser it costs. LLD provides an opportunity for thinking ahead and being proactive about the problems that might come up at later stages of the development. Such issues can be addressed and fixed early in the design phase resulting in stable code and costing less.
  • Finalizing implementation strategy for a project before actually implementing it and hence reducing conflicts in development phase and resulting in faster development
  • Providing program specifications to the developers and sparing them for doing what they are best at (coding) rather than confusing them with very high level requirements from requirement documents
  • LLD provides further lower level abstractions of the requirements and can be handed over to developers for coding programs based on that. So LLD is the input for the development phase of the project

What is an ineffective design?

It is important for the management to be able to quantify effectiveness of  a design. Following are a few signs that a good manager should always be looking at to find out if the designs are effective or not.

Characteristics of an ineffective HLD

Following are some signs for ineffective HLD
  • LLDs are inefficient (More about it in later section about LLD)
  • There are too much discussions happening among high level designer and low level designers instead of resolving issues via query register. This is a clear hint that something is wrong somewhere.
  • HLD’s inefficiencies are felt by the LLD designers and still it is not able to convince them or adapt itself to address these inefficiencies
  • HLD does not clearly specifies the high level domain model and high level interaction diagrams for all the subsystems defined
  • All subsystems and system boundaries are not specified clearly
  • Non functional requirements are not addressed and fulfilled
  • In LLDs there are significant deviations from the guidelines and service interaction models established in HLD
  • Non technical people cannot understand it

Characteristics of an ineffective LLD

Following are some signs for ineffective LLD
  • Its inefficiencies are felt and told by the developers and still it is not able to convince them or adapt itself to address these inefficiencies before it is tool late
  • There are significant deviations from the design in the actual implementation
  • Developers have to refer to HLD, SRS, UCs and other documents etc to get more clarity on implementation process
  • The code is unstable and defect churn rate is high
  • The LLD is not being referred for the actual implementation
  • Huge refactoring activities are occurring in the code to stabilize it 
  • It has not been reviewed by the author of HLD on which the LLD is based on
  • Development is taking much more then planned effort

Root causes of design problems

Problems with HLD

  • Lack of technical and business understanding for HLD: One common cause for inefficient design is the fact that the designers/architects/developers are not competent enough to do that. A good mix of business and technical knowledge is required to come up with good HLD. Even a slightly biased view towards business or technology can disrupt the balance and effectiveness of the HLD
  • Technology dominates business: It is important for a designer to keep a view of all functional and non-functional requirements that need to be addressed and not to get carried away by the promises and robustness of new frameworks and patterns. No matter how smart technology, architectures, frameworks and patterns you have used, it is all waste unless it meets customer requirements and satisfaction. Loosing focus on these results in ineffective designs.
  • Jazzy UML: The design may or may not have strict UML compliant diagrams as long as it can clearly indicate the intentions of the designer and the design decisions that he has taken. On the contrary the designer at times are more concerned about the UML compliance then the real need of design which leads to jazzy and nicely looking but ineffective designs.

Problems with LLD

  • Lack of willingness to do LLD before coding: Now that programming languages are evolving to be at higher level of abstraction and delivery mindset is to deliver fast and quick. Design has taken a back seat. Developers feel that every hour they are not writing code shall result in extra hours of overtime later at the time of delivery. This mindset has taken its toll on the quality of design.
  • Experimenting with new concepts and patterns: LLD designers often want to introduce modern design patterns and concepts into there design just for the sake of it. This ends up in unnecessary complexity in LLD for developer to understand. Most of the times a lot of irrelevant information is added in the design which is seldom looked at by the readers
  • Less documented information and more Diagrams: Diagrams are great tool for visualizing design but they cannot convey the thought process that the designer has gone through to come up with that design. Diagram might reflect the design decisions that the designer of the LLD has taken but can not explain the motivation behind those decisions. It is important for developers to understand the reason and thinking that has gone into it to avoid conflicts and gain consent.
  • One template doesn’t fit all projects: Each software development project is different from the other in some aspects which imply that same information and document structure may not meet all the needs for all the projects. Some thought must go into customizing design template to meet specific needs of a project.

What is an effective design?

It is equally important to for the delivery management to know if the design is effective as it is to know if it is not. It is management’s responsibility to not only improve the inefficiencies but also to encourage and carry forward and continue with the best practices and good design methodologies that are being followed.

Characteristics of an effective HLD

  • It is at just right level of abstraction so that it makes sense for all stake holders of the project including developers, designers and business analysts. It should speak for and to all stake holders of the project
  • It should clearly state all the functional and nonfunctional requirements of the project along with providing high level solutions for achieving them
  • Clearly identify all subsystems of the project and defines clear cut framework for enabling communication within these subsystems
  • Describes all the layers of the system and integration points with external and internal services
  • It should be able justify all aspects of design decisions taken in the HLD. If any aspect of the design cannot be justified, then it is probably worth reevaluating
  • It should document all benefits that results from the design decision
  • LLDs built on the effective HLD are effective as well and so is the code built on those LLDs. However, waiting this long to check the effectiveness of an HLD is not a good idea and it may be too late for any corrective actions.
  • It clearly documents all assumptions, dependencies and risks for the project and allow project manager a chance to arrange the dependencies, confirm the assumptions and mitigate the risks

Characteristics of an effective LLD

  • An effective design is a live document that continues to navigate and control the implementation and development process for complete life cycle of the project. It is not dumped aside after first few references.
  • It meets the system requirements in a meaningful way and explains the thought process that has gone into for arriving to the design decisions taken in the design
  • Code developed on effective LLD is stable and maintainable and churn rate of defects is not much.
  • It is at the lowest level of abstraction and provides sufficient clarity for the developer to start coding right away without having to wait for detailed one to one sessions for starting coding for a LLD

How to write an effective design?

So far we have discussed the characteristics of an effective as well as an ineffective design and also the importance of the design and why do we need it. Now it is time that we should get our hands dirty with the real stuff “Writing effective design”. Here we shall discuss the prerequisites and the step by step approach to create effective HLDs and LLDs for a project.

Prerequisites for an effective HLD

  • Baseline Requirements: It is always difficult to aim and hit at a moving target. To come up with an effective HLD you need to have complete control over the scope of your design. You need to have base lined requirements before you start with the design. Obviously, the requirements are going to change sometimes but it is better to have clear agreement on base-lined requirements so that you have the scope of negotiation if the requirements change drastically during or after the design.
  • Architecture document: There is a wide misconception that architecture is a part of HLD. However, it should not be that way and in fact there should be a separate architecture document for the project describing at least business architecture, information architecture, data architecture and application architecture of the system. The HLD may include references to that document or some specific part of it to throw more light on some HLD decisions. HLD is meant to talk to and for all stake holders of the project and having all these architecture documented inside HLD makes it meaningless for a section of audience. Also, architecture will overshadow the HLD, reducing and diluting the impact that it should have.
  • Design template: Same HLD template may not be able to cater all types of projects as every project has a uniqueness that may require customized solutions. It is responsibility of the high level designer to think through and come up with a HLD and LLD template suitable for the project. These templates should typical have the buy-in from business analysts, low level designers, developers and other stake holders of the project
  • Thorough understanding of all these documents: This is the most important point and design must not start without the designer having clear understanding of the documents described above. 
  • Query Register: All queries and confusions that the designer has about anything mentioned in these documents should be resolved before the start of design and the same should be documented neatly in project specific query register so that analysts, designers and developers can access it if required to clarify or respond to queries. The query register should act as a knowledge base for the project.

Writing effective design

  • Scope of design is often considered as mere formality and is filled up as last part of the design and is mostly copied from the similar designs that are already available. Some minor tweaks here and there in the copied scope and is considered done. However, this is where we are risking the entire foundation of the project. 
    • For HLDs scope section should clearly list down the project specific feature scope and it should be in agreement with the business analysts. Break down the requirements in to deliverable UCs and if required split the complex uses cases into independently manageable features for better traceability. 
    • High level designer should not only be clear about the scope for HLD but should also have a clear view about the breakups and scope for the LLDs that need to be developed based on the HLD. He should have the clear release plan for the LLDs and corresponding implementations.
  • High level domain model should define the most important domain objects and the relationships among them. This is an important step as in most cases and with the modern persistence frameworks being used, this part is going to define your database which will eventually become the backbone of your application. Effective domain modeling is must for an effective HLD.
    • First step for the designer is to identify the domain objects or entities for you project. For this a common way is to do it by reading the requirements and list down all the nouns that appear in the requirement. This is followed by elimination of duplicates and visibly irrelevant ones from the list. After this step is performed you should be ready with the first draft of domain objects. At this point it is important to get these objects verified from the domain expert. In fact ideally domain expert should be part of the domain object identification session.
    • Once domain objects are identified, you would need to think about the relationships between them. At this point it is probably ok to draw your first class diagram which is the high level domain model. The domain model can be at a higher level specifying associations and dependency relationships between domain objects. It should be attached with some relevant notes for low level designers about how and what to elaborate in it in their LLDs
  • Components or sub-system identification is the next important activity that the designer has to perform for effective HLD. This is a brain storming exercise that must be done by the high level designers. After this activity is done the designer should have clear view of various subsystems and reusable components of the application and probably also have some idea about how the communication shall work within these components.
  • Once all internal and external subsystems and components are identified, the designer has to think through the integration issues and challenges for enabling smooth and effective communication among these systems and at this point should draw the component diagram for the application. How a component shall be reused should be decided at this point.
  • High level service model is the next step which should be performed and this step varies depending upon various factors related to development methodologies, technologies, frameworks, project domain and other project specific parameters.
  • After this step is complete you should have identified higher level components of your service layer. Service layer image should be drawn in your mind that you would need to resolve specific queries from the low level designers.
  • Identify subsystems and integration points may be the next in the HLD. This activity normally goes on throughout the development of your HLD. However it is important to list the entire internal and external component that you can think of. The list can be refined and polished as the HLD progresses.
  • At this point you should plot the component diagram for your project clearly specifying all subsystems internal and external and the interaction between them.
  • Provide realization of identified UCs in the scope section. After all the above steps are done you would need to provide the high level realization for your UCs. For measuring the real progress and completion status of the development it may be required to introduce the concept of features.
    • One feature may be involved in the realization of several UCs
    • Full realization of a UC requires the implementation of all supporting features.
  • At this point high level designer would need to provide some interaction diagrams like sequence diagram or collaboration diagrams for each feature and UC. These diagrams may be at a very higher level indicating the message flow among various objects across various layers of the application and providing clear integration points that should be further elaborated on in LLDs.
  • High level designer may have to discuss the integration points and strategies with the architecture team to ensure that it fits in the overall integration architecture of the system. Also designer has to ensure that architecture guidelines are followed and the HLD doesn’t violate any aspect of architecture. That is why it was mentioned that the high level designer should thoroughly understand the application architecture.

Prerequisites for an effective LLD

Apart from all the prerequisites for HLD, the LLD designer should ensure that following artifacts are available before he/she starts with the LLD
  • HLD is the most important artifact that the low level designer should have with him before he starts with the LLDs. HLD establishes a contract and vocabulary that needs to be respected by the LLD and implementation code.
  • High level scope for the LLD. This should typically be provided by the high level designer who has the complete visibility about entire project and all LLD developments that may be going on in parallel. High level designer may have decided to break the functionality defined in the HLD in multiple LLDs and may have some planning about each LLD release. LLD designer must have the clear understanding about the things that are in scope of his LLD and also the overall release plan so that he can align his priorities with the project priorities.
  • Query register should be in place while the HLD was being created and should have query resolutions from the HLD phase. LLD designer should have access to this knowledge base so that he has clarity on business related discussions that might have happened in HLD phase and that are logged in there.

Writing effective LLD

  • Designer should first have complete clarity on scope of his LLD. He should analyze the scope provided by the high level designer and if required break it up further into some lower level scope items.
    • Low level domain model is the next thing that the designer has to create.
    • For this the designer should identify all properties and behaviors for each domain object and it should be filled in each entity in the low level class diagram.
    • Designer should also identify other domain objects that may not have been identified in HLD but may be required in LLD for its implementation.
    • Low level designer should name domain object as per project specific naming convention and should also name all the relationships between domain object along with the multiplicity of the relationships. This step is required for the developer to come up with a stable entity relationship model quickly.
    • Low level domain model should be at as lower level as possible to the code and should have one to one mapping with entities that are created in the code. Don’t expect and allow the developer to think and apply there thoughts in this area as domain model is the back bone of your application which you cannot afford to expose to the developer to mess around with.
  • Realization of the UCs and the features is the next step for the LLD designer. Designer has to work closely with the developers and should be aware of the lower level APIs of the code. It is important for the designer to have this knowledge to come up with design that is closer to the reality.
    • First step here is to study the higher level interaction diagram that HLD designer has provided to understand the integration points with internal and external components and interaction between various layer of the system
    • Next step if to do some lower level detailing for each interaction defined. For example the HLD may only mention that your service needs to talk to some external LDAP service for authentication. The low lever designer should however provide the complete flow that realizes that interaction. This would include exact class names, method name, parameters and return type.
    • So the high level sequence diagram may have only important participating component but the lower level sequence diagrams should have all the participating objects for that flow at lowest level of abstraction.
    • Each method call should be clearly defined in the interaction diagram giving no way to any confusion for the developer. At this point the designer has to see his diagrams from developer’s perspective and scrutinize it for lack of information. All findings should be corrected.
  • Document all the design decisions in the notes within the sequence diagrams. It is better to document all the design decisions, assumptions and dependencies related to an interaction diagram in a note within that diagram itself rather then putting these all these in one section of the LLD. This helps the developer in relating to the thought process of the LLD designer while designing that flow.
  • If required it is also a good practice to provide pseudo code for complex part of an interaction diagram. More information you provide for your flows, lesser thinking developers have to do. It is a known fact the earlier you think of a problem better control you have on them. So it is worth thinking and writing pseudo code for complex part of the interaction in the LLD itself.
  • Please don’t experiment with new design patterns or fancy stuff just for the sake of it and stick to what is known to work with your project. It doesn’t mean that you should kill your ideas and creativity. All I am trying to convey here is that to be pragmatic and ensure that you have done POCs before you are introducing something new in your design. Putting things in design first and then doing the POCs for them is not a good idea and is a huge risk for your project.

Conclusion

  • The important skill that designer need to develop is to have a flexible and open mindset. He should be able to think from perspective of an analyst and a developer, both at the same time and need to maintain a fine balance between them for the benefit of project.
  • Without complete clarity about the domain it is nearly impossible to come up with effective design for applications that are meant to work on that domain.
  • It certainly helps the designer if he has good command over UML as it helps creating accurate interaction diagrams quickly. However, it is still possible to come up with good design even if the designer in not an expert at UML but has an analytical mindset and can clarify his intentions and design decisions in the design document.
  • High level domain model, components of the system, clear and justified design decisions, assumptions, dependencies and risks for the project are some important things that must be included in the HLD.
  • Low level designs should be at lowest level of abstraction and should not allow the developer to think and take decisions that are critical for the application. All these decisions should be taken while developing the LLD.
  • All POCs should be done prior to suggesting new design patterns, technologies and approaches that the developers are not familiar with. Suggesting first and doing later may be a risk for the project and is a big NO.


Comments