End-to-end product process with Design Thinking, Lean, Agile and DevOps

Business expectation from product team has evolved. Speed to delivery, with quality deliverables is the new normal. Continuous innovation in existing products and a continuous stream of new product introduction is the DNA of any growing organization.

Jainendra Kumar May 04th 2018

It is a new era of IT management in general. This era has spawned around conversations about Lean, Agile, DevOps, Design Thinking and related tools. Why are we hearing these more often now? It is a speed of supply issue—the complexity of our environment, fragmentation of processes, changing business needs, the structure of the organization making it difficult to move faster. Now the conversation in software development has moved from process maturity and compliance to speed, agility, nimbleness, and innovation along with quality and non-functional requirements. The end-to-end product process supply chain has integrated, on the left with the customer and on the right with operations.

Figure 1: End-to-end product process

Lean: In lean we talk about the voice of the customer. It provides an explanation for why many of the agile practices work. It offers insight into strategies for improving software processes. Lean principles provide a philosophical foundation for scaling agile approaches. It provides techniques for identifying waste and provides a premise for a pull system. Lean brings to the table, the premise of establishing the voice of the customer along with speed of moving from demand to delivery.

The key concept of lean thinking is thinking in terms of small batches as against monolithic projects. First, complete a small batch of work to gain feedback in terms of value it has created and then do the next batch of work. This premise of a small batch of work was the genesis of agile project management. The problem with Agile is that it mostly focused on the plan and build cycle, but not the final delivery which is when the final value is delivered. Also, we have to have a speed with a direction which requires creation of a horizontal cross-functional team which DevOps propagates.

DevOps: The overall speed or velocity of value creation system – demand (strategic, customer, operation, non-functional), plan, build and run is limited by the bottleneck in the critical path. Not only the bottleneck in the process but resource overburden also has impacted the speed. So one activity in the overburdened state in the end-to-end flow will slow down the value delivery. Agile takes care of plan and build flow for the value stream, but the run part become the bottleneck. This is where DevOps comes into picture. DevOps is basically going back to the lean premise of optimizing the full value stream – demand, plan, build and run as a whole. DevOps principles are – culture, automation, lean thinking, measurement, and sharing.

Each of these – Lean, Agile and DevOps– are accelerators and are interrelated. They are accelerating service management, software development, and project management processes. Design thinking which we will discuss in subsequent article brings in the innovation angle.

Lean thinking focuses on maximizing value delivery at speed and within the cost constraints. It starts with customer identification by specifying value, mapping value stream, flow creation (without waste), respond to customer (pull) and continue to iterate through continuous improvement (perfection). In ‘Implementing Lean Software Development’ book, Mary and Tom Poppendieck show how the seven principles of lean manufacturing can be applied to improve the whole IT value stream. These principles are:

7-lean-software-development-principlesFigure 2: 7 lean software development principles. Courtesy: ‘Implementing Lean Software Development’ book, Mary and Tom Poppendieck

1) Eliminate waste: Lean manufacturing has identified seven wastes which are: in-process inventory, over-production, extra processing, transportation, motion, waiting and defects which in software world translates to partially done work, extra features, relearning, handoffs, task switching, delays and defects respectively. Value stream mapping is a good tool to identify and eliminate waste.

2) Build in quality: The goal is to build quality in code from the beginning, not test it later. It requires a disciplined practice.  Test-driven development, Pair programming, Automation and Incremental development are techniques to induce discipline in the development process.  

3) Create knowledge: Software development is a knowledge-creation process, where overall architectural concept is sketched before coding, design and validation process continues in an iterative process where continuous feedback from customer and market is taken in an ongoing basis. It is a complex process so it is important to have a development process that encourages methodical learning throughout the cycle which feeds into retrospectives leading to continuous improvement.

4) Defer commitment: Speed is important but direction is the prerequisite. Decisions which are irreversible or costly to reverse should be delayed until sufficient information is collected and analyzed. Planning with assumptions is not a sin, it will change often, but implementing at speed in wrong direction is.

5) Deliver quickly:  Speed, speed and speed is the central premise of Agile and DevOps. Speed to value and speed to market are critical for delivering value in the hands of the customer as quickly as possible. It requires optimizing operation processes, where wastes are eliminated while maintaining high quality. Repeatable and reliable speed is impossible without excellent quality.   

The Lean way of delivering quickly isn’t working longer hours and weekends, or working recklessly for the sake of speed. It is about releasing shippable product sprint after sprint. A disciple approach of backlog grooming, unit test, code review, build automation, QA automation and continuous integration along with release candidate testing are performed seamlessly in every sprint. It is like to shop floor conveyor system with an added complexity of possible design level changes as the work progresses. It requires proper blue print / reference architectural work in the beginning of the project and continue to update it as new information is attained.  

Architecture: Product architecture for improving speed to market is about reusability, fault tolerance, reliability, and scale. Items that distract engineers from adding customer value to focus on operational issues impact speed to market; so architecting for fault tolerance, reliability and scale at the very first time is so very critical. Now the question is how we create such a comprehensive product yet improving speed to market? Reusability of infrastructure, platform services, middleware, business services and engineering productivity tools makes it possible.

Reusability means using a segment of source code multiple times with slight or no modification. Whether it is an operating system, virtual machines, containers, PaaS services, databases, message queues, 3rd party APIs, etc. are all reusable source code. All of these along with including microservice architecture in some of the other ways are influenced by Service Oriented Architecture (SOA).

Reuse is strongly emphasized within service-orientation; so much so that it becomes a core part of typical service analysis and design processes, and also forms the basis for key service models. SOA has an enterprise scope, where applications communicate with one another. It exposes services through standardized interfaces between applications, making it easier to use their data and logic in the next generation of applications. The microservices architecture has an application scope, with a focus on the structure and components within an application where an application is broken into smaller, completely independent components, enabling them to have greater agility, scalability, and availability. The concepts of SOA are present in modern day architectures and have changed in numerous ways. Integration implements, design constructs, and standards have evolved so that functions and data are more facilely brought to the surface. Service end point management has evolved, it is API oriented, simpler with security and monetization option.

Open source software adoption has increased in the last few years as organizations find it helpful in improving efficiency, interoperability, innovation and time to market. As per 2017 Black Duck Software survey, top three reasons for open source use are: 1) low cost with no vendor lock-in, 2) freedom to customize code and 3) rate of innovation. Since 2015 open source has become core in many organizations as its uses in an operating system, database, big data, and development tools have soared.

High-level reference architecture for reusability

Figure 3: High-level reference architecture for reusability

Nowadays business is driving enterprise architecture. Architects are taking an outside-in perspective to create a to-be state reference architecture and after that technology components are fit in the enterprise architecture blocks. It is also actionable as it is aligned with business priorities and gets developed in small pieces across multiple teams. Cross-team collaboration and shared services ensure that middleware and open source components are common across the enterprise so as to ensure larger reusability and a wide range of value creation.

6) Respect people: The Poppendiecks also observe that sustainable advantage is gained from engaged, thinking people. People are more important than process in Agile development. Lean development teams can encourage respect for people by communicating proactively and effectively, encouraging healthy conflict, bringing forward disruptive ideas, surfacing any work-related issues as a team, and empowering each other to do their best work. It is also important to create a culture of ownership, trust, and empathy for each other.

7) Optimize the whole:  It’s based on the idea that every business represents a value stream — the sequence of activities required to plan, architect, design, produce, test, and deliver a product or service to customers. If our goal is to deliver as much value to our customers as quickly as possible, then we have to optimize our value streams to be able to do just that. First step to optimize value stream would be to first identify them. Every players in the ecosystem starting from customers, partners, internal staff, and regulatory bodies have to be mapped and optimized.

Related: Innovation with Design Thinking, Lean, Agile and DevOps

Jainendra Kumar is VP, Engineering, RateGain

Disclaimer: This article is published as part of the IDG Contributor Network. The views expressed in this article are solely those of the contributing authors and not of IDG Media and its editor(s).