Learning more from crossing levels: Investigating agility at three levels of the organization

TL;DR

This paper emphasizes the importance of analyzing organizational agility across macro, meso, and micro levels to develop comprehensive, actionable guidelines for organizations seeking to enhance agility.

Contribution

It proposes a multi-level approach to understanding organizational agility, integrating insights from organizational science at three different abstraction levels.

Findings

Highlighting the need for multi-level analysis of agility

Designing studies for validated measurement of agility

Aiming to guide companies in targeted agility investments

Abstract

Scholars have tried to explain how organizations can build agile teams by only looking at one level of analysis. We argue in this short paper that lessons can be learned from organizational science results explaining variance on three different abstraction levels of organizations. We suggest agility needs to be explained from organizational (macro), the team (meso), and individual (micro) levels to provide useful and actionable guidelines to practitioners. We are currently designing such studies and hope that they will eventually result in validated measurements that can be used to prevent companies from investing in the wrong areas when trying to move towards more agility.

Figures

Full text

I Introduction

With the paradigm shift in software development toward more flexible and agile development processes, comes a need for more adaptive leadership [1]. However, this adaptive leadership always happens in a context, and this context is influenced by factors at different abstraction levels of the organization [2]. Some studies have shown that agility needs to be present at a strategic level as well to fully be implemented and for the organization to gain the intended advantages [3]. Also, in industry, many agile transitions fail because of lack of team capabilities [4]. Leadership can be viewed as a person’s ability to influence other individuals [5], but how people are influenced is also dependent on the maturity of teams, what phase the project is in, and the organization as a whole.

From a team perspective, measurement is essential to evaluate progress [6]. However, one of the most significant challenges is to measure at the right abstraction level [2]. Hackman [2] divides levels of abstraction into three levels, namely micro, meso, and macro. When researching the creation of agile teams, we want to focus on group-level development, hence, the meso level. However, explanations to variance we observe might very well come from the macro (organizational) level or the micro (individual) level. Therefore, if we want to understand agility and how such cultural change affects a software development organization, we should, preferably, look at all three levels. Therefore, we want to conduct research on all three levels simultaneously and urge other scholars to follow suit.

II Lessons learned from crossing levels in other fields

Hackman [2] presents some examples of when another level of analysis could reveal where the variance was hiding in some of his organizational studies. When investigating how airline cockpit crews deal with problems and issues before they became severe, they found no explanations across teams when measuring team performance effectiveness, which comprised the design of flying tasks and the composition of the crews. Looking one level down, they also investigated personal leadership styles, which also failed to explain any differences across airlines. The differences across airlines were purely organizational. Together, cockpit technology, the regulatory environment, and the culture of flying, completely constrained any crew from making any difference in relation to their performance. Not even accidents change the way these airlines assemble crews, but instead always result in technological fixes. In relation to agile teams, trying to explain why some teams manage to self-organize by only looking at team-level differences, might as well be futile. The properties of each organization might be the critical discontinuing factor that needs fixing before any team can move towards more agility. Also, training staff in agile leadership would then also not give any tangible results. Some scholars in the agile software engineering research community have suggested which strategic factors need to be in place for an agile transition (see, e.g. [7]), however, to get the full picture we need to investigate the micro, meso, and macro levels at the same time.

Next, we will present the three levels we investigate at the moment and, after that, we discuss potential outcomes of such research.

III Situational leadership

Instead of finding an optimal leadership style, Hersey et al. [8] suggested already in the seventies that a leader much adapt and change the style depending on the group. This model consists of maturity levels of group members, but also a balance between relation- and task-oriented behaviors. A leader should act differently depending on the needs of the group. Modern organizational psychology scholars also advocate a dynamic team leadership adapted to emerging needs even in the same situation [5]. The steps suggested by Hersey et al. [8] are illustrated in Figure 1.

IV An integrated model of group development

Due to many years of research on groups, there are a diversity of group development models. Even though the models are somewhat different, there is e a reoccurring pattern. A popular integrated model was suggested by Tuckman [9] in 1965 with the phases; Forming, Storming, Norming, and Performing. A bit more recently, Wheelan [10] created a model called the Integrated Model of Group Development (or IMGD) that can more or less be translated into the stages created by Tuckman [9]. The stages are shown in Figure 2. The stages can be compared to those of a human. We first figure out what world we are in (being a child), then we question the rules and their existence (adolescence), and we eventually somewhat find our place in this world and can focus more on how to develop and mature in life. It has been shown that, on an overall level, human groups go through similar stages [10].

Stage 1: Dependency and inclusion

The first stage is categorized by three main areas; concerns about safety and inclusion, member dependency on the designated leader, and a wish for order and structure. The group is supposed to become organized, capable of efficient work, and achieve goals, so the first state must have a purpose in getting there [10].

Stage 2: Counter-dependency and fight

When the group safely navigated through the previous stage, the group members have gained a sense of loyalty. When people feel safer, they will dare to speak up and express opinions that might not be shared by all members. The second stage of a group’s development is, therefore, a conflict phase where a fight is a must to create clear roles to be able to work together constructively. The members have to go through this to be able to trust each other and the leader [10].

Stage 3: Trust and structure

The third stage is a structure-developing phase where the roles are based on competence instead of striving for power or safety. Communication will be more open and task-oriented. The third stage of group development is characterized by more mature negotiations about roles, organization, and processes [10].

Stage 4: Work and productivity

The fourth and final stage (excluding the termination phase) is when the group wants to get the task done well at the same time as the group cohesion is maintained over an extended period. The group also focuses on decision-making and encourages task-related conflicts. This stage is a time of intense productivity and effectiveness, and it is at this stage the group becomes a team [10].

V Project and group life-cycles

It is more and more common to work in the form of projects within organizations, and a project goes through a set of stages that can be described as Idea, Planning, Execution, and Termination [12]. The first stage (the idea stage) is when the idea comes to place, and the company realizes that a project is needed around a specific goal. The planning stage comprises detailed planning, budgeting, scheduling, recruitment, and procurement. The execution stage is when the main project work gets done, and in the termination stage, the work decreases and the results are delivered to the customer.

The group development life-cycle described in Section IV and the project’s life cycle have a mutual effect on each other, and the problem is that the group’s development and the project’s development are rarely synchronized. Therefore, the group members could avoid sharing their opinions in the project planning stage, because the group is, psychologically, in the Forming stage. Also, the group might as well be in the conflict (Storming) stage during project execution, which is when the team’s performance needs to be at its peak. As the cycle moves along, the team might be starting to perform at its best when the project terminates [12]. All these effects, are, of course, suboptimal (see Figure 3).

VI Organizational maturity

In the group development model in Section IV, the project development model, presented in the previous section, and the situational leadership model in Section III, the phases are divided into a formation stage, a crisis stage, a norming stage, and a work stage. These can also be compared to, for example, Greiner’s [13] model for growing organizations (see Figure 4). A new organization that is starts with the entrepreneurial phase that is characterized by growth through creativity. The manager is here individualistic, creative and an entrepreneur. At the end of Phase 1, the organization has a leadership crisis. The following phase is the collective phase where growth is managed through directives that often come from the leader. At the end of this phase, the organization goes through a crisis of autonomy. Phase 3 is a phase of formalization where growth is managed through delegation. Total delegation and autonomy is given from the leader but ends with a crisis of control. The development phase (Phase 4) is to grow by coordination. The leader acts as a watchdog and this phase often ends up in a crisis of bureaucracy (sometimes called ‘red tape’). The final stage is recognized through team-oriented work and interpersonal skills where learning and innovation are present. While we realize that this publication is less scientific, it largely overlaps with other findings in organizational dynamics (see, e.g. [14]). These phases are very similar to the group development stages, human development, situational leadership, and project development. The challenge is to be aware of these and synchronize them carefully.

VII Discussion

The fact that groups have different needs over time could be connected to leadership research that focuses on that different leadership is needed in different contexts depending on what the group and group-members need. The situational leadership model [8] includes maturity levels of the group members, but also that balance is needed between relation- and task-oriented leadership behavior. They present the four different styles ‘telling’, ‘selling,’ ‘participating,’ and ‘delegating’ that are somewhat translatable to the group development stages. With more immature groups, telling and selling are needed approaches for the leadership to be successful, while at more mature stages, participating and, finally, delegating are more effective styles, since the group can self-organize. This means that the agile practices need to be implemented using a different leadership style depending on the maturity level of the group. In the agile method Scrum, we find descriptions of ‘agile leadership’ as being facilitating instead of directing [15, 16]. This works well in a mature group, but if that is not the case, the leader will need to behave differently to move the group forward. That is why situational leadership adapted to the group development stages needs to be incorporated into software engineering processes, and if they are not, leaders and managers will wrongfully try to follow a method that could be hindering the progress of that specific team.

Connecting the three levels of agility to the three more general abstraction levels of organizational theory is needed to fully understand agility. As mention, there is some evidence showing that agility in its broader sense is required at all levels of an organization to reach the intended increase in productivity and it is possible to change the practices on a more superficial level without the cultural change [3]. We, therefore, need the whole organization to be on board with our agile transition, which is not very surprising, but difficult in practice. Returning to Greiner’s [13] model of growing organizations (see Figure 4), we can see that small organizations, like start-ups, are agile by definition, i.e. they do not have any extensive overhead processes to satisfy when making decisions or negotiating with customers, and are characterized by creativity. However, they will have a leadership crisis sooner or later when the organization gets too large, with the reason being that the founder is then unable to obtain an overview and have control over all operations in the organization. This reasoning provides an explanation to why larger companies can not, and should not, function like small start-ups. However, there are different approaches to growing an organization than the classical command-and-control paradigm, i.e. to instead: ‘trust the collective intelligence of the system’ [17]; however, such organizations are still rare. We also believe it is too early to know the potential of such approaches at a much larger scale since they are more of an exception than a rule today, but we see potential in extending the agile methods with ideas of entirely autonomous teams.

In the software engineering community, a large focus has been on internal process maturity. However, it is also cumbersome to only look at the process maturity, like CMMI (Capability Maturity Model Integration) or the ISO/IEC 15504 SPICE (Software Process Improvement and Capability Determination), since we want agile value-driven organizations that use agile practices to implement agile principles. In addition, process maturity models are based on building customer trust by process infrastructure instead of working software and customer participation [18]. The strategists (managers/leaders) and the employees of an organization need to set the vision according to the organization’s purpose of existence in alignment with the agile principles (the cultural change) and then select agile practices to support that journey [19] in relation to all three abstraction levels of the organization, i.e. the micro, meso, and macro perspectives, namely the organizational, team/project, and individual levels. Understanding more about these interactions could increase the predictability of when agile transformation efforts succeed or fail and provide explanations for why.