Options:

Luisa Mich Victoria Sakhnini Daniel Berry

Requirements Elicitation (ReqElic) in My Company

Preliminary Results of a Questionnaire


Have you ever wondered what is the optimal size for a group of business or requirements analysts that is doing creative requirements elicitation? Have you developed an intuition for what this optimal size is? Find out what your fellow analysts are thinking about these questions.

1. Introduction

A practicing business or requirements analyst (BoRA) often uses a creativity technique to help generate innovative ideas for requirements of computer-based systems. The most popular of these creativity techniques is brainstorming [Osborn]. One question that can be asked about these techniques is “What is the optimal size for a group of BoRAs using a creativity technique?” To attempt to answer this question, we developed a questionnaire to determine what creativity techniques practitioner BoRAs were using, what size groups they were actually using, and what size groups they felt that they should be using. We put this questionnaire at Google Docs [12], and began to solicit respondents through a variety of venues. We sought and received permission from the editors of this RE-Magazine to publish in its Academic Section a brief description of our research goals and an invitation to its readers to respond to the questionnaire.

2. The Data: Analysis and Results

As of the date on which we sat down to write this report, the questionnaire had 92 responses, and had been at that number for at least two months. This report distills the responses into meaningful observations that we believe are both interesting and useful to the readers of this RE-Magazine.

The first four questions, Q1 – Q4, attempt to identify the roles played by the respondent in requirements elicitation (ReqElic):

The four possible answers to these and to most of the questions are:

As can be seen in Figure 1, giving a stacked bar graph that shows the distributions of the responses to Q1 – Q4, our respondents are working more as BoRAs or in some other capacity in requirements elicitation (ReqElic) than in more general software development.

Figure 1: Responses to Q1 – Q4

The next three questions, Q5 – Q7, attempt to determine whether BoRAs are doing ReqElic as an individual activity or as a group activity. One of the questions caters to the possibility that one might be in a group in which each does ReqElic individually:

Q5 is intended to capture the situation in which one and only one BoRA is identifying requirements for a project, and Q6 is intended to capture the situation in which a group of BoRAs is identifying requirements for a project, but each BoRA is working alone.

As can be seen in Figure 2, giving a stacked bar graph that shows the distributions of the responses to Q5 – Q7, a bit more than twice as many BoRAs are doing ReqElic as a group activity than as an individual activity in any way.

Figure 2: Responses to Q5 – Q7

The next three questions, Q8 – Q10, refine Q7 for large, tight-deadline, and complex projects, respectively:

The four possible answers to these three questions are:

where “X” is the nature of the project.

Recall that Q5 – Q7 show that a bit more than twice as many BoRAs are doing ReqElic as a group activity than as an individual activity in any way. Figure 3, giving a stacked bar graph that shows the distributions of the responses to Q8 – Q10, shows that this tendency is even more pronounced in large, tight-deadline, and complex projects.

Figure 3: Responses to Q8 – Q10

The next three questions, Q11 – Q13, attempt to determine the techniques that the groups of BoRAs use to do ReqElic:

As can be seen in Figure 4, giving a stacked bar graph that shows the distributions of the responses to Q11 – Q13, brainstorming and other creativity techniques are used about 5 times more frequently than is JAD.

Figure 4: Responses to Q11 – Q13

Q14 attempts to determine the group sizes that are actually used in practice:

The possible answers are:

Figure 5 gives a bar graph that shows the distribution of the responses to Q14.

Figure 5: Responses to Q14

The sum of the answers to “5 BoRAs” and “> 5 BoRAs” is less than the answer to “4 BoRAs”. Therefore, we can say that among the groups, the group sizes, in order of decreasing frequency are:

We thought that it would be nice to put individuals, i.e., group size 1, into that list. We had not thought of asking that question when we wrote the questionnaire. However, it is possible to tease out an estimate from the existing data. From Q5 – Q7, we know the following numbers:

Therefore, on average, BoRAs are working alone on all or most projects 20 times. Also from Q5 – Q7, we know the following number:

Therefore, 45/20 = 2.25 times as many people are working in groups than are working alone.

Among groups, the frequencies of replies for each group size are:

Size Frequency
2 36
3 27
4 14
> 4 13

To normalize these frequencies relative to 13:

The sum of these norms is 6.93. Thus, the estimated norm for group size 1 would be 6.93/2.25 = 3.08. Therefore, with the estimate included, the normalized frequencies for the group sizes are:

Size Normalized Frequency
1 3.08
2 2.77
3 2.08
4 1.08
> 4 1.00

Thus, it looks like group sizes 1 and 2 are used with about the same frequency, with group size 1 being used a bit more than group size 2, but then the frequency drops off pretty quickly.

One could argue that instead of taking the average of 23 and 17 as the number of BoRAs are working alone on all or most projects, we should use the sum of 23 and 17, which is 40. If we use this number, then 45/40 = 1.125 times as many people are working in groups than are working alone. If we use this ratio to calculate the estimated normalized frequency for group size 1, we get 6.93/1.125 = 6.19 and the the normalized frequencies for the group sizes are:

Size Normalized Frequency
1 6.19
2 2.77
3 2.08
4 1.08
> 4 1.00

With this estimate for group size 1, group size 1 is used more than twice as often as group size 2.

Let’s consider these two estimates for the relative frequency of group size 1 as the two extremes. It seems clear that individuals are used more often, and groups of size 2 or more are used less often. Within the group sizes of 2 or more, the larger the group size, the less frequently it is used. This pattern suggests that group sizes are determined by cost, with a group size’s frequency being inversely correlated with the group size’s cost.

Q15 attempts to determine the ideal group size, what a BoRA would choose if he or she could pick the optimal size.

The question asks the respondent to choose the most effective way to distribute 4 persons into groups that are doing ReqElic from among the following three choices:

Figure 6 gives a pie chart that shows that in order of decreasing frequency:

are considered ideal (For uniformity in describing the data, the questionnaire and, therefore, this discussion call an individual a “1-person group”.).

Figure 6: Responses to Q15

If the ideal agreed with the estimated actual use frequencies, then in order of decreasing frequency, we would have:

The obvious question is “Why are these answers so different?” One possibility is that we only estimated the number of 1-person groups that are actually used; we should have asked the Q5 – Q7 in different ways to get the actual numbers. However, the ordering of the group configurations is the same for either extreme estimate. So the ordering is probably accurate.

Another possibility is that the ideal does not match actual experience because what experienced people believe is ideal should not match actual experience if the actual experience happens for reasons other than to achieve the ideal. For example, it might be that in practice, group size is picked to satisfy cost constraints by people who are not actually doing the ReqElic, that is, by the managers of the BoRAs who are actually doing the ReqElic. However, if we ask the BoRAs who are actually doing the ReqElic to pick what they think are the ideal group sizes, then they will instinctively apply what they have observed about the behavior of the different size groups to come to their answers.

It could be that the BoRAs that do ReqElic have observed that 2-person groups are significantly more effective than 1-person groups and than 4-person groups. Perhaps, they have noticed that synergy is better than none; so a group is more effective in ReqElic than an individual. Perhaps, they have noticed on the other hand, that group-interaction overhead grows significantly with increasing group size; so that a 2-person group is more effective than a 4-person group.

Examination of the reasons given by the individual anonymous respondents for their answers shows that only 29 of the 39 respondents that chose “2 groups of 2” as ideal gave a response in the reason-for-your-answer section. Of these 39, only 22 actually gave what could be classified as a reason. The remaining 7 gave responses that discussed something other than a reason, e.g., discussing how it was important to staff the groups with designers and testers as well as users. Of the 22 respondents giving bonafide reasons, 14 (64% of the bonafide reasons, 48% of the responses) gave reasons dealing with the group’s dynamics; e.g., the most succinct of these is

“There needs to be a balance between collaboration and cohesion.”

The remaining 8 (36% of the bonafide reasons, 28% of the responses) of those giving bonafide reasons brought up issues other than the group’s dynamics, e.g.,

“I consider a back up team useful for critical projects” and
“pairs cross check each other”.

Thus, the data seem to support the second possibility.

3. Comparison of Results with Other Studies

The results of this questionnaire are of extreme interest to us because we had conducted controlled experiments to determine ideal group sizes in the use of a specific creativity techniques [4], namely POEMPcreate. (Space limitations prevent describing the technique beyond saying that it is a systematic way to get the brainstorming focused for an equal time on each possible combination of stakeholder viewpoints, to better cover the space of ideas. See references [1]–[3] for details.) Very briefly, these results showed that among groups using POEPMcreate,

where effectiveness is measured by both the number of raw requirement ideas generated and the number of new requirement ideas generated. Moreover, a 2-person group is so much more effective than the other size groups that

That is, the power of the 2-person group is so high, that even if you eliminate duplicate requirement ideas among those generated by multiple groups working independently, 2 2-person groups outperform each of 4 1-person groups and 1 4-person groups. These results prompted us to develop and post the questionnaire described herein.

These experimental results were obtained by careful analysis of the data collected during carefully designed and conducted controlled experiments. They are matched by the questionnaire results, which were arrived at by informal observations by experienced BoRAs, who probably had no knowledge of any controlled experiment results. Wow!

The phenomenon that small groups generate more and better ideas than large groups has been observed in brainstorming [5]–[11]. Perhaps the larger group-management overhead in a larger group is decreasing the larger group's effectiveness in requirement idea generation.

4. Conclusions

The questionnaire results and the cited other work seem to be saying the same thing to practitioners who need to generate creative requirement ideas for a system under development, whether by brainstorming or another creativity technique: If there are more than two BoRAs available to assign to the requirement idea generation task, these BoRAs should be put to work in groups of two. After each two-person group has generated its own list of requirement ideas, all the lists should be combined for pruning to the final list of ideas. Part of the pruning process, which normally aims to find the high quality ideas in the generated ideas, is elimination of duplicates among the groups’ lists.

Acknowledgements

Thanks to Pietro Marzani for helping to prepare the figures.

References



Luisa Mich

Luisa Mich is an Associate Professor of Computer Science at the University of Trento, Italy. Her research interests include requirements engineering, creativity and web strategies. She is an author of more than 150 papers.
 

Victoria Sakhnini

Victoria Sakhnini is an adjunct lecturer at the Cheriton School of Computer Science at the University of Waterloo. Her research includes empirically validating the effectiveness of a new technique to enhance requirements-elicitation creativity and effectively teaching computer science principles.
 

Daniel Berry

Daniel M. Berry got his B.S. in Mathematics from Rensselaer Polytechnic Institute, Troy, New York, USA in 1969 and his Ph.D. in Computer Science from Brown University, Providence, Rhode Island, USA in 1974. He was on the faculty of the Computer Science Department at the University of California, Los Angeles, California, USA from 1972 until 1987. He was in the Computer Science Faculty at the Technion, Haifa, Israel from 1987 until 1999. From 1990 until 1994, he worked for half of each year at the Software Engineering Institute at Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, where he was part of a group that built CMU's Master of Software Engineering program. During the 1998-1999 academic year, he visited the Computer Systems Group at the University of Waterloo in Waterloo, Ontario, Canada. In 1999, Berry moved to what is now the the Cheriton School of Computer Science at the University of Waterloo. Between 2008 and 2013, Berry held an Industrial Research Chair in Requirements Engineering sponsored by Scotia Bank and the National Science and Engineering Research Council of Canada (NSERC). Prof. Berry's current research interests are software engineering in general, and requirements engineering and electronic publishing in the specific.