Requirements Elicitation (ReqElic) in My Company

1. Introduction

A practicing business or requirements analyst (BoRA) often uses a creativity technique to help generate innovative ideas for requirements of computer-based systems. The most popular of these creativity techniques is brainstorming [Osborn]. One question that can be asked about these techniques is “What is the optimal size for a group of BoRAs using a creativity technique?” To attempt to answer this question, we developed a questionnaire to determine what creativity techniques practitioner BoRAs were using, what size groups they were actually using, and what size groups they felt that they should be using. We put this questionnaire at Google Docs ^[12], and began to solicit respondents through a variety of venues. We sought and received permission from the editors of this RE-Magazine to publish in its Academic Section a brief description of our research goals and an invitation to its readers to respond to the questionnaire.

2. The Data: Analysis and Results

As of the date on which we sat down to write this report, the questionnaire had 92 responses, and had been at that number for at least two months. This report distills the responses into meaningful observations that we believe are both interesting and useful to the readers of this RE-Magazine.

The first four questions, Q1 – Q4, attempt to identify the roles played by the respondent in requirements elicitation (ReqElic):

• Q1: I’m directly involved as a business or requirements analyst (BoRA) [My role in ReqElic]
• Q2: I’m directly involved as a software engineer [My role in ReqElic]
• Q3: I’m supervising ReqElic as a project manager or similar role [My role in ReqElic]
• Q4: I’m involved in ReqElic as a representative of the client or customer [My role in ReqElic]

The four possible answers to these and to most of the questions are:

• in all our projects
• in most of our projects
• in some of our projects
• in none of our projects.

As can be seen in Figure 1, giving a stacked bar graph that shows the distributions of the responses to Q1 – Q4, our respondents are working more as BoRAs or in some other capacity in requirements elicitation (ReqElic) than in more general software development.

The next three questions, Q5 – Q7, attempt to determine whether BoRAs are doing ReqElic as an individual activity or as a group activity. One of the questions caters to the possibility that one might be in a group in which each does ReqElic individually:

• Q5: Requirements are identified as an individual activity, by a single BoRA, working alone [Group vs. individual activity in ReqElic]
• Q6: Requirements are identified as an individual activity, by more than one BoRA, each working separately [Group vs. individual activity in ReqElic]
• Q7: Requirements are identified as a group activity [Group vs. individual activity in ReqElic]

Q5 is intended to capture the situation in which one and only one BoRA is identifying requirements for a project, and Q6 is intended to capture the situation in which a group of BoRAs is identifying requirements for a project, but each BoRA is working alone.

As can be seen in Figure 2, giving a stacked bar graph that shows the distributions of the responses to Q5 – Q7, a bit more than twice as many BoRAs are doing ReqElic as a group activity than as an individual activity in any way.

The next three questions, Q8 – Q10, refine Q7 for large, tight-deadline, and complex projects, respectively:

• Q8: Requirements are identified as a group activity in large projects [Group activity in ReqElic as a function of size of the project]
• Q9: Requirements are identified as a group activity in tight-deadline projects [Group activity in ReqElic as a function of deadline 9 of the project]
• Q10: Requirements are identified as a group activity in complex (innovative or multi-disciplinary) projects [Group activity in ReqElic as a function of complexity of the project]

The four possible answers to these three questions are:

• in all our X projects
• in most of our X projects
• in some of our X projects
• in none of our X projects,

where “X” is the nature of the project.

Recall that Q5 – Q7 show that a bit more than twice as many BoRAs are doing ReqElic as a group activity than as an individual activity in any way. Figure 3, giving a stacked bar graph that shows the distributions of the responses to Q8 – Q10, shows that this tendency is even more pronounced in large, tight-deadline, and complex projects.

The next three questions, Q11 – Q13, attempt to determine the techniques that the groups of BoRAs use to do ReqElic:

• Q11: We use JAD in group ReqElic sessions [The kind of ReqElic technique used in ReqElic Group activity sessions]
• Q12: We use brainstorming in group ReqElic sessions [The kind of ReqElic technique used in ReqElic Group 12 activity sessions]
• Q13: We use other creativity techniques in group ReqElic sessions [The kind of ReqElic technique used in ReqElic Group activity sessions]

As can be seen in Figure 4, giving a stacked bar graph that shows the distributions of the responses to Q11 – Q13, brainstorming and other creativity techniques are used about 5 times more frequently than is JAD.

Q14 attempts to determine the group sizes that are actually used in practice:

• Q14: Groups usually consist of [Size of the groups]

The possible answers are:

• 2 BoRAs
• 3 BoRAs
• 4 BoRAs
• 5 BoRAs
• > 5 BoRAs.

Figure 5 gives a bar graph that shows the distribution of the responses to Q14.

The sum of the answers to “5 BoRAs” and “> 5 BoRAs” is less than the answer to “4 BoRAs”. Therefore, we can say that among the groups, the group sizes, in order of decreasing frequency are:

• 2
• 3
• 4
• > 4.

We thought that it would be nice to put individuals, i.e., group size 1, into that list. We had not thought of asking that question when we wrote the questionnaire. However, it is possible to tease out an estimate from the existing data. From Q5 – Q7, we know the following numbers:

• Single working alone, all or most: 23
• Multiple working alone, all or most: 17.

Therefore, on average, BoRAs are working alone on all or most projects 20 times. Also from Q5 – Q7, we know the following number:

• Group, all or most: 45.

Therefore, 45/20 = 2.25 times as many people are working in groups than are working alone.

Among groups, the frequencies of replies for each group size are:

To normalize these frequencies relative to 13:

• 36/13 = 2.77
• 27/13 = 2.08
• 14/13 = 1.08
• 13/13 = 1.

The sum of these norms is 6.93. Thus, the estimated norm for group size 1 would be 6.93/2.25 = 3.08. Therefore, with the estimate included, the normalized frequencies for the group sizes are:

Thus, it looks like group sizes 1 and 2 are used with about the same frequency, with group size 1 being used a bit more than group size 2, but then the frequency drops off pretty quickly.

One could argue that instead of taking the average of 23 and 17 as the number of BoRAs are working alone on all or most projects, we should use the sum of 23 and 17, which is 40. If we use this number, then 45/40 = 1.125 times as many people are working in groups than are working alone. If we use this ratio to calculate the estimated normalized frequency for group size 1, we get 6.93/1.125 = 6.19 and the the normalized frequencies for the group sizes are:

With this estimate for group size 1, group size 1 is used more than twice as often as group size 2.

Let’s consider these two estimates for the relative frequency of group size 1 as the two extremes. It seems clear that individuals are used more often, and groups of size 2 or more are used less often. Within the group sizes of 2 or more, the larger the group size, the less frequently it is used. This pattern suggests that group sizes are determined by cost, with a group size’s frequency being inversely correlated with the group size’s cost.

Q15 attempts to determine the ideal group size, what a BoRA would choose if he or she could pick the optimal size.

• Q15: Size of ideal groups

The question asks the respondent to choose the most effective way to distribute 4 persons into groups that are doing ReqElic from among the following three choices:

• 4 individuals,
• 2 groups of 2, and
• 1 group of 4.

Figure 6 gives a pie chart that shows that in order of decreasing frequency:

• 2 2-person groups,
• 1 4-person group, and
• 4 1-person groups

are considered ideal (For uniformity in describing the data, the questionnaire and, therefore, this discussion call an individual a “1-person group”.).

If the ideal agreed with the estimated actual use frequencies, then in order of decreasing frequency, we would have:

• 4 1-person groups,
• 2 2-person groups, and
• 1 4-person group.

The obvious question is “Why are these answers so different?” One possibility is that we only estimated the number of 1-person groups that are actually used; we should have asked the Q5 – Q7 in different ways to get the actual numbers. However, the ordering of the group configurations is the same for either extreme estimate. So the ordering is probably accurate.

Another possibility is that the ideal does not match actual experience because what experienced people believe is ideal should not match actual experience if the actual experience happens for reasons other than to achieve the ideal. For example, it might be that in practice, group size is picked to satisfy cost constraints by people who are not actually doing the ReqElic, that is, by the managers of the BoRAs who are actually doing the ReqElic. However, if we ask the BoRAs who are actually doing the ReqElic to pick what they think are the ideal group sizes, then they will instinctively apply what they have observed about the behavior of the different size groups to come to their answers.

It could be that the BoRAs that do ReqElic have observed that 2-person groups are significantly more effective than 1-person groups and than 4-person groups. Perhaps, they have noticed that synergy is better than none; so a group is more effective in ReqElic than an individual. Perhaps, they have noticed on the other hand, that group-interaction overhead grows significantly with increasing group size; so that a 2-person group is more effective than a 4-person group.

Examination of the reasons given by the individual anonymous respondents for their answers shows that only 29 of the 39 respondents that chose “2 groups of 2” as ideal gave a response in the reason-for-your-answer section. Of these 39, only 22 actually gave what could be classified as a reason. The remaining 7 gave responses that discussed something other than a reason, e.g., discussing how it was important to staff the groups with designers and testers as well as users. Of the 22 respondents giving bonafide reasons, 14 (64% of the bonafide reasons, 48% of the responses) gave reasons dealing with the group’s dynamics; e.g., the most succinct of these is

“There needs to be a balance between collaboration and cohesion.”

The remaining 8 (36% of the bonafide reasons, 28% of the responses) of those giving bonafide reasons brought up issues other than the group’s dynamics, e.g.,

“I consider a back up team useful for critical projects” and
“pairs cross check each other”.

Thus, the data seem to support the second possibility.

3. Comparison of Results with Other Studies

The results of this questionnaire are of extreme interest to us because we had conducted controlled experiments to determine ideal group sizes in the use of a specific creativity techniques ^[4], namely POEMPcreate. (Space limitations prevent describing the technique beyond saying that it is a systematic way to get the brainstorming focused for an equal time on each possible combination of stakeholder viewpoints, to better cover the space of ideas. See references ^[1]–^[3] for details.) Very briefly, these results showed that among groups using POEPMcreate,

• a 2-person is significantly more effective than a 1-person group, and
• a 2-person is significantly more effective than a 4-person group,

where effectiveness is measured by both the number of raw requirement ideas generated and the number of new requirement ideas generated. Moreover, a 2-person group is so much more effective than the other size groups that

• 2 2-person groups generate more unique raw requirement ideas and more unique new requirement ideas than each of
• 4 1-person groups and
• 1 4-person groups.

That is, the power of the 2-person group is so high, that even if you eliminate duplicate requirement ideas among those generated by multiple groups working independently, 2 2-person groups outperform each of 4 1-person groups and 1 4-person groups. These results prompted us to develop and post the questionnaire described herein.

These experimental results were obtained by careful analysis of the data collected during carefully designed and conducted controlled experiments. They are matched by the questionnaire results, which were arrived at by informal observations by experienced BoRAs, who probably had no knowledge of any controlled experiment results. Wow!

The phenomenon that small groups generate more and better ideas than large groups has been observed in brainstorming ^[5]–^[11]. Perhaps the larger group-management overhead in a larger group is decreasing the larger group's effectiveness in requirement idea generation.

4. Conclusions

The questionnaire results and the cited other work seem to be saying the same thing to practitioners who need to generate creative requirement ideas for a system under development, whether by brainstorming or another creativity technique: If there are more than two BoRAs available to assign to the requirement idea generation task, these BoRAs should be put to work in groups of two. After each two-person group has generated its own list of requirement ideas, all the lists should be combined for pruning to the final list of ideas. Part of the pruning process, which normally aims to find the high quality ideas in the generated ideas, is elimination of duplicates among the groups’ lists.

Acknowledgements

Thanks to Pietro Marzani for helping to prepare the figures.

References

• [1] Mich, L., Anesi, C., Berry, D.M.: “Applying a pragmatics-based creativity-fostering technique to requirements elicitation”. Requirements Engineering Journal 10 (2005) 262–274
• [2] Mich, L., Berry, D.M., Alzetta, A.: “Individual and end-user application of the EPMcreate creativity enhancement technique to Website requirements elicitation”. In: Proceedings of the Workshop on Creativity in Requirements Engineering (CreaRE) at REFSQ’2010. (2010)
• [3] Sakhnini, V., Mich, L., Berry, D.M.: “The effectiveness of an optimized EPMcreate as a creativity enhancement technique for Website requirements elicitation”. Requirements Engineering Journal 17 (2012) 171–186
• [4] Sakhnini, V., Mich, L., Berry, D.M.: “On the sizes of groups using the full and optimized EPMcreate creativity enhancement technique for Web site requirements elicitation”. In: Proceedings of the Workshop on Creativity in Requirements Engineering (CreaRE) at REFSQ’2013. (2013)
• [5] Dornburg, C.C., Stevens, S.M., Hendrickson, S.M.L., Davidson, G.S.: LDRD final report for improving human effectiveness for extreme-scale problem solving: assessing the effectiveness of electronic brainstorming in an industrial setting. Technical Report SAND2008-5971, Sandia National Laboratories (2008), prod.sandia.gov/techlib/access-control.cgi/2008/085971.pdf
• [6] Dennis, A.R., Valacich, J.S.: Computer brainstorms: More heads are better than one. Journal of Applied Psychology 78 (1993) 531–537
• [7] Furnham, A., Yazdanpanahi, T.: Personality differences and group versus individual brainstorming. Personality and Individual Differences 19 (1958) 73–80
• [8] Taylor, D.W., Berry, P.C., Block, C.H.: Does group participation when using brainstorming facilitate or inhibit creative thinking? Administrative Science Quarterly 3 (1958) 23–47
• [9] Isaksen, S.G., Gaulin, J.P.: A reexamination of brainstorming research: Implications for research and practice. Gifted Child Quarterly 40 (Fall 2005) 315–329
• [10] Kohn, N.W., Smith, S.M.: Collaborative fixation: Effects of others' ideas on brainstorming. Applied Cognitive Psychology 25 (2011) 359–371
• [11] Aurum, A., Martin, E.: Requirements elicitation using solo brainstorming. In: Proc. 3rd Australian Conf. on Requirements Engineering, Deakin University, Australia (1998) 29–37
• [12] “Questionnaire: Requirements Elicitation (ReqElic) in My Company” at Google Docs docs.google.com/spreadsheet/viewform?fromEmail=true&formkey=dFI2UWx0MWJuRUdvQ1JNZnh1NFN0SGc6MQ