FurtaSpace – www.afurtado.net

Doing Software Engineering research requires well thought out planning and validation approaches. Studying such topics, I have come across a couple of interesting papers. A summary of their most interesting points (for me) follows below.

Writing Good Software Engineering Research Papers
(Mary Shaw)

· A research paper should include a number of questions stating your contribution (question you answered, why the reader should care larger questions addressed), your new result (previous work , new knowledge, details) and why readers should believe you (evaluation standard/method, evidence connecting claim and result).

· Types of software engineering research questions include: method/means of development, method for analysis or evaluation, design/evaluation/analysis of a particular instance, generalization or characterization, feasibility study or exploration.

· Program committees look for a clear statement on the specific problem you solved and how it will scale. Tell your answer and why it matters.

· Quantitative models have a better success rate (as being recognized by the scientific community) when compared to quantitative models. Tools and notations are well represented, usually as auxiliary results in combination with a procedure or technique.

· Types of software engineering research results include: procedure or technique (should be really a procedure, not advice or guidelines), qualitative or descriptive model (e.g., a taxonomy, a framework, well-organized interesting observations), empirical model, analytic model, tool or notation, specific solution/prototype/answer/judgment, report.

· You should explain your result in such a way that someone else could use your ideas. Precisely tell what’s new: the idea, the application of the idea, the implementation, the analysis or what?

· For tools: was the contribution the technique that is embedded in the tool, the effectiveness of the tool when compared to others, the applicability of the tool or what? Does the tool simply support the main contribution, or is itself a principal contribution? Can the idea be applied without the tool?

· Explain how you build atop previous work, otherwise you may damage credibility if the committee can’t tell whether you know about related work.

· A paper that simply reports on using numerous elements together is not enough, even if it’s well engineered. There must be an idea or lesson or model readers can take and apply to other situation.

· Types of software engineering validations include: analysis, evaluation, experience, example, persuasion, blatant assertion.

· Analysis, actual experience in the field and good use of realistic examples tend to be the most effective ways of showing why your result should be believed. Careful narrative, qualitative analysis can also work.

· Pilot studies that lay the groundwork for controlled experiments are often publishable by themselves.

· If you claim to improve on prior art, compare your result objectively to prior art.

· If you performed a controlled experiment, explain the experimental design: hypothesis, treatment, what is being controlled, collected data, how it was analyzed, significance of results, confounding factor, whether the conclusions follows rigorously from the experimental data.

· For PhD thesis validation section: global statements (“always, fully”) often require analytic validation, qualified statements (“X% of improvement”) can often be validated by evaluation or careful examination of experience existential statements (“we found an instance of”) can sometimes be validated by a single positive example. Students can restate thesis claims to reflect more precisely what the theses actually achieve.

· Suggested abstract structure: two or three sentences about the current state of the art identifying a problem, one or two sentences on the contribution, one or two sentences about the specific result and its main idea, one sentence about how the result is demonstrated or defended.

Selecting Empirical Methods for Software Engineering Research
(Steve Easterbrook, Janice Singer, Margaret-Anne Storey, Daniela Damian)

· Five classes of research methods are considered to be the most relevant to software engineering: controlled experiments, case studies, survey research, ethnographies and action research. The choice of methods depends upon the theoretical position (stance) of the researcher, access to resources and how closely the method aligns with the questions that have been posed.

· The selection of methods for a given research project depends on many local contingencies, including available resources, access to subjects, opportunity to control the variables of interest and the skills of the researcher.

· One of the first steps in choosing an appropriate research method is to clarify the research question.

· The most obvious question is not the best choice for a starting point.

· Defining the precise meaning of terms is a crucial part of empirical research and is closely tied with the idea of developing or selecting an appropriate theory.

· In the early stages of a research program, we usually need to ask exploratory questions to understand the phenomena: existence questions (Does X exist?), description and classification questions (What is X like? What is it for? How to categorize/measure it?), descriptive-comparative questions (How does X differ from Y?)

· Then we need to ask base-rate questions about the normal patterns of occurrence of the phenomena: frequency and distribution questions (How often does X happen? What is the average amount?), descriptive-process questions (How does X normally occur? What is the process by which it happens? What is its sequence of events?), relationship questions (Are X and Y related?)

· Then we should ask causality questions and, finally, design questions (How to achieve X?)

· Theories support the process of empirical induction because an individual study can never offer conclusive results. Each study adds more evidence for or against the prepositions of the theory. Without the theory, we have no way of making sense of the accumulation of empirical results.

· A controlled experiment is an investigation of a testable hypothesis where one or more independent variables are manipulated to measure their effect on one or more dependent variables. Each combination of variables is a treatment (configuration).

· A precondition of conducting an experiment is a clear hypothesis. The theory also helps to decide who the subjects are and what the tasks should be.

· A case study is something very different than just worked example. It can be exploratory (initial investigation to derive hypothesis and build theories) or confirmatory (to test existing theories).

· Case study research is most appropriate for cases where the reductionism of controlled experiments is inappropriate.

· The major weakness of case studies is that the data collection and analysis is more open to interpretation and researcher bias. For this reason, an explicit framework is needed for selecting cases and collecting data.

· A survey research is used to identify the characteristics of a broad population of individuals. It can be done through questionnaires for data collection, structured interviews or data logging techniques.

· Sampling bias causes problems in generalizing the survey results, because the respondents to the survey may not be representative of the target population. Low response rates increase the risk of bias.

· Ethnography is a form of research focusing on the sociology of meaning through field observation. The goal is to study a community of people to understand how members of that community make sense of their social interactions.

· In action research, the researchers attempt to solve a real-world problem while simultaneously studying the experience of solving the problem. There’s lots of debate on the validity of action research as an empirical method.

· Mixed methods research recognizes all methods have limitations and the weaknesses of one method can be compensated for by the strengths in other methods.

· Criteria for validity (for positivists/reductionists) include: construct validity (whether theoretical constructs are interpreted and measured correctly), internal validity (whether the results really follow from the data), external validity (whether claims for the generality of the results are justified) and reliability (whether the study yields the same results if other researchers replicate it).

· In reporting positivist empirical studies, it is important to include a section on threats to validity, in which potential weaknesses in the study design as well as attempts to mitigate these threats are discussed in terms of the criteria above.

· Strengths for improving validity (from a constructivist point of view) include: triangulation (use different sources and methods), member checking, rich/thick descriptions, clarify bias, report discrepant information, prolonged contact with participants, peer debriefing, external auditor.

· It is important to document the original planned research protocol, and all subsequent deviations to it, to allow other researchers to understand the study design, interpret the search results and replicate the study.

· Methods that are primarily qualitative include ethnography, case study and action research. Quantitative include controlled experiments and survey research.

· With regard to research and validation, it is important to plan ahead, for smooth analysis and interpretation of results.

Qualitative Methods in Empirical Studies of Software Engineering
(Carolyn B. Seaman)

· In software engineering, the blend of technical and human behavioral aspects lends itself to combining qualitative and quantitative methods in order to take advantage of the strengths of both.

· Advantage of qualitative methods: they force the researched to delve into the complexity of the problem rather than abstract it away. The results are richer but labor-intensive.

· Participant observation and interviews are one of the most common data collection methods. Interviews can be structured or unstructured (in the latter case, interviewees are the source of both questions and answers).

· “Coding” is the process of extracting vales for quantitative variables from qualitative data in order to perform statistical/quantitative analysis. This approach is especially difficult when the concept to be coded is subjective in nature, when the terminology used to describe it varies and is difficult to interpret, and when different data sources disagree.

· The distinction between qualitative and quantitative data has to do with how the information represented, not whether it is subjective or objective.

· The “constant comparison methods” is a data analysis method that tags qualitative data with terms, and then groups related information together to look for relationships.

· Cross-case Analysis a data analysis method that cross-compiles data.

· A hypothesis cannot be proven; it can only be supported or refuted. This is true independent from the usage of qualitative or quantitative methods.

· One of the most important ways to help confirm a qualitatively generated proposition is to ensure the validity of the methods used to generate it. Some validation concerns are: representativeness, possibility of researcher effects on the study, triangulation, anomalies in the data, negative case analysis, replication and member checking.

· Researchers have a marketing role as well, trying to promote the importance and usefulness of the empirical study in software engineering.

· Empirical studies can be classified in: blocked subproject-project (multiple projects and subjects to reduce bias), replicated project study (same project, multiple subjects), multiproject variation (single subject, projects before/after some treatment), single project study (single project, single subject, similar to a case study, possibly compared to some organization baseline).

BR,
— AFurtado

These are the projects of individuals/companies who presented in the Microsoft’s Development Tools Ecosystem Summit 2009 how they are extending Visual Studio:

• Atmel – IDEs for AVR microcontrollers
• Oleg Synch – T4 Toolbox: a set of ready-to-use code generation templates and supporting tools
• Collabnet – Subversion integration
• ComponentOne – enriched VS/SharePoint components, such as a spell-checker for VS and a Silverlight XAP file optimizer
• GamCom – Talmia: process workflow for Team System
• Intel – Parallel Studio: enhances Visual Studio for parallelism (C++)
• JNBridge – .NET / Java interoperability plug-in (for VS and Eclipse)
• Micro Focus – DevPartner Studio integration into VS2010
• Odin Technology – Coded UI test generation from Excel spreadsheets (cool!)
• OpenMake Software – extends build support for environments with multiple versions of .NET (VS 6.0 through 2010)
• PreEmptive – Runtime Intelligence: code instrumentation for finding different kinds of hotspots (such as code heavily used but poorly tested)

I didn’t speak this time, but you are always invited to try out my SharpLudus project which extends VS by combining model-driven development with XNA, hopefully enabling a more productive game development process.

BR,
— AFurtado

• Singletons are objects and it might be more intuitive to deal with them for state manipulation rather than through the use of static fields in static classes.
• In the same line, you can’t pass a static class as parameter, while no such limitation exists for the Singleton object.
• Static classes cannot implement interfaces. The class that defines the Singleton can.
• The global static class can be loaded too early, consuming valuable application resources, while the Singleton supports lazy allocation where an instance is created only as needed.
• Static classes do not benefit from default serialization mechanisms.

BR,
— AFurtado