Education of an Extreme Programmer

This interview with Francesco Cirillo was carried out by Marco, a University student, through an e-mail exchange which took place in January 2003.

MARCO: We will begin with an outline of your XP experience: How did you come to learn about this new method and who introduced it to you? Can you also describe major milestones, that is, how you began, what you did before, what you are doing now, and when you decided to seriously dedicate yourself to XP?

FRANCESCO: In 1999 I began reading the first few e-mails about Extreme Programming on major American mailing lists.

At that time, I worked for Sun Microsystems as a consultant, mentor, instructor and technical writer on software engineering issues – specifically on process improvement, software architectures and object-oriented development methods. As mentor I was working in Milan for a team at a prominent Italian bank. Along with training on traditional software engineering issues, we were working on the definition of a strict iterative and incremental development process that would enable their team to respond effectively to the requests of the traders. (That team would later be called Klondike).

At the same time I was consulting with another company in Florence, working to improve the development process of one of their teams. The objective was to enable that team to reach Level III and then Level IV of the Capability Maturity Model (CMM).

For years my work focused on studying and experimenting with many great software development methods, with the aim of verifying their effectiveness and applicability. My professional experience has grown by working with developers who failed on projects and learned lessons. This has always lead me to continue my search for something – anything – to deliver quality software on time and within an established budget, and yet with respect for human dynamics and with the objective of satisfying the needs of the client.

As I was saying, during that period a series of e-mails appeared, mostly signed Ron Jeffries, that sent a tremor through the world of mailing lists dealing with object-oriented analysis and design. The recurring subjects of those lists, “extend vs. uses,” “aggregation or composition?!” “How do I apply the substitution principle to my concrete case?” etc. were, for some time, substituted by e-mails from a strange man, Ron, who made strange proposals about an even stranger thing that he kept calling “eXtreme Programming”.

For a person like me, whose idols were – and still are – Parnas, Jacobson, DeMarco and Meyer, anyone who talked about “extreme programming” without first having done robustness analysis and strategic closures, was naive, to say the least. Yet something appealed to me. Ron had had project experience, and in my opinion, any project experience can provide useful indications for improvement.

I began researching XP through the few sites on the Internet available at the time. My interest in the concept grew, even though I still couldn’t understand how XPers solved complex architectural and design issues without anticipation. This was a very enticing challenge. Everything else about XP seemed to me to be a very well-organized way to work which required discipline and an in-depth background in the issues of traditional software engineering. I already thought that I could apply some of those things with the team I was mentoring in Milan. But first I had to understand them better.

I knew of Robert Martin’s reputation, and so I decided to participate in the first “XP Immersion” in Chicago in December 1999. I was the only person there from continental Europe. Among others, I met Kent Beck and the “strange man”:-) behind the signature of those ever-stranger e-mails. If, on one hand, those days were learning intense, on the other I was quite confused. I still couldn’t see how to make a system grow without the slightest bit of anticipatory design. At that point, I was willing to do anything to understand XP better. I asked Kent what I needed to do. His answer was “keep on studying and applying XP.” That’s what I did.

Once I got back to Italy I contacted a person Kent had worked with on this evolutionary approach to developing, someone who would help me answer my questions. That person was Massimo Arnoldi, CEO of Lifeware. After we got acquainted, he let me participate in their project with their team in Zurich.

I started working in Zurich: an unforgettable time, unforgettable pair sessions with Massimo, like the dinners in Italian restaurants in Zurich. (I earned my title as Chief Cooking Officer in the kitchens of Lifeware in Zurich, a title that is now official in the XP community at large!)

The Lifeware project experience is an excellent example of how a so-called small team using XP can create something as complex as an entire software system for managing an insurance company from front- to back-end, all in just a few years.

For a number of reasons – mostly logistical – I was forced to interrupt my work in Zurich, but by then I was changed irreversibly. I had finally overcome doubts I had about evolutionary development and I was ready for the next step.

I returned to Italy for good in September 2000 and, encouraged by Kent, I created the Italian list of Extreme Programming. I also founded XPLabs, where I was general director until December 2002, and CEO from January 2003. We decided to remodel a country villa just outside of Rome to use as our headquarters. (That project was done by using planning games and user stories – exploiting all the benefits of XP.) Curiously, the company’s beginnings boded well for communication – with five collaborators from five different countries on four different continents.

I resumed consulting with Sun for XPLabs, and proposed XP mentoring to the bank team in Milan. Sun was open to the idea, and the trusting Milan team accepted. The team took the name Klondike, and the result of our work was the Repo Margining System.

MARCO: What kind of resistance did you find, and what were the biggest problems you came up against as you approached XP?

FRANCESCO: Basically, I faced two challenges.

I had to let go of the conviction that change had to be predicted, and this frightened me. The objective of traditional software engineering is a very powerful idea: for every new requirement one must write new classes, not modify the ones that have already been created and tested. For seven years I worked with these principles. For a person who works this way, – deciding on primary abstractions, applying anticipatory design, “managing complexity with complexity,” drawing design patterns, – the first step in approaching XP is to stop and think about the current requirements for an application, and then open up to the idea of a small system that grows in which abstractions and design patterns emerge from the code and all behaviours are continually changing.

I had to understand the role of principles and design patterns from an evolutionary viewpoint, and learn to grow software. The knowledge of principles and design patterns is fundamental in XP. Unlike traditional object-oriented design, this knowledge is utilised to “recognize shapes,” rather than to “draw shapes.” What I needed to develop in order to use XP was the ability to listen to the code, letting abstractions and design patterns emerge by eliminating duplication. To do this, the ability to recognize those structures is essential. The only way to be able to recognize design patterns is to study them continuously.

MARCO: Does XP represent a revolution or an evolution? What is innovative about XP?

FRANCESCO: In my opinion, Extreme Programming represents an evolution of traditional software engineering. “Extreme” derives from the fact that a series of good, well-known traditional software engineering practices are taken to the extreme. “Programming” refers to the fact that our final goal is to deliver something that works, that gives value to the customer, workable code – to do this, we need to program.

It is worthwhile to emphasize that XP is a method that is firmly rooted in traditional software engineering. The real innovation lies, I see it, in the specific selection of a limited series of practices, and in the people-centered and labour-intensive character of the method. This means, most importantly, that the whole team shares a set of values: communication, simplicity, feedback, and courage. Only the spontaneous sharing of these values can produce the continued effort needed to give continuity to the practices involved.

XP is not a silver bullet; it’s not a panacea for software development problems. Without continuous refactoring, or without continuity in the practice of incremental test-first, any XP project would fail. I repeat, XP is not the solution to all evils of software development, but it is an important first step in that direction.

MARCO: What advantages have you found in using XP?

FRANCESCO: Customer satisfaction. The client can add, modify, or delete functionalities during every planning game – which we do each week. He or she can bring in revenue sooner by setting up shorter releases.

Effectiveness in introducing new functionalities. For the developers, change is viewed differently: instead of being something to fear, it becomes an excellent opportunity to improve the design of the system itself. Once the programmer acquires the ability to introduce new functionalities into the system, minimizing relative costs, every form of design anticipation ultimately leads to an undesirable increase in complexity and thus becomes ineffective.

Programmer satisfaction. The work is collaborative, creative, intense, focused, and motivating. Team members continually deliver value to the customer in the form of completed user stories. There is no routine overtime – it’s ineffective. People don’t work like machines.

MARCO: But what is the basis for these advantages?

FRANCESCO: Steve McConnell, in his book “Rapid Development,” proposes a series of reasons why projects fail: lack of user input, insufficient planning, gold-plating, abandoning planning under pressure, feature creep, heroism, weak personnel, etc. Reclassifying this list, two principal elements emerge: the inability to manage complexity and speed. Together, these two factors create a dangerous, vicious circle.

While complexity is public enemy number one for developers, speed is very often a real value for developers. There is a very simple test to verify if in your value in software development is speed. If, every minute of the day, you ask yourself “Am I going fast enough?” your value is speed. Obviously, this is a negative value to have for someone who develops software; among other things it’s a value that easily spreads within the team. The reality is that if we see this value as an engine of behaviours that take place within the team, it’s not a positive thing, because the final effect of this value is to make developers take short cuts in quality that result in a series of negative consequences: functionalities that don’t correspond with those requested by the client; rigid, fragile and immobile design; and an intensification in bug detection activity. By means of a phenomenon known in software engineering as devolution – a degenerative evolutionary process – if bug fixing is not supported by an adequate restructuring of system design, it leads to greater rigidity in the same system and a decline in relative performance. In this way, each of the above mentioned negative consequences caused by speed generates an undesirable increase in complexity.

So what does XP suggest? XP proposes to substitute the speed engine with one based on the values of communication, feedback, simplicity and courage. If one constantly asks, “Am I communicating? Am I giving feedback? Am I working in a simple way? Am I courageous?” i.e. “Do I ask for help when I need it?” then the XP engine is in place, and this is the necessary condition for doing XP. A series of 12 practices is based on these values, on this engine. These values, in turn, give continuity to the practices. XP values and practices are focused on reducing the complexity of the business on one hand, the problem area, and on the other reducing the complexity of the construction of software. Hence the aim of XP values and practices is combating developers’ number one enemy: complexity. The final effect is that these values, this XP engine, lead to speed, that is, an increase in the rhythm of delivery, while reducing complexity. These practices require constant application and that is why values must live.

In other words, the advantage of applying XP is an increase in speed in terms of the rhythm of delivery of value that derives from continual complexity reduction, both on the business and technical ends, obtained through the continual application of simple practices, made possible through the spontaneous sharing of values.

I repeat, complexity is the number one enemy of developers. XP provides effective instruments to combat this complexity by focusing on a series of values based on communication, instead of augmenting complexity, as many other methods do through the request for formality.

Any attempt to understand XP by starting with the practices and ignoring the values is destined to fail.

MARCO: In which projects were the benefits of XP most evident and in which less so?

FRANCESCO: My experience as mentor of XP teams, and as coach of the Bees Team at XPLabs has shown me that the more complex the project, the more obvious the benefits derived from the application of XP.

MARCO: What was the outcome of the XP projects in which you participated? Specifically, can you quantify the percentage of success in the use of XP for each one?

FRANCESCO: For the three XP projects taken on so far at XPLabs, the Bees Team, which I coach, is able to deliver the functionalities requested by the client, on time and within the allocated budget, working in an enjoyable way and learning something new every day. For this reason, I would say that all three projects are successful.

MARCO: What advice would you give someone who wants to approach this new methodology for the first time?

FRANCESCO: I would advise anyone who is sincerely interested in improving his or her development process to focus on enhancing these personal qualities and skills:

Humility – Delivering value to your client is certainly a bigger satisfaction than being speculatively effective by delivering complex and well-designed artifacts. But a greater responsibility is involved. In truly complex and critical situations, no team member alone can completely master a problem, but each one can have partial understanding of it. Through a real collaborative effort, this problem can be solved more effectively and with greater customer satisfaction. To be able to communicate with others, seek feedback, work in a simple way – perhaps by agreeing to start from the code – and have the courage to admit that no one is able to solve every problem alone requires humility. One must realize that each team member can learn from any other team member. Responsibility and collaboration are the winning words in XP. The presumption that you can completely master a problem and solve it in the violent dynamics of change proves to be less than optimal, devastating, you lose.

Open-mindedness – In order to change, a person must be willing to accept and try out new ideas. “Process improvement is difficult because people are reluctant to try new things. Their current habits seem so natural they can’t believe the change would help” (Watts S. Humphrey, Introduction to the Personal Software Process).

Critical ability and observation skills – To improve, a person needs to measure and evaluate the effects of change.

MARCO: Who would you advise against using XP?

FRANCESCO: First and foremost I’d like to recommend that everyone who works in the field of software engineering make a sincere effort to understand what XP is and how to apply it, beginning with the values, and then working on the practices.

Having said this, I would not recommend XP to anyone who, after understanding the foundations and trying out the practices, does not share the values. As I already said, the first step in XP is to substitute the values engine. Practices are constructed on values. In fact, values give the possibility of providing continuity to said practices. Without continuity, the castle of practices tumbles down. XP requires spontaneous sharing of values within the team.

MARCO: Is there any aspect of XP or anything about XP that you don’t believe in or that you think is not important?

FRANCESCO: Personally I completely share all values and practices proposed by XP.

What don’t I believe in?! I don’t believe in attacking XP the way some do without first making an effort to understand it.

MARCO: In your opinion, and based on your experience, do you think that an extreme methodology such as XP can work along side a traditional one on the same project?

FRANCESCO: My question is, why would we want to make XP work along side a traditional method?

Rick Kazman, my professor of software architecture at Carnegie Mellon University, used the expression: “Software Engineering = Cost Effectiveness.” I agree whole heartedly with this definition. Of course, our objective in software engineering is to be effective. So there is no better development method in theory. Better or worse in this context means more or less effective. Thus, the question is: Can XP be more effective by using it along side a traditional method, or cross-breeding it with other traditional techniques? My answer is no. Right now XP doesn’t need any traditional method or technique in order to be more effective.

The only reason why XPers may use other techniques is because the client explicitly requests it. If the clients asks for documentation with models in UML notation, or an estimate by applying function points or COCOMO or something else, the XP team will create a user story for that need, estimate the relative effort and then submit it to the client during the planning game. If the client decides to invest in that card, the team will develop it, but those techniques are not part of the production function of the team.

This does not mean that in the future XP might not benefit from new methods and techniques. XP was born to evolve. XPers are constantly searching for greater effectiveness. The most important XP practice is that by which the practices themselves can change. This entails a continuous effort focused on comprehending and verifying the effectiveness of methods and techniques, especially if derived from significant project experience. Learn from anyone, anywhere.

In terms of comparing the effectiveness of XP with other methods, there is one comment I would like to make. A number of people say that XP is appropriate for small to medium sized projects, comparing the reduced number of person-months for an XP project to those for a traditional project. I want to point out that there is a methodological error here. The two projects are based on two different software development methods, and hence two different production functions which imply diverse ways of dealing with complexity.

To compare the effectiveness of two different production functions, first you must be working under the same conditions. Given the same complexity of the functionalities to be resolved, and the same series of constraints, one must measure which of the two production functions requires less effort to reach the established goals.

Assuming that functional complexity can be compared to a given distance, we would need to find the car that, all conditions being equal – distance, requested time limit, maximum usury permitted, etc. – uses less fuel. If we wished, for example, to compare the effectiveness of XP to that of another method (that is, another production function) in a real life context, we must ask the following question: all conditions being equal, in order to develop an analogous system with the same capacity to respond to the dynamics of the business when constructing the software (same rhythm of delivery, same speed to maintain to satisfy the client, etc.), how much effort must be applied using the various production functions? The car that uses less fuel under the same conditions is the one that we will chose.

If it is true that a project with so many person-days represents higher communicative/constructive complexity, it is also true that this can not be the best way to organise production factors. More importantly, that complexity remains limited to the production function of the development team. The hypothesis of rational behaviour holds that an entrepreneur will minimise production costs – at the same quality level – optimising resource allocation.

Reduced effort (in person-months) needed to solve a problem in conditions of violent change, which requires the development of a system with thousands of application classes, could mean that the process followed is very effective and therefore relevant.

At Lifeware, for example, in only a few years a so-called small team of programmers using XP was able to support the needs of an extremely dynamic and complex business, an insurance company, with fast and frequent deliveries, guaranteeing full satisfaction to the client.

MARCO: Do you believe XP really has the necessary qualities to gain wide popularity? Specifically, in our country would it be possible?

FRANCESCO: One of the fundamentals of XP is the clear division between business and development. On one hand is a customer who has a problem and is interested in solving it, or who has a business idea and wants to develop it. On the other hand are developers who deliver the functionalities requested and carry out changes quickly and cheaply, encouraging new requests from the customer. In XP, the customer and the developers interact continuously on a single team.

As far as the situation in Italy, I would make a distinction between large companies and small to medium sized ones.

Continued unification in Europe and subsequent growth in new markets is increasing competitive pressure on large Italian companies. This means an effective solution must be found to accommodate continuous demands for change. Technocratic behaviour and resistance to change can no longer be allowed. Software for some time now has become a way to characterize and identify business services offered by a company. Business dynamics and software construction are more closely linked than ever. In this context, investments in long term, multi-million dollar software projects with long term financial/economic returns are no longer possible.

Inevitably corporate executives see the need to adopt new methods for software development. They are looking for more effective methods, methods that are reactive to change and capable of supporting business development, also from a financial point of view. Attention to the aspects of the definition of software functionalities to develop becomes a vital, strategic element, in which the creative component of defining business belongs solely to the company through the vision of management. The process of defining business has to be checked continually against reality in order to be effective. Introducing change at any given moment in any given direction becomes indispensable. All this can happen only if there is a development team capable of dealing with change quickly and economically. The above mentioned separation between business responsibility, or defining the functionalities to develop, and technical responsibility, that is how those functionalities will be developed, requires continual collaboration between the two sides which materializes in the work of the team.

At this moment XP is the ideal ally of major Italian companies. Worth noting is the fact that an XP team gets hired for its ability to minimize the cost of change, to continually adjust the system to the requests of the customer, to test new ideas, and in doing so to maximize the customer’s business development. My experience has been that having the customer participate in the project team is an aspect of XP that large companies more often see in a favourable light, once initial (mostly organizational) difficulties are overcome. The reason for this is because the results are impressive.

Just as many large Italian corporations could gain a competitive edge from applying XP, there are many more small and medium sized companies that could profit in the same way. These companies know their problems and are full of ideas for solutions that they want to implement with custom-tailored software. Mistreated for years by software houses that were very good at selling an initial product, and just as good at retreating at the first request for change, these small and medium sized companies are ideal for the application and diffusion of XP.

MARCO: To what extent do you believe XP will be adopted? What do you think is needed in order for XP to become widely used?

FRANCESCO: The answer to the first question depends to some degree on how effective we are at answering the second.

What needs to be done so that XP becomes more widely used? Point by point:

We need to demonstrate replicability by means of controllable results. This isn’t a quest for a useless inductive demonstration of the validity of XP (as Popper asserts: “No number of sightings of white swans can prove the theory that all swans are white. The sighting of just one black one may disprove it”), but serves to give us a better understanding of XP. According to Popper’s theory of epistemological fallibility, the content of a scientific theory consists in its consequences. If we want to prove it, it must be controllable, in principle. In other words, we must be able to extrapolate consequences from the theory, consequences that can be refuted, that is, falsified. In this, we’re testing a logical asymmetry between verifiability and falsifiability of a theory, by which billions and billions of confirmations do not make a theory certain, while only one negative fact falsifies the theory from a logical viewpoint. XP still has a great deal to teach us and the only way to learn is to observe, extracting measurable and controllable consequences. In an epistemological sense, our (XPers’) efforts must be focused on making XP controllable and therefore, paradoxically, falsifiable, to discover how to improve it. Popper writes: “I shall not require of a scientific system that it shall be capable of being singled out, once and for all, in a positive sense; but I shall require that its logical form shall be such that it can be singled out, by means of empirical tests, in a negative sense: it must be possible for an empirical scientific system to be refuted by experience.” The primary objective of XPLabs has always been to create a series of labs in a number of different sectors in order to extract informative content. All our labs are fully tracked, and we publish related documentation. This is the spirit of XPLabs. We need feedback to improve. Any comments are precious to us.

We need to work harder to make it clear that XP is not only a set of practices, but more importantly it is a set of values spontaneously shared in the team. I notice more and more often in the presentations I do that the people I speak to focus on practices, minimizing (sometimes maliciously) the importance of the values (as if including the values is nothing more than a good marketing idea). Instead, the spontaneous sharing of the values proposed by XP is fundamental for the success of the team. It exalts two aspects that are truly relevant in the work of an XP team: responsibility and collaboration.

We need a systematic approach to studying software development methods. “Software engineering has only been around a few years; it’s still young?” That refrain is well-worn and well-known by now. Any attempt to see the study of software development methods as a “new” subject, in some way separate from other subjects, can only lead to useless limitations. The study of methods of software development is the offspring of the philosophy of logic, anthropology, architecture, psychology, biology, economics, etc. It is not a young science, but is rooted in centuries of human knowledge. With humility we need to learn to gain knowledge from all these subjects effectively.

More “availability” from academicians is necessary. There is a need for clarity, simplicity, but above all both respect for those who offer a different vision, and exactness in the application of scientific method. The philosophy of science teaches that true theories do not exist, only theories that are more or less close to the observation. All theories are falsifiable. Progress comes about by falsifying theories in a scientific way. To manage not to fail on a project and support our client in the development of business is already, in and of itself, a sufficiently complicated thing. It is necessary that XPers and the academic world collaborate more actively in order to comprehend and resolve, with scientific method, the problems of software development.