Engineering activities vs. management activities
This is a letter to my friend Larry Maccherone, who is an expert on software processes and Agile in particular.
Larry,
Here’s something I’d like your opinion on, and it might become another blog post for you. I believe we should try to understand engineering activities separately from management activities. It’s tough to disentangle them, but regardless of management process you’ll need to decide what the system should do, assign responsibilities to parts, and think about (analyze) if indeed the thing you designed/coded will do what you were asked functionally, and if it will have acceptable qualities (performance, modifiability, etc.)
I also believe that success depends on aligning the management and engineering activities. XP and other agile practices are actually a nice merging of the two. Engineers on space systems may be forced into waterfall processes (you can only launch a rocket once) and XP developers may use iterative processes (you can relaunch the website each week), but both have the same engineering vocabulary — processes, modules, analyses, tests, etc. My goal with the architecture book is to help popularize the software architecture ideas (mostly the CMU/SEI versions of those ideas) and help them become de facto standards across all engineers/developers. For example, Szyperski’s definition of a component is pretty different from the CMU/SEI definition, and his book is not so old. My book is actively trying to avoid promoting a particular management process but instead showing that the core engineering ideas and activities are compatible with a variety of management processes (agile in particular), but that doesn’t make the book title “Agile Software Architecture”.
I hate to think of a situation where two trained engineers are at a whiteboard trying to solve a problem and they do not have the right abstractions / models to think about and solve their problems. Sure, they’re bright, so they figure something out ad hoc from first principles and some fuzzy intuition about chunking large bits of software. But I’d like them to have a standardized, shared vocabulary of parts (components, connectors, protocols, properties, etc.) that are effective for understanding the problems they face. (That’s the “abstractions” part I discuss in the book’s introduction). And over time we’d be more effective at sharing our solutions (which is the “knowledge” part from the intro) the way design patterns were a success. Right now, most folks have a hard time seeing that the styles used in operating systems are effective there, and the styles used in IT systems are effective there, but a client-server OS or a CSP billing system is probably a bad match. Another example of this is folks arguing that “your language is stupid”, because it’s not a good match for problems from a different domain (e.g., Java is probably not the best choice for OS device drivers and ditto for C and billing systems).
I’d be happy to hear your thoughts on any of this, but in particular on these questions:
- Do you agree that we should strive to separate management processes from engineering activities?
- Do you think that components, connectors, etc. are effective abstractions?
- Do you believe the majority of developers understand these abstractions?
Regards,
-George
Comments
I don't see the advantages of components, connectors, and ports
Whew! What a lot to think about. I had to go back and re-read your intro before starting to respond. While I was re-reading more carefully, I decided to make some suggestions which I emailed to you. It looks like a lot of edits but most of them serve to sharpen your point that you are proposing a new abstraction expressed with connectors, components, and ports as the next escalation in the arms race against complexity and scale.
OK now to your questions.
First the short form:
1. Do you agree that we should strive to separate management processes from engineering activities? LM> Yes, and I think others will also. In particular, this issue has been hashed around in the agile community in the last few years, with many folks saying that XP is more collection of engineering practices, whereas Scrum (and others) are process definitions that focus on project management.
2. Do you think that components, connectors, etc. are effective abstractions? LM> No (sorry), but I don’t know enough to say conclusively. I should read your other draft chapters. Maybe starting with architectural patterns and considering components, connectors, and ports incidental would be more palatable for your book.
I think most developers will prefer to use object-oriented terms and supplement that with OO design pattern terms when necessary. For instance, I recently worked on a team where we had to expand our system to support a remote service for storage (Amazon S3), whereas the system currently supported a local embedded storage. The conversation we had went something like this:
Developer A: Let’s just put a remote proxy in front of the embedded storage interface with the same methods as the current embedded storage API. We can block on calls to S3.
Developer B: Yeah, and if that’s too slow, we can add pre-fetch to the remote proxy.
Developer C: I disagree. By doing that, any code calling this might not anticipate the delay. The more general model is the async one. Let’s bite the bullet now and implement a pub-sub “observer” eventing system and create our own internal async interface, which will look a lot like S3’s. Then when using the local storage, we’ll just immediately trigger the callback event.
We all agreed with Developer C and that’s what we did. How would you address this conversation with ports, connectors, and components? Is that approach any more concise or revealing? If your response is that you are really targeting higher level issues, then you’ve greatly limited your audience because what I described is closest to the highest level that most developers deal with. I’ll follow this comment response with a more detailed one at http://maccherone.com/larry/blog.
3. Do you believe the majority of developers understand these abstractions? LM> Definitely not, I’m in the same PhD program and I don’t think I understand them.
I have a more detailed response at: http://maccherone.com/larry/2009/03/30/ports-components-and-connectors-the-next-great-abstraction
I bet you informally think about components and connectors now
Thanks Larry, I enjoy our discussions.
That’s good to hear. It’s not completely obvious they should be treated separately — there are plenty of places where they are tangled up but I think it’s a good aspirational goal to separate them. This means it may be possible to convince folks that architecture activities can be part of agile development, just like refactoring, which does not produce customer-visible product, but is a necessary to repeatably deliver value.
Altogether too often folks incorrectly assume that Architecture = Big Design Up Front (BDUF). I do not agree. I think of architecture as an engineering activity that you can do up front, in between, or later during refactoring. Of course there are things that are easier to decide in advance, like the architectural style of the entire system (e.g., client-server, REST, peer-to-peer, pipe-and-filter), but agile folks already agree with that too. As you say, choosing a framework such as J2EE or .Net can effectively decide your architectural style.
You are right that most developers are thinking in terms of objects and design patterns. Interestingly, though, they weren’t thinking in terms of design patterns a decade ago (roughly) when the GoF book came out. Of course they were applying patterns informally and may have had pet names for them. The GoF book made a convincing argument that we should officially recognize what we’ve been doing already and give the patterns standard names. It also gave us a better handle by which to describe and compare potential solutions. Your short dialog about the developers is a great example.
There are striking parallels here with architecture. Only rookie developers consider functionality exclusively. Almost every project has some kind of internal divisions, be they clients and servers or various modules. We talk about the state of a protocol between two objects (or clients and servers). What’s missing here is that these ideas have not been reified and made explicit. When you say “component” you might be referring to a chunk of code, while I might mean a runtime chunk (e.g., you could have two runtime components as clients but just one compile-time module that has the source for them). Obviously we’d prefer to standardize some terminology. Being able to name architectural styles, which are just patterns at the architectural level, has the same benefits as naming design patterns.
You can solve the same problem thinking 1) just about the objects, or 2) by objects + design patterns, or 3) objects + design patterns + architectural abstractions. If you’ve got 5 objects, choose #1. if you’ve got 500 objects, #2 is still practical. If you’ve got 5000 objects, your brain starts to get full and you have to reach for #3.
We should probably be explicit about the hypothesis. One might be that “Objects are all you need”. Of course that’s easy to poke holes in, because we don’t strictly need objects. We could code in C or assembly. But we find objects a convenient abstraction. We think about objects rather than the machine code that makes them run. That way we free up our brains to solve the problem rather than thinking about registers and moving data between memory and the CPU.
Another possible hypothesis is that “Architectural abstractions like components and connectors can help us to understand large codebases”. We can quibble about what is large and if they are helpful or a nuisance on small codebases. I have done some work on big projects and my experience is that at some point you need to clump together big chunks of code. About 15 years of work has gone into defining and explaining the component and connector abstractions. They’re not perfect but they work better than 5000 objects.