Commitment vs. Forecast: A Subtle But Important Change to Scrum (2011)
scrum.orgI still think it ironic that when looking at the actual Agile manifesto at http://agilemanifesto.org/ (which is short and uncomplicated), that the first line is "Individuals and interactions over processes and tools" and these days practicing agile, at least in a big-house corporate environment feels like anything but.
The Agile Manifesto suffers from "the curse of knowledge": The signatories are all very experienced, and deeply understand all of the differences between projects and people and when to be flexible and when to be rigid, and they know that it's important to not specify a method for every situation. But there are many, many people entering the software industry who don't have their years of experience, and who need very clear, basic instructions as a starting point, and hence Scrum steps in to fill the gap. What's unfortunate is that scrum isn't a program for developing software managers from rigid, by-the-book managers into experienced, agile managers: Instead it encourages newbie practices for everyone, forever.
> Instead it encourages newbie practices for everyone, forever.
I feel like YAGNI is similarly abused, I don't know how many times I've said "You know what, while we're at it we should...." only to be overridden with "YAGNI!", only to be proven right 3 months later, only now the cost of refactoring > value of the feature, and saying I told you so isn't being a "team player", so no one ever learns.
I very often wonder if I am the only one that feels this way.
> I very often wonder if I am the only one that feels this way.
You're not. One of the bigger issues I've seen with Scrum is not being able to shoe horn in work that needs to be done because business didn't identify it as a "story".
I believe development is there to serve business and any processes used to facilitate this serving should be defined by development. I don't dictate how a general contractor satisfies my requirements for my home build, I leave that to them. I just expect my requirements to be met at or near budget.
A lot of these software process issues boil down to managements never ending quest to commoditize developers.
There's an analogy I like to use... you're baking bread. Bread is just flour, water, yeast, and salt, plus maybe decoration ingredients. You mix things in the correct proportions, let it rise, bake at the right temp/time, and you get bread. Easy, right? Then someone wants you to put in raisins. "It should be easy! It's just a handful of raisins, I don't see why you're telling me it can't be done!", they shout, five minutes before the oven timer goes off.
Scrum is, at heart, designed precisely to stop the behavior you're demanding - that is, the endless stream of "small" interruptions and constantly shifting priorities. The raisins.
Why are you trying to "shoe horn in work" for this iteration that you weren't aware was even an issue when the iteration planning happened? Is production down? Is it a hair-on-fire emergency that threatens the business? Or is it just "important". FUCK important. If it's so important, put it in the story backlog and have it done in the next iteration.
If it's important enough to disrupt the iteration, it's important enough to cancel the iteration, toss all that iteration's unfinished work onto the backlog, and start over. That's how Scrum is supposed to work, but never does, because someone wants raisins at the last minute and thinks it's not a big deal.
Yes, I have seen businesses die, both ways. The problem is that YAGNI cashes out to "make the right decisions" - it's logically a tautology, but an emotional encouragement to drop features. Dropping is usually the correct decision - but too often catastrophic when it's not. If you invoke YAGNI you have to be careful you aren't "picking up nickles in front of steamrollers" - a great idea until suddenly it's not. Fortunately, frequent iteration gives people a more realistic view of the cost of raisins.
I haven't seen businesses die because they can't wait two weeks for a new feature, and had to have it right now. I have seen businesses die because a development team was so paralyzed by constant interruptions that they were dysfunctional and couldn't get any real work done.
I've seen products fail because of a refusal to support flexibility in core functionality because the extra work couldn't be justified due to YAGNI, and then get totally outclassed by competitive products where they were more agile.
Want to know the correct answer to this agreement? Sometimes YAGNI is good, sometimes it's bad. Anyone who doesn't realize this is not qualified to decide which case a team finds itself in.
If it's a better data structure, then two weeks might set the course, alas. It's happened to me. Fact is, one can't get away from judgement calls, slogans might nudge in one direction or another, but it all remains a judgement call what's just a raisin and what isn't. The fatal problem is bosses up the chain who want to demonstrate they matter and are worth their expense by throwing in superfluous raisins; having a crude slogan (misleading or not) to deter them is fine by me.
A "better data structure" is almost never an emergency. And being unable to adapt or extend a data structure in the future is not agile, and it's not the simplest thing that can possibly work.
Back when I first started building a startup, I thought "Thank Dog I no longer have to deal with stupid compromised software, and can start writing everything right!" By the time I was getting anywhere, I was well into toss-over-wall methodology. I did things that I knew full well were compromised and would hurt me later, because the work needed done, and needed to "be done". It was a real education.
I actually have a lightning talk in mind on this subject, called "Why software sucks", that argues that suckage is the nature of software development, and that "barely works" is the best we can realistically ask for - or even should ask for.
> If it's important enough to disrupt the iteration
Do you realize your argument is based on an assumption that is not necessarily true? If Scrum is as good as advertised, it shouldn't require untruths to justify it.
Be wary of confirmation bias. I think YAGNI is typically a good idea. There's a balance between YAGNI and planning (as the extreme in either case is obviously wrong). You probably understand the system fairly well and have good judgement, but YAGNI definitely saves more than I think it hurts in my experience.
Exactly. I look at it as following the Dreyfus Model. If you don't know what agile delivery looks like or how to approach it, you start with Scrum and follow it religiously. Once you get the feel and the team is accustomed, you can modify it to suit your needs or just wing it and keep the goals in mind.
Unfortunately, what teams do religiously is the process, contrary to the manifesto. So they put a Scrum Master pin on someone, they do Sprints and burndowns. They cargo cult.
I did my first CSM course with Ken Schwaber. I did another recently with someone else. The recent course was all process. Ken's course was mostly "why".
Jeff Sutherland said there are three things that must be true for a team to be a Scrum team:
1. It must be self-governing.
2. The team members must be on the team 100%
3. The team members must stay for the duration.
With the exception of start-ups, I've seen maybe one team which met these criteria. But they all had sprints. And of course, they were all held to their commitment. Scrum is now a tool for management to death march people.
Think of all these metrics that management measures scrum teams on. Can you imagine any of these teams turning around and saying, "We're going to measure how many of your bullshit wishlist items meet the definition of an actionable story when presented in sprint planning, and how many hours are wasted?" Of course not. Management can't abide a team that meets rule 1. "I'm a manager! I must manage! What is my job if my teams are self managing?!"
Scrum, as Ken Schwaber describes it, is a fantastic way to develop product. Unfortunately, if you are in an organization that would truly let you do Scrum, then you probably don't need it. You are already qualified, experienced and empowered. If its a hundred twenty-year-olds being told to "do scrum", it probably ain't Scrum.
I worked on self governing teams without manager, twice. Never again. Relationships were hell with constant power struggle between wanna be micromanagers over who is going to be non existent leader and get to have his vision.
Clear responsibilities, accountability and personal autonomy are so much, so much, better arrangement then self governing team.
Hear hear! Scrum predates Agile. It welcomes mixing in other tools as needed, e.g. XP, Agile, etc. After 15 years of professional software development, I just picked up a scrum master certificate a couple days ago after my employer paid for the class. I'd rather see teams using scrum than just winging it unless they are provably experienced or extraordinarily competent. Individuals can be good at their work but it's often difficult to get everything in sync on a team, or even if the team is super amazing the complexity of getting things done in a corporate environment with lots of moving parts and dependencies is usually aided by using tools such as scrum. Even when a complex corporate environment commits to trying to follow it, it's still a struggle to get the basic elements of scrum adhered to (enough time with product owner, enough time figuring out dependencies, any time at all spent identifying ways the team could do better in the future and reduce friction, impediments, or needless work). Take a look at the Scrum Guide - I'd be satisfied if just the few events in the minimal scrum framework could be fit in, but often not even that happens.
* edit: provably
I worked at a large consultancy in the UK where the platform team were 'agile' in that they had 2 week sprints, but those 2 week sprint goals were fixed for months ahead. If you needed a small API addition or specific bug fix in one of your apps (elearning courses) to launch a product on the platform and you hadn't foreseen this 2 months+ in advance then you were out of luck (unless you had leverage on one of the platform team higher ups of course).
That was my first introduction to uppercase A for Agile.
Ultimately the wider business never changes in terms of it wants predictable delivery and deadline forecasts, but internally they have to then wedge some version of lowercase a for agile into that because the benefits day to day are there so you have this clunky enterprise friendly version that poorly tries to get best of both.
In my experience this kind of Agile inevitably emerges when the desire to be agile is limited to a (small) subset of the organisation.
Absolutely - some people hear, "people before process", think it sounds great, then immediately go looking for a process to implement it and find Scrum.
I think that's largely the fault of the manifesto. It is entirely unclear about what putting "individuals before process" actually means.
One way I've seen it used was as a way to justify organizing meetings as a means to solve an increase in defect regressions rather than allocating extra work on automated tests.
The former was "individuals and interactions" and the latter was tools and processes. I thought it was a reasonable interpretation and also wrong.
On another note, what metrics could be commonly used to judge individual dev productivity, and should they be used?
I'm wondering because I have had places bring up metrics like average story points completed per sprint in performance reviews. Which kind of confused me when numbers for our team seemed to be doing fine, and it seemed like my work was going well based off of previous conversations with management.
In my experience, using story points to judge performance is typically not useful. It is a very simplistic view. It doesn't include the quality of the code. And who knows if the estimation was accurate. And then you can have story point inflation. Also, someone may figure out an easier less complicated solution for a problem, and so that would have a smaller story point, which is better.
I would be curious in hearing how people judge dev productivity as well, it's something that I don't like to just use a number, though there may be some benefit in having that being a small part of the judgement.
Well, nobody makes money by pointing at the manifesto. But a whole business can be built around "certification" and all that.
Maybe people are ultimately imperfect and others are looking to process to smooth over those imperfections.
> That is, while there is value in the items on the right, we value the items on the left more.
Trying to explain this to people who are "certified" is exhausting.
My team moved from scrum to a kanban style process a couple years ago. The benefits were immediate. We no longer have drawn out sprint planning meetings where we discuss requirements of features that we never end up working on in that sprint. We don’t waste time debating complexity of features. Everything is now just ad hoc. When we need more requirements definitions, we pull the necessary members of the team together and discuss it. When we see that there needs to be architectual discussions and high level planning, we do it immediately when we recognize the need for it. The idea that you can try and commit to or even forecast how much can be completed for a period of time is pretty absurd. Just identify the minimum requirements of what needs to be done and do it. Retrospective is still productive. But sprint planning is a waste of time IMO.
Scrum is a sad joke. But corporate loves feeling "safe", with estimates in hand, even if they readily blow out. Usually it ends up with "watergile", a total mess.
You can just time kanban tickets vs estimated size (all the forecasting, tracking, etc, should be done by a manager, without wasting the team's time) combined with pulling from the right hand side and vigorous action towards blockers. Optimise for throughput.
During the best version of this, as I experienced, we spent about 5 minutes in the morning as a team, and occasionally visited the board throughout the day (not as a team, sometimes as a pair, to discuss something and add/move tickets). My manager at the time spent some time at the end of each week by himself collecting cards and doing the tracking to make the higher-ups happy.
If you read the actual Agile manifesto, what you are doing now is more agile than scrum :)
A lot of the artefacts of a process are intended to help teams transition from the old way of doing things and to help teams who have stalled or failed to deliver. Sprints, as far as I understand them, were intended to get the wheels turning and to demonstrate results at regular intervals so the customers for any software development effort could see progress and gain confidence that the team was going to deliver. Once everything is up and running smoothly then it's entirely reasonable to drop all that and just focus on delivering - you don't need a lot of packaging an rituals to do that.
But Scrum is working! Why would we ever change?
I’m familiar with some of the artifacts of a kanban style system - do you have any documentation of what your particular kanban process looks like (that you can share)?
> We no longer have drawn out sprint planning meetings where we discuss requirements of features that we never end up working on in that sprint. We don’t waste time debating complexity of features. Everything is now just ad hoc.
I'm curious about a few things, if you don't mind answering.
When do tickets get broken down into manageable chunks? Who does the breakdown?
When is a rough cut of effort put on a ticket? S/M/L often matters when making priority decisions.
A breakdown step in your development process breaks down work into equally sized chunks, that all count against WIP limits. If you're interested in using Kanban for software development I suggest checking out Eric Brechner's book and youtube videos.
Scrum doesn’t prescribe enough techniques to be successful. Somewhere, maybe in an interview, I heard someone say that they never saw a successful Scrum team save that they were doing half of XP too.
So far I have not found an exception to this. Thing is that means everyone is doing Scrumbut (we do Scrum, but...) Which means you have to relearn or reargue with every single team about the exact same concerns. It’s gotten old.
Scrum does not discourage adding additional tools as needed. It tries to provide a minimal framework and nothing more.
Same here. Just pick the next item from the backlog and do what needs to be done. It's much easier.
> But sprint planning is a waste of time IMO.
Agreed. Don't forget to throw daily standups in their. My current project spends 2.5 person hours per day on the daily standup.
Have any good resources on your team's kanban system?
Do you not have due dates or stakeholders that want predictability?
In a waterfall organization many of those due dates are months out.
Finish and deliver don’t have to be the same thing. Especially when features depend on each other.
A group of unaffiliated contractors slipped kanban into Boeing in part because we never said what we were doing. But every time there was a hiccup we proposed another kanban process. WIP limits took a while but saved a lot of headaches. If we are over the limit and you finish a story you had to help someone else or get a special dispensation for the leads (sometimes we had stories that depended on other teams and the next story was straightforward).
It is by far the biggest silent conspiracy to do good that I ever participated in.
Unfortunately after we got reported into a sustaining team there was some peer to our manager who would not shut up about Scrum. If you’ve done Kanban, Scrum is like when your parents make you hang out with your kid brother’s friends.
I agree, I have the same experience.
We had Ken Schwaber come in to our shop early on in Scrum when we were struggling to get a new product going, and at its basics Scrum made tons of sense. Get a small group of people together to figure out how to make the highest priority items to ship to the customer (each written in a few sentences), then leave that group alone until the sprint was done. The "commitment" was on delivering the value in the few sentences, not matching mockups, specs or some never-ending chain of tasks.
It was a really simple and effective approach, but where it broke down was: there's a lot of people/roles/depts who have no idea how to work incrementally. UX "needs to work ahead", product "needs the whole backlog", Ops "have their own backlog", etc.
Scrum didn't work with everyone hawking over a few engineers - then it just becomes task tracking bullshit.
Now I would suggest to deprecate "sprint". It is not healthy to sprint continuously. Sane software development is much more like a marathon.
I don't know if this is meant to be facetious, but I actually fully agree. "Sprint" is a terrible analogy because in reality it is impossible to be sprinting all the time. I usually just say "iteration".
I like "sprint" because it sounds energetic, but the reality of two week sprints is more like interval training. Sort of like this:
Days 1-3: Walking, talking, and enjoying life.
Days 4-7: Jogging. You climb some minor hills and get a little out of breath, but life is still good. You've got this.
Day 8: Big hill day! You eat an energy gel and start on up. You expect to be exhausted at the top, but you'll have two more days to relax a bit afterward.
Day 9: Dark night of the soul. You ran all night but got lost in the night. You're way off course, further down the hill, and the entire hillside is on fire. It's all you can do to just find the course again.
Day 10: Intent on staying on course, you slowly slog on up. You get burned a bit from the raging fires around you. Life sucks. The job sucks.
Day 10+1: You aren't quite sure how or when you agreed to this, but your Saturday is shot as you try to climb up the final stretch.
Day 10+2: You're there! Everyone else is there and you're all exhausted, but proud of your accomplishment. Sure your Sunday was shot, but it's a one time thing. You tell yourself you'll never over commit again.
....
Days 1-3: Walking, talking, and enjoying life....
I'm confused, doesn't your statement mean the analogy is good? You can neither "sprint" at work continuously nor constantly sprint in reality.
Let's say you do 2 week sprints as part of your process. You do a sprint, finish it, and then do another sprint immediately after. How is that viable? Effectively it means that you never stop sprinting. I don't think it's sustainable.
Agreed. I wonder if any manager would ever put in "walks", or "cool downs" into the schedule. Meaning allocate a week (or some time span) in between each sprint to do all those things you wish you had time to do in the sprint.
I know a "walk" is yet just another physical analogy for a software production process (which the analogies don't always work), but maybe it is more of a cultural mindset to value longterm team health without it becoming just a "slacking" week?
Is that actually what scrum suggests sprints are? Thanks, I missed that, that's silly if so. I assumed by the name that this was like a one week a month kind of thing.
I think there's quite a lot of getting hung up on the word itself here... According to the Scrum guide, a sprint is just:
> a time-box of one month or less during which a "Done", useable, and potentially releasable product Increment is created. Sprints have consistent durations throughout a development effort. A new Sprint starts immediately after the conclusion of the previous Sprint.
(http://www.scrumguides.org/scrum-guide.html#events-sprint)
There's nothing that says you need to "sprint" through your sprints, quite the contrary it's supposed to be a way of measuring your team's steady-state output by making the feedback loop short.
If people are really hearing the word "sprint" and thinking "ah, Scum is telling me to work at 110% all the time without stopping", then I put forth that no methodology or change of terminology is going to save them from themselves. However, I suspect there's something about the timebox structure that makes short-term thinking the default unless discipline is applied, and discipline is hard.
Exactly. The point of time boxing is that you have to stop and reflect on the time spent. It gives you the chance to switch tasks if priorities have changed or to split the current task.
Or to go on to the next sprint with the same task. But here lies the problem in many cases. The sprint is taken not as a time box but as a deadline. A sprint should meen: "You can work 2 weeks, 5 days a week, 8 hours a day on this. Then you stop to think."
Running a marathon is not the healthiest thing to do either. Even "marching" has a bad connotation in this context.
People get necks and backs broken in real scrums too. A real scrum is a violent fight for control between opposing teams.
A lot of these analogies were not well thought through. I will wager the methodology merchants behind scrum were neither runners nor rugby players ever.
Ah yes, the shenanigans that occur in a real scrum are no joke. Some of our forwards would wrap tape around their entire head (including ears) to prevent their ears from being torn off.
We used to play rugby when I was at school, it was banned when a kid in the next town ended up in a wheelchair, then my school switched to tennis or something. Nowadays I only watch, the atmosphere at Twickenham is incredible
Funny to see this. I thought getting commitment was one of the soft mechanisms of scrum. An annoying one, because power dynamics are always at play. Still a mechanism, though.
I think I'm supportive of the idea on removing it. Seems the goal is ultimately to find ways in rhetoric and action to align the teams in working to the end goal. Which, often, might require tradeoffs to reach a timely delivery. And timing is a requirement.
It probably made more sense to put it in in the beginning when the process was still being sold to senior managers. Now that scrum is much more embedded, taking out for the reasons given makes more sense.
Yes yes yes. It never made sense to me that our ticket sizing was supposed to be an estimate and yet the planning that used those estimates was considered a commitment. Totally insane.
I have always thought commitment in this context as "we commit this much time for this task, then we stop and reevaluate" instead of a deadline.
I am a big fan of scrum precisely because of the word commitment (I hand't realized it was removed). Forecast means we need to use a larger estimate next time. Commitment means: "how can we still get this live". In a good culture the answer to this should rarely be work extra hours or reduce code quality - it should be reduce scope of functionality. Without the commitment you often get micro feature creep, where a bunch of non-essential micro features are added. A bit like this (https://lawsofux.com/parkinsons-law.html). It is also the obvious moment in time to tell a product owner "we need to remove functionality x or not ship". Without the commitment that important conversation is never had. In my experience micro & macro feature creep is one of the biggest issues in sw development. Obviously you can still do the above with the word forecast, but forecast empowers you less to actually change anything (apart from better aka higher estimates next Sprint). On a general note: I feel scrum is often misused to "manage" engineering teams, when it is really a productivity tool that the team should use for itself.
Depending on the size or the organization, that word commitment can create numerous issues.
It can keep teams from collaborating because of a need to finish their commitments first, effectively blocking the other teams and delaying actual delivery of the product. You can see the same issues in customer support requests. The second that somebody decides to measure completed sprint commitments, you are in deeeeeep trouble because it forces people to low ball to hit that number. It removes your ability to pivot.
The entire concept of sprint commitments in a moderately sized team is borderline destructive.
This change is the best news about software development I’ve read in YEARS.
I wrote an article on the negative psychological impact of commitments in sprints (https://www.linkedin.com/pulse/scrum-makes-you-dumb-daniel-j...). It's great to see renewed focus on shifting away from a term that has been used to make developers' working lives unpleasant, leads to lower-quality code, and gives false certainty to stakeholders.
This PDF is pretty nice. Regarding the differences of Kanban vs Scrum. https://www.crisp.se/file-uploads/Kanban-vs-Scrum.pdf
Is Agile & Scrum still relevant? I am seeing more and more consultants who used to selling this stuff have moved upmarket into Lean and Digital Transformation of companies.
Can we deprecate scrum in 2018?
Please don't post unsubstantive comments here. I'm sure few people here have any fondness for software processes, especially in the corporate decadence stage, but that's no reason to make HN worse.
What does our velocity say?
This is perfect.
Zed Shaw is a treasure. I'm not sure he's all that much of a role model.
Well, he's good at heart and knows his stuff.
Like all of us, he's just human
I don't know anything about the guy but his analysis of agile consultants and agile development agencies is spot on.
Oh yes, this rings true with me.
I once worked at a company where the powers that be decided to go all Scrum. They hired a PM to convert all project, inc ongoing ones, to Scrum. It turned an enjoyable job in to a miserable one.
I suppose it's as good a programming religion as any other.
do people really pair program in real life? I've never actually seen it
I find it useful for knowledge transfer. It's a good way of bringing new hires up to speed for the first week / two weeks. If you leave them to their own devices they have a tendency to get overwhelmed and fearful of continually bothering you to ask questions.
It's sometimes a more efficient way of building stuff where, for a particular task, you need a whole bunch of knowledge or information that's currently living in somebody else's head.
Also, if I'm having one of those days when I feel sluggish having another person there kickstarts me a bit...
Defaulting to pair programming for every task - especially when the conditions above are not met is a super inefficient and overbearing way of developing though.
Yes. Pair programming as the default for me since 2004. I've done it full time for 8 years at an XP shop (Scrum was added later) w/ 3 teams of 20+ total on 3 different continents. Worked for an Agile consultancy and trained people on Pair Programming and TDD for 4 years. I'm now working for an org that does both Pairing AND mob programming (https://en.wikipedia.org/wiki/Mob_programming). It takes time, patience, and humility to learn how to do well. I find teams that pair simply build better software that has less bugs. There's literally a book on it to consider (https://www.amazon.com/Pair-Programming-Illuminated-Laurie-W...).
Just to counter the other responses here. Yes, and it's even more dreadful than you might imagine if you've never done it. I recommend running fast and far away if you ever find yourself interviewing on a team where pairing is the default assumption.
I have done it and it works great if you respect the work and each other. It's a nightmare with people who don't care.
Yes, and I highly recommend it!
source: used to think pair programming was odd until I gave it a proper go.
Yes.
at which company did you see it, and was it everyone
just saying motherfucker a few times does not magically create a solid argument
I don't know. Samuel L. Jackson seems to do alright even when he doesn't have Tarantino writing for him.
Tag [2011] is missing...
Thanks! Updated.