No VIPs
The following story might be familiar to you. The specifics are fictional, but in broad strokes it is something an experienced engineer may see during his or her career.
A B2B software product is launched, and perhaps a whole company around it. The founding team is insightful - they’ve identified an opportunity and nailed the minimum viable product. It does the right things for the right kind of user, and keeps the unique value proposition in mind. They gain their first few big customers. There’s a lot of churn in the beginning, they gain and lose customers, but a core group emerges.
By the time the dust settles, the founding team realizes that a handful of customers account for the majority of the business’ recurring monthly revenue. A couple of these customers have realized it too, and start lobbying for their problems and feature requests to be treated as high priority by the engineering team and competing with the product team’s vision for the product.
Over time, special escape hatches for workflows are built for specific customers or a set of customers, fragmenting the expected behaviour of the system as a whole and increasing the cognitive load on everyone responsible for supporting these customers. Feature flag checks, magic constants and various levels of complicated branching logic begin to strangle the codebase. The engineers become demoralized as even small changes require increasing amounts of testing, and errors in production happen more often and less predictably.
⁉️ How did we get here?
A customer support request comes in.
“The description field doesn’t linkify URLs pasted into it! It used to do that but it doesn’t anymore…”
The customer care team looks into it and indeed the behaviour is as the user says, and everyone on the team remembers that it did use to linkify the URLs automatically a while ago. Since then, the page in question was re-written with a nicer look and feel. Perhaps the old functionality was left off the rewrite? A bug ticket is filed and picked up by an engineer for implementation.
The product team (like most product teams in most companies) is overworked and pulled in a thousand different directions, and don’t notice the ticket go through. The engineer pushes his pull request and it is code reviewed and deployed the next day. The reporting user is notified and is delighted. A couple of days go by and nobody hears anything more about it.
Until Monday morning, when an email appears in the CEO’s inbox. It’s the company’s contact at a large, newly-signed customer - and he’s irate. Turns out that their workflow involves copying and pasting back and forth from another data source into the field in question, and they require that links aren’t link-ified or copyable as full URLs, even though this is ideal for every other customer. The product team is told they must undo the fix, since losing this customer would be a huge blow. Besides, it’s such a little bug, is it really worth sticking to our guns on this?
🗣 The customer is always right
The product team, and the company as a whole, have already conceded the key point. The finances of the company in the short term have been elevated above the integrity and cohesiveness of the product in the long term. The kicker is that even the product team can see that right now, this is the right choice. Jobs may have been at stake. Once past this first hiccup, the executive team breathes a sigh of relief. They can see the imbalance in their customer base as the risk it is, but don’t have cost-effective strategies to combat it. Besides, they really need to be focused on growth right now.
The next time such a conflict occurs, there’s no option of changing the behaviour across the board to match the VIP customer’s request. What this customer wants is incompatible with a piece of behaviour relied upon by the rest of the customer base. The team is in a bind, and a meeting is convened to find a solution. Someone pipes up - “I mean… it’s all just code. What if we put in branching logic to check the user’s account ID? We could provide 2 different experiences based on who the customer is….” Everyone loves this idea - except the product and engineering team.
This seems like a terrible idea to them, and for good reason. They now have to maintain 2 different sets of expected behaviour for the same use case. The maintenance, testing and support burden in this part of the app has just doubled. This objection is voiced but overruled - the higher-ups are elated that a solution has been found, and the technical objections seem trivial to them, compared to losing this customer. Besides, it’s ‘just this one time….’ Grumbling but assenting, the team implements the behavioural fork in the code, writes extra tests to ensure it keeps working into the future and makes every effort to document it.
🎡 Slippery Slope
As much as the engineering team had protested, a precedent had been set. The second time such a situation arises - maybe a customer wants a feature totally disabled for them, or are willing to finance one to be built, exclusive for their use - it’s easier for everyone to justify. Pretty soon, the culture of the team has become compromised to the point that nobody bothers trying to push back on these customer-specific hacks anymore.
Code is the most malleable of all substances from which our world is built, which is both a blessing and a curse. The best thing about it is that it’s so easy to change. The worst thing about it is that it’s so easy to change. Tough problems like the above, which are really business and culture problems within the organization, can be ‘solved’ - to the satisfaction of everyone who isn’t directly responsible for the health of the codebase - by taking on more technical debt. In the case of a SaaS business with a modern continuous delivery setup, a change made in the morning may be in the hands of customers by the afternoon.
The temptation is understandable, even to the engineering team. After all, their job is not to produce beautiful and elegant code, but to solve business problems for the customer and turn these solutions into cash for the business. Over a longer timescale however, these decisions make the rest of the team’s job more difficult. Codebase quality metrics suffer, features and bugs take longer to implement, QA becomes more involved, etc.
🌉 The Bridge
To get off this path, the business has to understand the destination and why it must be avoided. The root cause of what presents to the engineers as a code quality issue is in fact a weakness in the organization itself. Ratcheting down the business’ reliance on a small segment of the customer base must be made a priority. It might find other ways of making revenue, or sign more comparably-sized customers to balance things out. Neither of these options are quick or easy, this will be a long-term commitment to take on.
The tactical must be paired with the strategic, updating the business’ core values to include an insistence on simplicity and clarity of the product. The product team must be empowered and trusted to be the final arbiter of what’s in and not in the product, no exceptions. This requires a leadership team that is cohesive and committed to crossing this bridge. Gratitude and respect must be paid to the sales team at this point, as imposing discipline on the company with regards to one-offs in the product will make their job harder in the short term.
The discipline doesn’t need to be total, however. For example, once in a while a large customer is willing to finance a feature that is on the roadmap anyway, and for all customers. They’re paying to bump up the feature in the roadmap. As long as the final decision is still the product team’s, if a convincing argument can be made in favour of the new revenue, it should be considered. In the end, knowing your customer’s problems and knowing why the product is the solution to those problems will light the way.
