In engineering, we strive for commoditization of our products, software, etc. We like everything to be orthogonal, configurable and we hate application specific hacks and kludges to our architecturally pure baby. The sales and support people, on the other hand, love it when we come up with “solutions” to make the end customers happy. One person’s kludge is another’s solution. I sometime spin this as the two characters Eeyore and Tigger. Eeyore laments the kludge but Tigger loves the solution.
The commodity product can only command a commodity price. People want more bang for the buck.
A product that offers solutions can command a premium price. The vendor gets more bucks for the bang.
The solutions, the business logic; that is the value proposition. The commodity part of the product is just the platform.
There seems to be two natures to technology. In some cases, it is accessible to startups and inspires innovation. In other cases, you can’t even get your foot in the door without an initial outlay of huge amounts of investment. I refer to the former as “grass roots” and the latter as “feudal”.
When a particular technology is in a grass roots state, people aren’t afraid to try it out, tinker with it to see what they may come up with. Who knows, you may even invent something cool. Most grass roots technology these days is well supported by open source ecosystems.
On the other hand, when a technology is in a feudal state, you have to be well established and of a size sufficient that the initial outlay to invest in it is not prohibitive to your organization. Even so, there usually has to be a strong business objective towards a large enough scale of production to consider it. In some cases, a product runs through pilot stages using grass roots technology, but you have to go the feudal route in order to benefit from economy of scales of higher volume production.
Examples of grass roots technologies accessible to anyone are the PIC and AVR (Microchip) microcontroller technologies, the whole Arduino ecosystem and even the Raspberry Pi ecosystem. Examples of feudal technologies, well just try doing business with Broadcom or Qualcomm. You won’t get them even answering your calls unless you are big enough to perk their interest. Or try getting into automotive electronics and see just how quickly you will be buried in regulatory red tape.
Ironically, we often associate “Skunkworks” with a grass roots mindset but in reality, the name came from experimental work done using very “feudal” aerospace technologies.
Sometimes a feudal ecosystem helps, usually in a highly regulated technology niche, especially where it involves extensive research and development among a handful of high-end players, for example 5G cellular where you probably don’t want household “hackers” all connecting random devices directly to the network. You want them to have to at least use modules that have been certified for the cellular networks.
Sometimes you luck out and you create your own grass roots technology. I have fond memories of a time working on systems using Intel 80188. We had discovered that certain off-the-shelf DOS “C” compilers generated object code that could be made ROM’able using the right tools. We even wrote a few in-house tools to help. We even used C++ back in the days before they came up with an Embedded C++ spec. We used Novell networks before connecting to the Internet, and had diskless network booted PCs (before the days of the Windows registry). But IIRC correctly, this was still after a number of years using the more “feudal” Intellec (big blue) boxes and huge emulators to initially get into it.
So what would you say is the nature of the technology that you work with? Is it something that you can buy an inexpensive kit and work with it when working from home? Or is it something that requires you be in an expensively equipped lab to even do anything?
There have been times when digital logic doesn’t behave logically. In such cases, you must consider that logic blocks are an abstract model implemented using electrical, substrate physics and even electromagnetic physics processes. This is where you run into hardware errors, even chip errata. Below are a few examples I ran into:
“It’s as if when this interrupt occurs, it makes this clock line radiate like an antenna and it gets picked up by this other interrupt pin”. Hardware engineer runs out of the room, checks schematic and discovers that the affected lines were not terminated (left floating). After a patch to the board, worked perfectly.
“It’s like these two C code statements don’t exist after the previous statement to change the CPU clock frequency”. SoC chip errata discovered with a new date code batch of chips. Solution was to put five embedded assembler NOPs after changing the clock (and ensure interrupts are disabled).
“This thing never turns off! … even when I disconnect the power.” Wiring error putting out of spec voltage on an input port. Enough of this leaked through circuits to keep the MCU running.
And I could come up with more. Takeaway here is that sometimes you have to think outside the logic block boxes.
The idea here is that someone signs a blank check, not knowing what the final amount will be. It can come back to sting the person.
Now how is this relevant to a development project? This is applied from the perspective that if someone makes a short sighted design decision that makes sense from the perspective of a single stage in a project’s lifecycle, it can come back later to sting the organization. A short term gain can be at the expense of an inordinate amount of recurring activity later.
For example, a “hack” patch is made to a free open source software (FOSS) package to implement a feature. This is fine until the package is upgraded and the patch doesn’t work. Someone has to do the work again. What happens if no one understands what the patch did? What happens if the patch was buried with the original recipe and hence gets discarded when the new version is imported into the project?
That original design decision led to the signing of a blank check.
Another perhaps more common case is an upstream design process (e.g. mechanical or PCB design) makes decisions that software team suffer from. Or perhaps the software team makes decisions that the test and documentation teams have to live with.
Using custom drivers could be an example here. After each major upgrade, the custom drivers have to be revisited. The same could be said of custom board support hacks to the kernel. These have to be revisited with each kernel upgrade. Someone rushes a project to completion without regard for recurring maintenance updates and system upgrades down the road.
A flip side to this is “one-of” customization activities. We expect a feature to benefit multiple customers and lead to recurring revenue. That way we can amortize it over many sales and many releases. Hopefully the recurring revenue exceeds the costs of recurring activity. But this is NOT so for a one-of customization for a specific sale. You cannot amortize that one-of cost. It has to be entirely covered by that sale. Who else wants to pay for something that only the one customer uses? The project never ends with fulfillment of that sale. That customer won ‘t like it when a standard upgrade results in loss of that customization. If there is a maintenance obligation associated with that sale and customization, then you just signed a blank check.
The picture I use to illustrate this is, imagine a tide pool by the ocean where all the tide pool creatures are fighting for supremacy over the tide pool. But they are oblivious to the fact that the tide is coming in and that there are sharks outside just waiting for a feast.
Nearly 25 years back I remember a CEO of a leading paging infrastructure company saying that he always knew we were capable of beating the competition in battles but never envisioned that we would win the war. The competition was pulling out the of the paging infrastructure business. I remember everyone on the engineering team with our jaws dropped making comments like, “Is he for real?” Shortly after that, paging infrastructure revenues dropped to critical levels in one quarter and the company had to take drastic restructuring measure just to survive. One of BC’s top tier high tech firms was decimated practically overnight, due to marketing myopia at the senior management level.
When swimming in a pool, if you swim along one edge, or along the perpendicular edge, you are always within arm’s reach of the edge. If you swim out away from both edges to the middle of the pool, you have nothing to grab a hold of, hence there is increased risk of drowning.
The idea here is that most hardware designs take a vendor’s reference design and then adapt it with as few changes as possible to their own custom hardware design. The idea here is that you remain close to a known working “edge”. When you customize in multiple areas at the same time, there is increased risk both from the perspective of making a mistake, and from the perspective that you may stumble upon errata, especially when dealing with bleeding edge technology.
We tend to be a bit naive, assuming that everyone knows what they are doing. With bleeding edge technology, we are inventing it. In other words we are making it all up as we go.
In the 1990’s I saw a center fold ad depicting a technical professional buried in his desk papers working late at night. The caption read, “We would like to thank you for all your late nights spent reinventing middle-ware”. When you go to the opposite page, it read “Signed, the competition”.
That stunned me. I realized that any time and effort reinventing stuff that could easily be adapted from readily available sources, could be construed as time spent working for the competition.
This ties into the concept of revenue generating versus cost control. The effort may be a great idea to save time or control costs later, but the gist of the ad was that development should focus on the mandate of development and not get overly distracted with process improvement tools. (The ad was from a middle-ware vendor).
This is not to say that streamlining process is a bad thing. It just needs to be part of a development mandate, not sneaked in.
