My dad is an electrical engineer. When I was in school, I remember asking him, what’s the difference between an “electrician” and an “electrical engineer”? After 30 years of working to build a successful software projects business with marquee enterprise clients, I still worry about this question. Are the developers in the software services industry engineers or technicians?
What’s the difference?
The key difference is one of conceptual understanding, as far as I can see. An electrician will know how to wire up a motor and two switches but will not know much about anything below the surface. That’s what I see with the programmers in the industry. They can write code, but cannot explain how or why their own code does what it does. They use MongoDB for two years, build applications to store data in it, and never learn how the non-SQL paradigm differs from relational algebra, or how ACID properties work with MongoDB.
The second difference I see between technicians and engineers is the ability to design something new. An electrician does not design a fan – he just wires it up, makes sure its body is earthed, and replaces a failed starter capacitor. Engineers design the fans. But this is where the software world deviates. Software programmers are designing software all the time. Every 50-line function is a work of engineering design. Can a technician design an engineering artifact?
The answer is: yes, but only in the simplest cases. This is where experience can act as a stand-in for understanding. Over a few years of programming experience, a young programmer gathers a large mental database of code patterns. “First, my code will validate the input parameters. Then, my code will compute the derived values. Then, check if the data matches the records in the database. Finally, update the database. Then generate a response and exit.” Once such a pattern is identified for the task at hand by mapping with previous tasks, she can churn out code. Ask her to sort an array, a very simple problem. She’ll be stuck, frantically doing Google searches for code fragments she can copy, because she’s always called a sort() function to sort arrays, never thought about how the code actually sorts an array.
So, in the world of custom software development, each programmer is performing a small design task every day, with poor conceptual clarity and very limited knowledge of underlying principles. Young programmers in 99% of these jobs have never thought about programming paradigms, or data structures, or complexity of algorithms, or anything related to hardware, architecture, operating systems or networking.
Why does this happen? This is because the vast majority of professional programmers today come from academic programmes (even those with with titles like “Bachelor of Engg in Computer Technology”) where they have explicitly learned to be technicians – they have learned the syntax of coding, not learned computer science, not even the semantics of their code. They are the software equivalent of electricians – they can wire up the lights and fans in a room, but they don’t know anything about electricity generation or storage.
There are degrees and then there are degrees
There is a vast difference between Bachelor of Engg degrees in Computer Engg or Computer Technology or Information Technology from Tier 3 colleges and a BTech degree in CS&E from one of the leading IITs. A close examination of the curricula highlights this. A BTech CS&E student of IIT Bombay is taught just one course called “Programming Languages”, and is introduced to a bunch of programming paradigms at a conceptual level, with one programming language to illustrate each paradigm. More than half of these programming languages are rarely used in the industry, like Prolog, Lisp, or Haskell. This BTech student is then expected to teach himself these various programming languages by reading books and trying small pieces of code, and is expected to write small algorithms in these languages and submit them as assignments, on which he is then graded.
An engineering graduate from a lesser engineering institute travels a very different path. He is taught subject after subject where he is taught various programming languages and taught how to write code in these languages. He is put through a course in “C/C++”. Then he’s put through a course in “Java”. And so on. It is assumed that these students can’t teach themselves these languages, and there is no attempt made to focus on the underlying paradigms, e.g. logic programming, functional programming, etc. They don’t even understand more basic concepts like scope and extent of an identifier. Therefore, the lesser engineering institutes, in the guise of delivering a bachelor’s degree in computer science, is bringing up these students to be little more than coders. The focus of these engineering institutes is to get their graduates job-ready for the industry. The focus of a BTech in CS&E from an IIT is to build a base for a Master’s or PhD programme at a leading university.
The IITs are not unique; all Tier 1 technology institutes in India and elsewhere build engineers with conceptual clarity.
How does industrial software development work?
To be more precise, how does the software services industry deliver software? This is where the industry has tried to take lemons and make lemonade. They have learned from the manufacturing sector, where blue-collar workers assemble complex cars and television sets. In the world of software, processes, tools, and separation of roles have been applied to make things work in an assembly-line fashion. There is a software designer, who is separate from the developer, because the average developer cannot design to save his life. There is a developer who writes code but does not test it well – the tester is brought in for that. (In my younger days, I took it for granted that my code is not complete until it is thoroughly tested. By me.) And when the pieces of the system are being put together, special build engineers try to make the system integration happen; developers do not know how to build their own code outside their own laptops. When it comes to arranging the components in the tech stack, there are architects. And none of these roles know how to set things up on an actual server and operate the pieces in a production environment – we bring in the devops guys and SRE for that part. And even the devops guys have little or no idea about, say, server hardware, load balancers, or network switches. In essence, when depth of understanding is absent, bring in a small army of specialised technicians where three smart and well-educated engineers should have sufficed, and hope that they can hold the show together.
“Data processing” versus “technology”
I was chatting with a dear friend who is now very senior in one of the Big Four software services giants of India. He was referring to tech salaries in the US, and I interjected, saying that the salaries which TCS, Cognizant, or Infosys pays to their engineers in the US are nowhere near what he was citing. He cut me off, saying “Oh, these companies where we work are not in the tech sector. These are what I can at best call the data processing sector. The tech sector is the sector where new technology is being created, like the product companies, or the cloud hosting giants, or the SaaS giants.” I agree. A senior developer with a decade of experience posted in the US with one of the software services giants will earn perhaps half of what a developer in Google, Amazon, or Microsoft earns after a year. The market has its own intelligence; it knows the truth of what I’m seeing.
Do we really need this conceptual clarity?
Do we really need conceptual clarity for developing software? These millions of programmers neither want to be academics nor researchers — they just want to write code. The banking industry (well, the retail banking part, at least) works on an army of tellers. Do tellers understand the principles of banking? They don’t. Why do we ask for conceptual clarity from programmers then?
This question is usually posed by those who do not understand the difference between normal physical manufacturing and software development. In software design and development, each (genuine) developer is, in effect, a mini-researcher. He is thinking about patterns and correlations and architectures every day, at a micro-level, with his code. It’s insightful to see the education levels among programmers in the large tech companies:
Out of these PhDs, a small percentage are doing cutting-edge research, but I have a lot of friends with PhDs who are individual contributors in Silicon Valley companies. They actually design and write code. The point here is: good software design and development needs both good education and high intelligence. The software services industry tries to make do, but please let’s not claim that technicians are adequate.
Bank tellers, who are technicians, are not expected to wade through a jungle of new transaction types being developed every day. A bank teller is not asked to design new transaction types even once a year. Banking does not require development, only operations. Ditto, manufacturing (other than the small design and R&D teams). Software is different. In software, everyone is doing micro-design and micro-R&D.
Aren’t business applications easy to do?
This is the next question that is often thrown up. The hypothesis is that those fancy PhD developers in Google and Microsoft must be writing far more difficult software. How difficult is it to write an accounting or an HR application?
To be honest, some parts of business application software can be routine, specially if you are making tiny incremental changes to existing code. But even such applications need to be designed carefully to deliver reliability, scalability and maintainability over the 10-20 years of their lives. One of our clients gave Remiges a contract to re-write a business application which they had developed more than a decade ago from another vendor and were using heavily. Why not just extend and maintain the old one? Partial answer: it was designed with a short-term perspective and it cannot be maintained any more; it’s almost falling apart at the seams. Plus: our client has seen a 10x increase in transaction volume in the last four years, and expects to see another 10x increase in the next four years. Even wizard programmers with PhDs will not be able to maintain and extend that old and leaky application — a complete re-design and re-write was called for. If the original application had been designed by good designers, perhaps 80% of the need for the re-write would have not arisen today. The three keywords which are sacrificed over and over at the altar of technician-based software development: reliability, scalability and maintainability.
Are there ways out?
Yes, I can think of three. One: productise the custom code to the extent possible, deliver as much of the application code as possible in the form of reusable products and frameworks. Two: use GenAI, not necessarily to generate code for the applications, but a lot of peripheral code like automated test cases, scaffolding code, code reviews and generating throwaway code. Test automation can improve the quality of code delivered by mediocre programmers by many multiples, and GenAI now allows us to add automated tests at low cost. Three: build depth in specific platforms and technologies, and use this specialisation to improve the technical expertise of your team. In other words, don’t build an HR product from scratch, use an open source product as a base and build the final system by adding custom layers on it.
I’ll expand on these three ideas in another post one day, and talk about our experience with items 1 and 2 in Remiges. I am sure there will be other strategies for risk mitigation.
I have been known to use the terms “semi-programmers” and “real programmers”. Twenty-five years ago, I used to refer to PowerBuilder developers as semi-programmers. In the dystopian 21st century, I now find that 99% of the software services industry worldwide has become staffed with semi-programmers. Software development is hard. We need to figure out how to do a better job as an industry as a whole.
Leave a Reply