Policies and Standards Compliance

So far I've covered what software engineers can do to increase the software quality within their code and infra-structure and what engineering managers (or group coordinators) can do to assist engineers to produce and maintain good coding practices to ease staff relocation and infra-structure migrations, but without global policies and good practices being enforced throughout the whole institute (or company) it can dissipate easily with time and all the effort will be lost.

Producing good quality code, stable infra-structures and excellent results is a lot of hard work. To keep this infra-structure running still needs a lot of hard work but with just a few quick hacks you can render it useless and make it impossible to take it back to the quality state without a complete re-write.

Heroes can organize chaos, write thousands of pages of documentation, diagrams, flowcharts and so on, keeping the programs and procedures as simple as they can be but if others are not encouraged to follow the same lines it'll all go away when the hero is gone. Worse, the remaining team will half go back to how it was done before the hero (and breaking all the flow which in turn probably won't work any more) and half will change the programs to something less legible and logical, filling it with dirty hacks.

There must be a culture of software quality institute-wise, enforced by the high administration and verified by specialists (which in most cases can be the software engineers themselves). Quite seldom the high administration understands about software quality or even software engineering, all they want is the results as fast as you can produce, no matter how. This is the fundamental point of software quality and this is why there is so few institutes (or companies) that can say they have good software.

Software quality costs a LOT of time and effort. Most programmers don't like documenting or writing tests, most coordinators won't demand quality when requesting a new feature and most scientific team leaders or high administration staff won't even know what's the difference and why bother.

I hope to have shown you how important software quality is and why it's fundamental in the long run, why its inherent additional cost is well worth paying and why they have to be enforced from top to bottom, on all areas that involves software or infra-structure. If you haven't understood yet please read the Part 1 and 2 and if again you're not convinced please talk to software engineers in your team. You'll find those that defend quality and those that prefer spending the time adding more features, then ask other software engineers to compare their code.

In bioinformatics there are lots of standards, much more than it is healthy. Every team on every institute has it's own unique identifier, standard algorithms, interfaces and file formats and some teams have more than one. When every team has the liberty of creating whatever they please without any conformity to any established standard or even discussing with the community before implementing anything that's the scenario you'd expect.

This scenario gets worse when some of those internal standards actually become an international standard and still remain upon the control of a very small set of people. It becomes impossible for the community to change it or discuss about it and improve beyond the local team's point of view, making it a standard by dictatorship and not by democracy.

Standards must be defined, created and maintained by the community and not a single team or institute. It must hold the interests of the majority of teams and projects that will actually use the data, it must progress with time to better technologies and not stale forever in the same old inefficient infra-structure. The community must control it so the enforcement of those standards becomes a good thing for all of them.

Standards must be accessed, enforced and receive feedback world-wide by means of consortia between institutes. Unique identifiers must be unique and must identify something, file formats and service definitions must be global and common to all teams, libraries and algorithms must be shared and maintained by the teams using it. Standards are created to be used and changed globally.

The hierarchical structure of research institutes have, for obvious reasons, a lot of scientific coordinators, managers and leaders, those who know a lot about what is being researched and know what to expect from the scientific point of view. In bioinformatics institutes they'll know how good the result data is and how they should interact with other data but they seldom know how good the software that makes this interaction is or could be.

It's also common to see teams of system administrators to deal with the desktop and servers, network and storage. Database is also important with DBAs and their managers but it's not easy to find a software engineering manager, with the responsibility of defining and controlling the quality of software institute-wise.

Teams can define their own quality control but if there is no central enforcement of the commonly implemented methods each team will go in a different direction and even with quality they still won't communicate with each other and again create their own idiosyncrasies and unique identifiers and formats. This is not a task to the scientific, database or systems teams, as they will rely on different opinions from different competent software engineers about quality and still you won't have an unified strategy. You need an experienced team of software engineers, solely focused on software quality and business continuity plan, to help others on defining the best practices, gathering what's done already and refine the methodology to enforce software quality that can actually be implemented within the institute.

It's not, though, a matter of listing the practices and enforcing it top-down without any knowledge of the structure or how those standards and methodologies will interact with the current software. This team must understand the internal structures, and idiosyncrasies and adapt the good practices to the institute's reality, only then you'll have software engineers programming in unison.

It is also the role of this team to act as a barrier when the other teams demand timelines or features that will conflict with software quality. There is a common scenario in software development called user dictatorship, when there is no restrictions of what the user can ask and the software engineers must implement at all costs. This is a common anti-pattern for internal teams in non-technological companies (such as banks, industrial, electricity) where the board doesn't have the faintest idea of what is software quality, they just want the job done.

The user must understand that there is a cost for everything and that re-training the user to better use the current architecture is most of the time much cheaper than refactoring the whole infra-structure to have one new button on the interface. Experienced software engineers focused on software quality can calculate those costs and provide an accurate report on all outcomes.

Therefore, this team has two roles: first, to gather internal software development methodologies, study software quality measures and have the power to enforce it to the teams. Because those teams were the ones choosing the rules, it should be trivial and much more like guidelines instead of brute-force. Second, the team must assist teams with requirements, planning development and interacting with the users (other teams or scientific board) in order to protect software quality and business continuity.