Enterprise Software, Now

A previous post outlined a fictional, but highly desirable, scenario where a new executive was able to quickly orient herself* to her very complicated job merely by use of an (also fictional) ideal enterprise software application.

In a sense, a modern desktop or laptop computer, loaded with appropriate software could now play that role. This is how most people in an organization view business computers.

Close examination challenges that view.

For example, productivity software, in other words office suites, with word processors, spreadsheets, diagram makers, presentation software, databases, etc. are everywhere in business. Despite recent moves to make the user interfaces more intuitive, all the popular programs require frequent googling to get non-trivial things done, and perhaps three of the ten programs in the suite actually see frequent use.

The very large collection of websites dedicated to “How-To’s” covering the exact same software tells us something important about their usability. Such software annoyances are commonplace and waste precious time.

The state of large enterprise software provides even more challenges. Organizations face one of three implementation choices: buy commercial off-the-shelf (COTS) software, outsource/near-source the application, or build the application in-house.

COTS software is expensive to purchase, is usually written for a specific knowledge domain (e.g. accounting), has high initial configuration costs and long configuration times, is notoriously unfriendly, and ages rapidly, forcing costly migrations to new software every few years.

Outsourcing (especially off-shore) has a well-deserved poor reputation, and near-sourcing is expensive.

In-house software production can have large relevance, usability and maintenance advantages, if well designed and thought out. Designing, coding, testing and implementing the application requires expensive expertise, can take years, and is fraught with hazards. Retaining the talent needed to build and maintain such large systems can be formidable.

There are niche exceptions to the above; more and less intuitive software. What should be clear is that currently, it takes the average person years to learn enough software expertise to be marketable.

Apprenticeship is unpopular in the USA. The above, combined concerns with software create skills shortages that limit business, slows the creation of wealth and inhibits progress. It also creates a new division of haves and have-nots: those who understand software naturally or by applied discipline, and everybody else.

Software runs banks, our cars, airplanes, airlines, most large businesses, most government agencies (developed world), our phones, our lives. There is no escape. The demand for software skills increases faster than people can be trained, and the skills shortages continue.

Why is the software world like this? Why can’t business software work like browsers, Facebook, early Macintosh computers and iPhones? One clue is the current dominance of time-to-market. The race for ever-faster methods of developing software has taken a big toll. Many ordinary users and industry experts feel software quality has been locked in a downward spiral for decades.

Much more important is the usability of the software, which is concentrated in the design phase of development. To do it right, the designer actually has to know the job she is automating, must have a feel for it, and must know how the user will actually use the software. In our CEO example, this task is large, and, in general, usability is a pervasive and persistent issue.

Notes: *Both feminine and masculine gender pronouns are used in this blog and always include the other gender.

The RDBMS Players

RDBMS stands for relational database management system, and means that:

1. Data is organized into tables, most likely collections of tables
2. There are very specific rules governing tables, organized into normal forms.
3. The tables can be connected to each other in parent-child relationships, or simply used for reference

RDBMS are highly organized and take considerable effort and expertise to create and maintain, thus they are expensive.

The big players in the proprietary RDBMS field today are:

• Oracle, by the Oracle Corporation, currently the market leader
• DB2, by IBM, similar in structure and use to Oracle
• SQL Server, by Microsoft, child of Sybase, the up-and-coming database of choice

The big players in the open-source RDBMS field today are:

• MySQL (owned by Oracle) and MariaDB (open-source fork of MySQL created after the original developers of MySQL sold it to Oracle)
• PostgreSQL, a serious, up-and-coming challenger to Oracle

All of the above have millions of web pages devoted to them, so no links provided.  Which should you learn?  All of them, starting probably with MS SQL Server.

The No SQL Players

These very fast databases, all different, do not organize along the “counting entities” (rows in a table) model.  Instead, they store documents of various sorts in collections of loosely related documents, in turn stored in databases with loosely related collections.

The five top players (all open-source except DynamoDB):

1. MongoDB
2. Cassandra
3. HBase
4. DynamoDB (Amazon product offered in their AWS)
   
5. CouchBase

Why No SQL?  Actually, storage and retrieval of data is done by object-oriented methods that replace SQL, so that isn’t really the point.  The real point is that these databases store and retrieve massive amounts of unstructured data.  The developer can literally make up a new JSON or document type on the spot, in minutes, and not have to spend hours carefully making a highly structured RDBMS (and the surly DBA watching over it) happy.

This is the fast and sloppy way to do business, and it works just fine for what it does.  Good luck applying structure to such a thing after it grows large!  MDM probably won’t play nice with NoSQL.

Master Data Management

Master Data Management or MDM is a strategic scheme to formally organize the data for an organization (company, non-profit, government) in a predictable, transparent and uniform manner, so that there is a single, common point of reference for that organization’s data.  Using some sort of master data plan allows easy collection and curation of mere data into information easily, and organizes information into knowledge. Thus we get two promotions in the hierarchy of know-how, provided the system is designed correctly in the first place.

Everyone knows the basic organization of a typical western kitchen; refrigerator and freezer, stove, oven, sink, counter-tops, drawers, cabinets.  Even if you are searching for a glass in someone else’s kitchen, you won’t have to search for long.  Thus, kitchens (usually) follow a fairly common master design, with predictable storage of typical items.

MDM schemes do the same for an organization’s data.  There is a box, closet, cubbyhole, bottle, etc. for all the data in a predictable and fairly obvious place, so that workers developing and maintaining the information system know where to find what they need fast, and can create applications to share this data with the non-technical people in the organization.  Everyone sees the same data, stored in a manner that makes sense technically, but presented to each user in a manner that best suits them.

MDM can help with such things as stopping an automated e-mail fundraising system from sending solicitations to donors who have already contributed.  In a sense, MDM allow the organization to “know what it already knows”.

Large Scale Automation

Imagine for a moment that you were suddenly placed in charge of a huge company, with tens or hundreds of thousands of people working there. Your first job is to find out, as quickly as possible, what your job means: what the company does, how does it do it, who does what, when, etc. Big, worrisome, ongoing task.

On the desk in front of you is your computer, with a big screen. Someone told you, during the hiring discussions, that everything you need to know is contained in a single program in that computer. At the time, you didn’t believe it, but now is the time to find out. The company has a very aggressive IT program and a reputation for efficiency.

You log in by touching the screen. The computer has already opened up the single program running on it; there is no desktop, no program icons, minimal controls. You can log in, log out, lock the computer, shut it down, start it up, nothing else. The simplicity is almost suspicious.

The screen in front of you is so simple, you can take it all in with your first glance. You see a mere six items displayed, with clear graphics: Business, Company, Communications, Daily, Climate and Customize. There is no help button, and none is needed.

You touch Business. The screen changes to a new one showing the lines of business, customers and suppliers for each. Touching one business line brings up a new screen with a brief, high level overview, and various touchable objects to drill down to view increasingly detailed information. In minutes, you grasp the business line basics. You repeat for each line of business. In less than an hour, you know to a satisfying level of  detail, what the company does.

You touch Company next. There you see an organization chart, news of relevant recent events inside and outside the company, physical sites for the company’s operations, history of the company, operations, financials, taxes. In less than an hour of browsing, you know where to go to find out what you need to know about the company, quickly. No need to ask anyone anything.

In turn, you do the same with each of the top-level items. Communications is much more than e-mail; Daily organizes the events and decisions of the day very clearly and succinctly; Climate shows market information, competition, important and relevant news, trends, legal and government information, and detailed forecasting. Customize allows you to easily and intuitively design and build your own screens to suit your way of thinking and doing business.

Before lunch time, you are already oriented, and ready to begin some actual work. You go to lunch impressed and far calmer. As soon as you return, you will go right to the Daily screen and dig into the productive work of the day.

The scenario described above doesn’t happen in the real world; typical companies use many different computer systems and programs, few, if any, really integrated to make high level work easy and fast. Meetings with various people often fill the days, and are often less than productive.  Instead of half a day to get oriented and “find out where everything lives”, a new chief executive may take months to get the same grasp of the company, working long hours, enduring endless meetings, talking, listening, thinking, asking questions.

The job is not rocket science; routine information gathering, organization and presentation shouldn’t be a struggle, but always seems to appear so. Creating such a system to run a huge company can be done, even if incrementally; however, someone has to conceive of it, design it, build it and then maintain it. All of this is a lot of work.

Every operation or concept described above can be automated. At every level, from the Board of Directors to Senior Management down to individual contributors, machines can be built to take care of routine tasks, freeing people up to do the things only people can do: conceive, create, think, decide, buy, sell.

Large scale automation, as an ongoing mindset and practice, brings an organization ever closer to the idealized story told above. In future articles, the topic will be examined in detail.

Up the mountain path

Welcome to Back End!

Here we will consider all things about software the user never sees: the databases that holds all that data; the programming that moves the data back and forth, cleanses it, stores it, retrieves it, changes it; and the over-all manner in which massive amounts of data are governed in order to use tons of data effectively.

We start with data (tons of it), hope to turn data into information, information into knowledge, knowledge into wisdom (good use of knowledge).

Data that is relevant, coherent and useful can become information.

Information that is organized, curated, and descriptive of reality can become knowledge, which we can use to change our world.

Knowledge applied in an optimum, humane and moral manner becomes wisdom.

And that, in a nutshell, is what we are all about.  Cheers!

Good company in a journey makes the way seem shorter. — Izaak Walton