intrust

What can we feasibly know? What do we want to know of “what we can feasibly know”? What can we feasibly do with “what we want to know of what we can feasibly know”? Well for starters, we don’t want to know everything. We don’t want to deal with 1:1 maps of reality. But there is no question that our willingness to accumulate knowledge is influenced by our potential for doing so, including the potential for actually making use of that which we accumulate. In Hollerith’s own words, arguing for increasing the statistical base and the computations carried out:

To know simply the number of single, married, widowed, and divorced persons among our people would be of great value, still it would be of very much greater value to have the same information in combination with age, with sex, with race, with nativity, with occupation, or with various sub-combinations of these data. If the data regarding the relationship of each person to the head of the family were properly compiled, in combination with various other data, a vast amount of valuable information would be obtained. So again, if the number of months unemployed were properly enumerated and compiled with reference to age, to occupation, etc., much information might be obtained of great value to the student of the economic problems affecting our wage-earners.

And in the words of his boss, General Francis A. Walker, Superintendent of the Tenth Census in a letter to Hollerith:

In the census of a country so populous as the United States the work of tabulation might be carried on almost literally without limit, and yet not cease to obtain new facts and combinations of facts of political, social, and economic significance. With such a field before the statistician, it is purely a question of time and money where he shall stop.

The savings in “time and money” delivered by Hollerith’s tabulator were astounding. Rather than taking ten years to add up the results of the census, the automated tabulation took only weeks. It was only natural for Hollerith and Walker to wish to reinvest these savings in more elaborate statistical models. In a modern perspective, with the realization that we could carry out the entire 1990 census tabulation on our home PCs, while taking a coffee break, the 1890 constraints of computation seem very distant, but the collection of W3 is still limited by time and money, even if there are exceptions.

The Ministry for State Security – better known as the Stasi – was the "shield and sword" of East Germany’s state party, the Socialist Unity Party of Germany (SED). [...] At the time just before the Berlin Wall fell in 1989, the feared secret police had 91,000 full-time employees and around 175,000 unofficial informers whose job it was to spy on people in the German Democratic Republic (GDR). This they did to an extent that is barely imaginable and that took on almost grotesque proportions.

"We must know everything," was the mantra that the Minister for State Security, Erich Mielke, never tired of repeating to his employees, who numbered approximately 5.5 people for every 1,000 citizens. And they took him seriously. Over four decades they gathered information about their victims, writing down even the smallest details and accumulating 184,000 metres of written material in the process. Not to mention 986,000 photographic documents, 89,000 films, videos and sound recordings and 17,870 electronic data storage devices – and this is just the material at the Berlin headquarters¹.

Apparently a state harbouring the paranoiac suspicion that any citizen could be a covert agent of subversion would want to keep tabs on, well, everyone, damned the cost. Stasi believed they could feasibly eavesdrop on the GDR’s 16 million inhabitants and seem to have had the resources to do so. Though just how they actually accessed and drew conclusions about this 184,000 meters of written material, plus pictures and sound recordings, etc., is something I know little about, I would assume there were problems. After all, they didn’t have Google.

Most directories are not as complete as Stasi’s, because such completeness is neither practical nor affordable: Decisions must be made about what to include and what to exclude. The design of the Hollerith punchcards was made at a time where the limits of physical space and technological feasibility still played a decisive role in determining the quality and quantity of what was stored on them, and until quite recently this has always been the case. W3 has always been at the mercy of the medium built to hold it and the tabulating technology meant to calculate it.

// posted by Greg FitzPatrick @ 14:30