Too Big to Know

Rethinking Knowledge Now That the Facts Aren’t the Facts, Experts Are Everywhere, and the Smartest Person in the Room Is the Room

David Weinberger
Rating: 7.6

“If anyone knows anything about the web, where it’s been and where it’s going, it’s David Weinberger. . . . Too Big To Know is an optimistic, if not somewhat cautionary tale, of the information explosion.”
-Steven Rosenbaum, Forbes

The Network of Knowledge

Until the arrival of the digital age, most people accepted a standard system of knowledge. Students studied subjects, earned credentials to prove their proficiency and became experts. Then they put their knowledge to work conducting research and writing books and articles to share their findings and conclusions. Other specialists vetted their work and deemed it accurate or mistaken. As new discoveries gained acceptance, they joined a body of established knowledge that provided a foundation for further exploration and learning. Throughout history, this system relied on paper to communicate, until the introduction of digital media changed everything. The mere abundance of online information makes it difficult to discern what is true and what is false.

“Knowledge is a treasure, knowing is the distinctively human activity, and our system of knowledge is the basis for the hope that we might all one day come to agreement and live in peace.”

As digital media test beliefs about knowledge, institutions such as universities, libraries and science labs must rethink knowledge’s infrastructure. Questioning long-held assumptions raises fears. These fears manifest as warnings about the Internet, such as “Google is making us stupid,” or “the web gives every nut a platform.” Even as people voice these fears, science moves forward at an unprecedented pace and people tap into previously unavailable resources. Today unlikely thinkers offer approaches to problems that previously only a privileged few had ever tackled, much less solved. The digital age is evolving the “shape and nature” of knowledge into something new and networked. Now “the smartest person in the room is the room itself: the network that joins the people and ideas in the room, and connects to those outside of it…Knowledge is becoming inextricable from – literally unthinkable without – the network that enables it.”

It’s Not Information Overload; It’s Filter Failure

In 1988, organizational theorist Russell Ackoff reported on the need to filter data in order to offer usable information. He described a pyramid that captures how material is filtered and presented. The pyramid’s base is data. The next layer is information, then knowledge, followed by understanding and, at the tip, wisdom.

“As science has gotten too big to know, we’ve adopted different ideas about what it means to know at all.”

Before the Internet, editors were the initial filters of published material. The next filtering came from those who decided which published works to include in libraries, on magazine racks and in bookstores. “Old knowledge institutions like newspapers, encyclopedias and textbooks” decided what information reached public view. Some ideas never made it to the page since the world of print didn’t have the space or paper to house every idea. Most people saw only the wheat; the discarded chaff disappeared.

“Transform the medium by which we develop, preserve and communicate knowledge, and we transform knowledge.”

But now digital media are unsifted and unconstrained. The filters between people and available knowledge are either “algorithmic or social” or a combination of the two. Algorithmic techniques rely on computer processing to sort and manipulate data. Social tools and online networks serve as guides for making choices. All the information ever digitized in any form – worthy or unnecessary, correct or false – remains accessible online. “Filters no longer filter out. They filter forward, bringing their results to the front. What doesn’t make it through a filter is still visible and available in the background.” Modern filters even provide links to further layers of unfiltered information. Because the web offers limitless content, traditional filtering techniques no longer work. Thus, lots of idiotic or just wrong material emerges. And somewhere on the web, someone is willing to contradict any and every idea, regardless of its acceptance or scientific merit.

Traditional Expertise

Even as the structures of knowledge shift radically, people still have access to the information that allows them to function. In fact, it is easier to retrieve than ever: Bus schedules tell you what time to be at a stop. Two plus two still equals four, and Albany remains the capital of New York. People continue to rely on experts and academics to provide opinions and teach lessons. However, knowledge that experts once vetted and committed to paper is giving way to knowledge that thrives in networks that facilitate the development of ideas by groups rather than individuals. Traditional expertise, now under siege, shared several characteristics:

• “Expertise was topic-based” – For books or articles, authors focused on a single subject, used a “single voice” and tried to be consistent throughout their work.
• “Expertise’s value was the certainty of its conclusions” – Once a book went to print, it was difficult to revise or change the opinions expressed in its pages.
• “Expertise was one-way” – Experts spoke through their books to the public, and the public had no way to respond.
• “Experts were a special class” – Since only a select few could publish books, having work published gave authors credibility.

Networked Expertise

The Internet provides a vehicle for networked expertise – that is, for new possibilities from large groups of unrelated people. The idea that people are smarter in groups isn’t new; it’s why think tanks, research centers and universities exist. Virtual crowds are far larger than physical ones. “Crowdsourcing,” a term Jeff Howe coined in a Wired magazine article, uses work by many people across a large geographical area to answer a question, gather data or tackle a problem.

“Information is to data what wine is to a vineyard: the delicious extract and distillate.”

Crowds on the Internet contain people with different backgrounds, viewpoints, skills and experience. This diversity enables a new type of expertise. Internet crowds are not made up only of individuals; people congregate online in subnetworks that share interests and experiences, and combine their expertise to address problems.

“Our new medium so unifies information, communication and sociality that it’s nearly impossible to keep the strands of that triple helix apart.”

The Internet retains every entry, every post, which generates a form of networked know-how. Expertise accrues in real time online. For example, when a technology company introduces a new operating system, its users encounter bugs, they report them, and the network works out the kinks. The Internet functions as a hub medium, allowing all kinds of interactions. It “scales indefinitely,” working for one person, two people or an infinite number of users.

“There are no isolated ideas, and there never were; there are only webs of ideas.”

Crowds of diverse people do not guarantee the best results. In his book, The Difference, author Scott Page concludes that the best diversity occurs when individuals in a group hold different perspectives, with varied views and experiences, and use different approaches for tackling problems. He recommends finding the appropriate amount of diversity, using moderators to keep groups on task and “forking” discussions into separate channels when they stray off topic.

Echo Chambers

One concern is that web collectivism leads to the creation of echo chambers, online groups that form around shared ideas. Since conflicting opinions do not penetrate its walls, the chamber strengthens and reinforces existing beliefs, no matter how misguided. People’s views become more narrow and extreme when echo chambers buttress them. Of course, the idea that people arrive at truth and knowledge via reasonable discourse among those holding different viewpoints isn’t just contemporary. In fact, it dates back at least to Socrates. With the Internet, which could act as a rational public forum, the cacophony of voices makes it difficult to agree on anything.

“Knowledge now lives in the messy web that has grown around it, the way life lives not in our neurons, bones, blood and marrow but in their connection.”

Central postmodernist ideas are finding a firm foothold in the digital age, including this pivotal understanding: “All knowledge and experience is an interpretation, [which is] social…Interpretation always occurs within a culture, a language, a history and a human project…There are always many ways to interpret anything…Interpretations occur in discourses. You can’t make sense of something outside of a context.”

Beginning, Middle and End

Long-form argument was the accepted format for human reasoning for centuries. It worked wonderfully in a paper-dependent medium because the layout of books “enables and encourages” this structured process. Unlike long-form argument, networked knowledge is nonlinear.

“The Internet is what you get when everyone is a curator and everything is linked.”

Jay Rosen’s blog, PressThink.org, provides an appropriate example. He writes blog posts in long-form argument style, but his work differs from the paper-based knowledge process in that the length of the argument is unlimited, readers post comments that influence the subject matter, public discussion of his work affects its progression, and Rosen can write about an idea before his thoughts reach completion. As he writes, Rosen’s work influences other people. He’s neither a lone voice nor a lone expert. The shape of the networked knowledge on Rosen’s blog is not pyramidal. In fact, it has no shape. In a digital world, issues, ideas and knowledge intertwine in nebulous webs of links, discussions and differences of opinion.

Scientific Knowledge

The scientific method calls for testing hypotheses by conducting controlled, repeatable experiments. However, today’s abundance of data overwhelms the structure of the scientific method. In weather research alone, meteorologists receive climate data from satellites, ocean buoys, human observers and various instruments. How do humans wrap their minds around topics that have grown “too big to know”? Digital media is changing the properties of scientific knowledge in the following six ways:

1. Scientific knowledge is “too big for theories” – The Human Genome Project aims to map the “genetic blueprint” of humans. It now has more than 250 billion pieces of genetic data in its databanks. Such data “galaxies” are complex beyond the understanding of the human brain. Eureka-style scientific discoveries may be a thing of the past. Instead, computer models processing oceans of data will predict future behavior. Computer simulations show how things work even when people may not completely understand why they work.
2. It is “flatter” – Although amateurs have always contributed to scientific knowledge, now the line blurs between accredited professionals and amateur scientists. For example, a physicist in Italy – working alone – accessed Google Maps and found a crater in Sudan, an astonishing discovery, since Earth has only 175 known craters.
3. It is “continuously public” – Before digital media, researchers published mostly positive results and discarded negative outcomes. Now many researchers post “open-notebook” records so other scientists can learn from failed experiments.
4. It flows through “open filters” – Scientific journals publish very few papers, and rejection rates are quite high. Open access sites, such as PLoS One, publish 90% of submissions. Old-style commercial publishers no longer have the final say on disseminating scientific knowledge.
5. It offers “science with a difference” – Digital media take away the absolute authority scientists used to hold. Today, the Internet vets the imperfections in experts’ findings.
6. It is “hyperlinked” – Networked science connects knowledge to its sources. This means that no conclusion is ever final since the network continues what individual scientists begin.

Leadership Networked

The evolving nature of knowledge affects leadership because access to so much information makes decisiveness difficult. Jack Welch, former CEO of General Electric, represents the traditional top-down management structure. He consulted executives, examined data, checked his gut and decided. Jimmy Wales, the co-founder of Wikipedia, follows a different strategy. He relies on the principles of Wikipedia and input from its vast network to guide his choices.

“The last word is now never the last word.”

Online collaborative projects such as Wikipedia and Linux influence the traditional hierarchies of organizations. The best responses to a large-scale crisis – for example, Hurricane Katrina – emerge from “hypernetworks” that depend on “distributed decision making.” In some circumstances, relying on local expertise works best. In others, including more people in the process means more solutions and more invested participants. As business environments become increasingly complex and the amount of information grows steadily, a network contains more knowledge than a single leader can manage. Therefore, a network makes the best choices.