Notes

Why Do People Pirate Software?
The inaccessible nature of the deep Web may conjure images of hackers doing nefarious things, but that's not really accurate.

© aetb/iStock/Thinkstock

What a tangled web we weave, indeed. About 40 percent of the world's population uses the Web for news, entertainment, communication and myriad other purposes [source: Internet World Stats]. Yet even as more and more people log on, they are actually finding less of the data that's stored online. That's because only a sliver of what we know as the World Wide Web is easily accessible.

The so-called surface Web, which all of us use routinely, consists of data that search engines can find and then offer up in response to your queries. But in the same way that only the tip of an iceberg is visible to observers, a traditional search engine sees only a small amount of the information that's available -- a measly 0.03 percent [source: OEDB]. Well, a lot of it's buried in what's called the deep Web. The deep Web (also known as the undernet, invisible Web and hidden Web, among other monikers) consists of data that you won't locate with a simple Google search.

No one really knows how big the deep Web really is, but it's hundreds (or perhaps even thousands) of times bigger that the surface Web. This data isn't necessarily hidden on purpose. It's just hard for current search engine technology to find and make sense of it.

There's a flip side of the deep Web that's a lot murkier -- and, sometimes, darker -- which is why it's also known as the dark web. In the dark Web, users really do intentionally bury data. Often, these parts of the Web are accessible only if you use special browser software that helps to peel away the onion-like layers of the dark Web.

This software maintains the privacy of both the source and the destination of data and the people who access it. For political dissidents and criminals alike, this kind of anonymity shows the immense power of the dark Web, enabling transfers of information, goods and services, legally or illegally, to the chagrin of the powers-that-be all over the world.

Just as a search engine is simply scratching the surface of the Web, we're only getting started. Keep reading to find out how tangled our Web really becomes. Today's Web has more than 555 million registered domains. Each of those domains can have dozens, hundreds or even thousands of sub-pages, many of which aren't cataloged, and thus fall into the category of deep Web.

Although nobody really knows for sure, the deep Web may be 400 to 500 times bigger that the surface Web [source: BrightPlanet]. And both the surface and deep Web grow bigger and bigger every day. You can read all about it with How Internet Search Engines Work, but we'll give you a quick rundown here.

Search engines generally create an index of data by finding information that's stored on Web sites and other online resources. This process means using automated spiders or crawlers, which locate domains and then follow hyperlinks to other domains, like an arachnid following the silky tendrils of a web, in a sense creating a sprawling map of the Web.

This index or map is your key to finding specific data that's relevant to your needs. Each time you enter a keyword search, results appear almost instantly thanks to that index. Without it, the search engine would literally have to start searching billions of pages from scratch every time someone wanted information, a process that would be both unwieldy and exasperating.

But search engines can't see data stored to the deep Web. There are data incompatibilities and technical hurdles that complicate indexing efforts. There are private Web sites that require login passwords before you can access the contents. Crawlers can't penetrate data that requires keyword searches on a single, specific Web site. There are timed-access sites that no longer allow public views once a certain time limit has passed.

All of those challenges, and a whole lot of others, make data much harder for search engines to find and index. Keep reading to see more about what separates the surface and deep Web.

©Jan Will/iStock/Thinkstock

As we've already noted, there are millions upon millions of sub-pages strewn throughout millions of domains. There are internal pages with no external links, such as internal.howstuffworks.com, which are used for site maintenance purposes. There are unpublished or unlisted blog posts, picture galleries, file directories, and untold amounts of content that search engines just can't see.

Here's just one example. There are many independent newspaper Web sites online, and sometimes, search engines index a few of the articles on those sites. That's particularly true for major news stories that receive a lot of media attention. A quick Google search will undoubtedly unveil many dozens of articles on, for example, World Cup soccer teams. This is especially true as a news story ages. The older the story, the more likely it's stored only on the newspaper's archive, which isn't visible on the surface Web. Subsequently, that story may not appear readily in search engines -- so it counts as part of the deep Web.

Deep Potential

If we can unlock the deep Web to search professional databases and difficult-to-access deep information, fields such as medicine would immediately benefit.

©shironosov/iStock/Thinkstock

Data in the Deep Web is hard for search engines to see, but unseen doesn't equal unimportant. As you can see just from our newspaper example, there's immense value in the information tucked away in the deep Web.

The deep Web is an endless repository for a mind-reeling amount of information. There are engineering databases, financial information of all kinds, medical papers, pictures, illustrations ... the list goes on, basically, forever. For search engines to increase their usefulness, their programmers must figure out how to dive into the deep Web and bring data to the surface. Somehow they must not only find valid information, but they must find a way to present it without overwhelming the end users.

As with all things business, the search engines are dealing with weightier concerns than whether you and I are able to find the best apple crisp recipe in the world. They want to help corporate powers find and use the deep Web in novel and valuable ways.

For example, construction engineers could potentially search research papers at multiple universities in order to find the latest and greatest in bridge-building materials. Doctors could swiftly locate the latest research on a specific disease.

The potential is unlimited. The technical challenges are daunting. That's the draw of the deep Web. Yet there's a murkier side to the deep Web, too -- one that's troubling to a lot of people for a lot reasons. extreme mining On the dark Web, where people purposely hide information, they'd prefer it if you left the lights off.

The dark Web is a bit like the Web's id. It's private. It's anonymous. It's powerful. It unleashes human nature in all its forms, both good and bad. You can find illegal goods and activities of all kinds through the dark Web. That includes illicit drugs, child pornography, stolen credit card numbers, human trafficking, weapons, exotic animals, copyrighted media and anything else you can think of. Theoretically, you could even, say, hire a hit man to kill someone you don't like.

But you won't find this information with a Google search. These kinds of Web sites require you to use special software, such as The Onion Router, more commonly known as Tor.

Tor is software that installs into your browser and sets up the specific connections you need to access dark Web sites. Critically, Tor is an encrypted technology that helps people maintain anonymity online. It does this in part by routing connections through servers around the world, making them much harder to track.

Tor also lets people access so-called hidden services -- underground Web sites for which the dark Web is notorious. Instead of seeing domains that end in .com or .org, these hidden sites end in .onion. On the next page we'll peel back the layers of some of those onions.S. authorities shut down Silk after the alleged owner of the site Ross William Ulbricht was arrested.

© David Colbran/Demotix/Corbis

The most infamous of these onion sites was the now-defunct Silk Road, an online marketplace where users could buy drugs, guns and all sorts of other illegal items. The FBI eventually captured Ross Ulbricht, who operated Silk Road, but copycat sites like Black Market Reloaded are still readily available.

Oddly enough, Tor is the result of research done by the U.S. Naval Research Laboratory, which created Tor for political dissidents and whistleblowers, allowing them to communicate without fear of reprisal.

That leaves U.S. law enforcement in the ironic position of attempting to track criminals who are using government-sponsored software to hide their trails. Tor, it would seem, is a double-edged sword.

Anonymity is part and parcel on the dark Web, but you may wonder how any money-related transactions can happen when sellers and buyers can't identify each other. That's where Bitcoin comes in.

If you haven't heard of Bitcoin, it's basically an encrypted digital currency. You can read all about it on How Bitcoin Works. Like regular cash, Bitcoin is good for transactions of all kinds, and notably, it also allows for anonymity; no one can trace a purchase, illegal or otherwise.

Bitcoin may be the currency of the future -- a decentralized and unregulated type of money free of the reins of any one government. But because Bitcoin isn't backed by any government, its value fluctuates, often wildly. It's anything but a safe place to store your life savings. But when paired properly with Tor, it's perhaps the closest thing to a foolproof way to buy and sell on the Web.

© audioundwerbung/iStock/Thinkstock

The dark Web has its ominous overtones. But not everything on the dark side is bad. There are all sorts of services that don't necessarily run afoul of the law.

The dark Web is home to alternate search engines, e-mail services, file storage, file sharing, social media, chat sites, news outlets and whistleblowing sites, as well as sites that provide a safer meeting ground for political dissidents and anyone else who may find themselves on the fringes of society. Dark Web search engines may not offer up personalized search results, but they don't track your online behavior or offer up an endless stream of advertisements, either. Bitcoin may not be entirely stable, but it offers privacy, which is something your credit card company most certainly does not.

For citizens living in countries with violent or oppressive leaders, the dark Web offers a more secure way to communicate with like-minded individuals. Unlike Facebook or Twitter, which are easy for determined authorities to monitor, the dark Web provides deeper cover and a degree of safety for those who would badmouth or plot to undermine politicians or corporate overlords.

A paper written by researchers at the University of Luxembourg attempted to rank the most commonly accessed materials on the dark Web. What they found was that although sites trading in illegal activities and adult content are very popular, so too are those concerned with human rights and freedom of information [Source: ArXiv].

So although the dark Web definitely has its ugly side, it has great potential, too. Its store of human knowledge and trivialities grows more massive every day, complicating our efforts to make sense of it all. In the end, that's perhaps the biggest challenge behind the Internet that we've created.

Programmers will continue to improve search engine algorithms, making them better at delving into deeper layers of the Web. In doing so, they'll help researchers and businesses connect and cross-reference information in ways that were never possible before. What you really want it to do is find the most relevant information. Otherwise, you're left awash in a sea of cluttered data that leaves you wishing you had never clicked on that search button.

That's the problem of so-called big data. Big data is the name for sets of data that are so large that they become unmanageable and incoherent. Because the Internet is growing so quickly, our whole world is overrun with data, and it's hard for anyone to make sense of it all -- even all of those powerful, all-knowing computers at Bing and Google headquarters.

As the Internet grows, every large company spends more and more money on data management and analysis, both to keep their own organizations functioning and also to obtain competitive advantages over others. Mining and organizing the deep Web is a vital part of those strategies. Those companies that learn to leverage this data for their own uses will survive and perhaps change the world with new technologies. Those that rely only on the surface Web won't be able to compete.

In the meantime, the deep Web will continue to perplex and fascinate everyone who uses the Internet. It contains an enthralling amount of knowledge that could help us evolve technologically and as a species when connected to other bits of information. And of course, its darker side will always be lurking, too, just as it always does in human nature. The deep Web speaks to the fathomless, scattered potential of not only the Internet, but the human race, too. But while researching this story, it was easy to conclude at least one thing for sure -- most news headlines tend to sensationalize the dark Web and its seedier side, and rarely mention the untapped potential of the deep Web. Articles about illegal drugs and weapons obviously draw more readers than those detailing the technical challenges of harvesting data from the deep Web. Read the negative, breathless articles with a grain of salt. It's worth remembering that there's a whole lot more to the deep Web than the obvious criminal element. As engineers find better, faster ways to catalog the Web's stores of data, the Internet as a whole could transform our society in amazing ways.

How Google Works

Why Do People Pirate Software?

How the MPAA Works

How BitTorrent Works

Bagot, Martin. "Deep Web Criminal Marketplace Completely Disappears After '58 Million Heist'". The Mirror. Dec. 3, 2013. (Dec. 6, 2013) http://www.mirror.co.uk/news/technology-science/technology/sheep-marketplace-deep-web-criminal-2879995

Bergman, Michael K. "White Paper: The Deep Web: Surfacing Hidden Value." Journal of Electronic Publishing." Aug. 2001. (Dec. 6, 2013) http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104

Berman, Fran. "Deep Web Film Will go Inside the World of Bitcoin, Silk Road and Beyond." Mashable. Nov. 20, 2013. (Dec. 6, 2013) http://mashable.com/2013/11/20/alex-winter-deep-web-documentary/

Biggs, John. "Deep Web Users are Ready to Launch Silk Road 2.0." Techcrunch. Oct. 4, 2013. (Dec. 6, 2013) http://techcrunch.com/2013/10/04/deep-web-users-are-ready-to-launch-silk-road-2-0/

Bingham, John and Kirkup, James. "David Cameron Wins FBI Support for 'Dark Web' War on Paedophiles." Telegraph. Nov. 18, 2013. (Dec. 6, 2013) http://www.telegraph.co.uk/news/uknews/crime/10456108/David-Cameron-wins-FBI-support-for-dark-web-war-on-paedophiles.html

BrightPlanet. "Understanding the Deep Web in 10 Minutes." Mar. 12, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/03/whitepaper-understanding-the-deep-web-in-10-minutes/

BrightPlanet. "How Does Data from the Deep Web go from Results to Actionable Intelligence?" Jan. 31, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/01/how-does-data-from-the-deep-web-go-from-results-to-actionable-intelligence/

BrightPlanet. "The Future of Online Search." Aug. 27, 2013. (Dec. 6, 2013) http://www.brightplanet.com/2013/08/the-future-of-online-search/

Couts, Andrew. "TorSearch Makes Finding the Next Silk Road a Lot Easier." Digital Trends. Oct. 11, 2013. (Dec. 6, 2013) http://www.digitaltrends.com/web/torsearch-tor-network-hidden-services/

Couts, Andrew. "That Was Fast: 2 Sites Vie for Silk Road's Criminal Customers." Digital Trends. Oct. 3, 2013. (Dec. 6, 2013) http://www.digitaltrends.com/web/silk-road-fbi-ulbricht-sheep-marketplace-bmr/

Gallagher, Diane. "Peeling Back the Layers of the 'Dark Web' WCNC." Nov. 19, 2013. (Dec. 6, 2013) http://www.wcnc.com/news/local/The-Dark-Web-232606871.html

Here and Now. "The Deep Web: Where Google Won't Take You." WBUR.org. Nov. 8, 2013. (Dec. 6, 2013) http://hereandnow.wbur.org/2013/11/08/the-deep-web

Hockerson, Lauren. "TorSearch Offers Private Search of the Deep Web." Gigaom. Oct. 11, 2013. (Dec. 6, 2013) http://gigaom.com/2013/10/11/torsearch-offers-private-search-of-the-deep-web/

Lederman, Abe and Lederman, Sol. "Understanding Deep Web Technologies." New Idea Engineering. Jun. 2004. (Dec. 6, 2013) http://deepwebtech.com/PDFs/Understanding%20Deep%20Web%20Technologies.pdf

Love, Dylan. "There's a Secret Internet for Drug Dealers, Assassins and Pedophiles." Business Insider. Mar. 6, 2013. (Dec. 6, 2013) http://www.businessinsider.com/tor-silk-road-deep-web-2013-3?op=1

Seymour, Andrew. "Anonymous 'Deep Web' is New Frontier of Child Exploitation, Conference Told." Ottawa Citizen. Nov. 16, 2013. (Dec. 6, 2013) http://www.ottawacitizen.com/business/Anonymous+deep+frontier+child+exploitation+conference+told/9175718/story.html

Swift, Tim. "What is the 'Deep Web'? And Other Questions About the Shadowy Virtual World of Silk Road." Baltimore Sun. Oct. 3, 2013. (Dec. 6, 2013) http://articles.baltimoresun.com/2013-10-03/business/bal-silk-road-deep-web-explainer-20131003_1_satoshi-nakamoto-bitcoin-silk-road

University of California, Berkeley. "Invisible or Deep Web: What It Is, How to Find It, and Its Inherent Ambiguity." (Dec. 6, 2013) http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html

Read More: https://extrememining.org/