The Deep Web (otherwise known as DarkNet, Undernet or the Invisible Web), has a simple definition:
“A website that is not available through searching for it on Google and the like”
After having some quite heated conversations over the last couple of weeks (with some quasi-technically savvy people I should add), I’ve put this list together to hopefully clarify some points, I appreciate it’ll take away quite a lot of the mystery of the whole thing, but here we go…
1 – The deep web is a portal to information that others don’t want you to have
Status – Partially True
Yes, it’s true, the “Deep Web” exists, and people store on it information they don’t want you (as Joe Public) to have. But not for the reasons you probably think.
I’ve worked with some pretty large organisations in my time, one smaller one had 7.1Tb of office documents, images, and the odd movie on their intranet. This intranet also had a 9.6Gb database full of other information.
Now this system mentioned above was accessible via the internet, and had some pretty heavy security features (for pretty obvious reason). However, since it was relating to the companies operational factors (and not full of alien autopsies) it would make pretty boring reading to even the most ardent of conspiracy theory buff. The only way you could get on is if the company in question gave you a log on.
Deep web yes, interesting probably no. But still fits into the definition.
2 – Hiding Information from Search Engines Is Rare & Indicates Some Deep Web
Status – False
Most sites hide some information from search engines. Gyp the Cat dot Com hides information from search engines. It’s a very common thing to do. There is a file on web servers called robots.txt, and one called sitemap.xml.
These files tell the search engines what they can and can’t look at on the site. Telling a search engine not to look at something doesn’t indicate it’s exciting, it usually indicates the exact opposite. When a webmaster excludes a directory it’s usually full of techie stuff (ie programming) that would actually damage a sites ranking on a search engine. It’s not so much being evil, it’s more about being a responsible net-citizen.
So there can be certain parts of sites that are available via a search engine, and other (none interesting stuff) that isn’t. Good practice.
Deep web yes, interesting probably no. But still fits into the definition.
3 – Websites make you confirm you’re a human being must be hiding something
Status – False
This theory goes that some websites make you prove you’re human before they let you on, that happens, it’s true (ie Captchas, query boxes, simple arithmetic etc). However they’re not necessarily hiding something.
People seem to loose their common sense on this one. Search boxes on websites, how is a search engine supposed to know what to type in, any information gained from a search on a website is probably “Deep Web”. However, many Webmasters use a search engines own code to let them have access to all the data. Plus many sites allow you to access the results by clicking any way. So most the data will be accessible.
Now, lets take the example from the 1st Myth I posted above. To get to some of the 9.6Gb stored on the database you had to search for it. You had to search like a human would. It’s not hidden, it’s there available (to employees).
Now, the occasional site really doesn’t want automatic search bot or crawler bot on their site at all. Discussion forums are one example, if what is being discussed is changing so quickly that a web crawler taking a snap shot of the information would be an unfair gauge of whats on there.
4 – Freenet, Tor and others of the ilk are deep web
Status – Partially True
Freenet appears to be a content distribution platform built around peer-less VPN technology. Tor appears to be a content distribution platform built around multiple peers with an inbuilt proxy service.
In looking at this entry of the list we need to look at the content inside the delivery platforms. Search engines don’t have the mechanism to look inside these encrypted sessions (although I dare say though governments or those without enough resource could possibly look at it), but again that doesn’t necessarily mean it’s of any interest. Tor was invented by the US Navy for secure communications.
Many corporate bodies use similar systems, and see Issue 1 about that. It’s called site to site VPN, and usually uses IPSec. I’ve worked in the UK where all my internet traffic was going out via the USA. This is the exact same set up as above. And again Issue 1 is that it’s the fact that it’s sensible information, but again no alien autopsies here.
So again, Google isn’t interested because there is nothing of interest.
5 – The size of the Deep Web is immense!
Status – True (but not for the reasons people think)
Going back to our definition on Issue 1, if we look at the size of the website for the same company. I dare say it was no bigger than 10mb. Verses 7.2TB on their internal information store. This is not a rare occurrence. Companies do not make their private information publicly available.
The hidden web is bigger, but there’s nothing exciting here for Joe Public. Any platform that is accessible via the internet seems to fit into the definition of the Deep Web. So that includes intranet sites, the like of Google Docs and other such document collaboration tools. Google are aware that the so called Deep Web exists, and they sell appliances to business to search internally.
Don’t forget also some information cannot be searched easily. YouTube for instance contains TBs of information, yet searching on it is only as good (usually) as the people describing their own videos. Books and libraries online are sometimes only accesible to those with permission, yet they are not available via Google and you usually can only logon if you a member of that institution (for instance Colleges and Universities).
10 White Hat SEO Tips for IT Professionals
care home tameside
cms
cms checker
desktop wallpaper
search dns records
visio tool for mac
vps hosting
wordpress themes
You’re an idiot. I’ve been in deep networks all over the place and I have read and seen shit you couldn’t even fathom. The surface web is quite small compared to the amount of information and gold nuggets you can get in the deep networks. Penetration is necessary to get this information and I have done this many times. I was able to boost my pure intelligence greatly with gold nuggets scattered about these deep networks, I have mathematical algorithms memorized which enable me to mentally calculate above the level of a savant that was born with the gift, this information is not given to the public and many of the governments keep it secret/esoteric. Knowledge can lead to intelligence and vice versa. The deep networks have changed my life and opened my eyes to understand things that other people do not understand and could never understand unless they have this gold too. I don’t share my gold with anyone and I don’t care to. I have brought this proof and this argument to you because I know you’re wrong, but I will not show my cards to you because I don’t really care. For those of you reading this — get there to the deep networks and find this shit yourselves and you will be amazed. Never stop looking for knowledge and truth. All information should be free; knowledge belongs to the world yadda yadda. I would share some of this fruit with you all, but this fruit is that much sweeter when you find it yourself.
Hi there Anon, thanks for dropping by and thanks for taking the time to write your comment.
A couple of points I’d like to make, firstly I agree with you entirely in that the deepweb is huge, it is easily magnitudes of scale larger than the surface web. However not all of this magnitude is anything worth looking at, namely corporate systems and online academic libraries.
The next point I’d like to make is that I’ve heard the argument before that only the enlightened or those who search for something will gain enlightenment from it. This has been repeated over the centuries by many quasi philosophical and religious bodies. In saying that knowledge should be free yet not providing any help or assistance to find it you seem to be contradicting your own stance somewhat.
Again thank you for taking the time to write your comment, it’s always good to hear peoples views on things I find interesting
Gyp
so much bs for “I don’t care”
Uhhm yeah,
I agree with anonymous. I run from marketing of all types and I realize that there is just as much marketing in the deep web, but it’s immense size allows me to go further away from it. It reminds me of what the “net ” used to be. I would venture to say that the 10% web is for the more narrow minded human and the other 90% for the broader, more adventurous minded human. This is just a hunch, but I bet you giggle at a geico commercial.
Great read.
the deep web does have some disturbing stuff, but only a small portion of it is. So it is rarely seen.
Thanks for dropping by.
I agree, but I dare say you can find disturbing content on the surface web too. I’d also be interested in ratios, since the “deep web” is so much bigger than the hidden web I’d be curious if the ratio is similar across both webs
I stumbled on an ebook on Amazon that focuses on the secrecy and security side of the Deep Web – accessing Tor and adding on all sorts of elements to keep you anonymous. It’s not a guide to the Deep Web as such but its incredibly useful and surprising that you really can travel the surface web anonymously as well as access hidden networks. Its as much about protecting yourself from the corporate big boys as it is about hiding from governments, etc. I don’t think I’m allowed to plug books here (am I?) by if you type Deep Web into Amazon there really aren’t that many books to choose from.
Good article indeed, i have been on deep web it’s been 1 year and i have seen some things. I haven’t had access to this called “big amout of information” yet and i can’t tell if it is true or not, hope it is. I don’t completly agree with what i am about to paste here as i think we should first warn our kids and let them choose what they want to learn and experience…they are born free spirits. My computer doesn’t have any content restriction software and i have never seen my little sister who is 10 years old now searching for inappropriate content over the internet. Maybe because she knows some things are not worth watching…it seems that they don’t wan’t to negatively impact their life somehow, they are really smart
“Data itself isn’t good nor bad, data just represents the surrounding reality. The more data we may access, the more accurate model we may create of the reality, thereby also define our actions in ways that are maximally beneficial to our aims. The net must not be restricted by moral, ethical nor legal issues prevailing in different world locations, data must flow freely.”
Author? It don’t know…people on deep web don’t seem to care about this
Thanks for commenting
And I’m glad you’re exploring!
your an idiot the deep web is in fact a great place to buy illegal drugs and hire assassins. even nuclear weapons. so u and ur tech savvy freind can lick my nutsack. oh and there is a difference between deepweb and the DARK NTERNET
Hi, thanks for dropping by.
I’ve got little doubt you could do all the things you mentioned on the “surface” web too. My whole argument is that people seem to loose common sense when talking about websites not indexed by web browsers. There is very large number of those sites which class as Deepweb, but they are not in the least part interesting.
The deep web is a load of crap. The surface web and deep web are synonymous. A lot of children claim they find deeply powerful information on the deep side. I can find sources just as easily, and more of them on the surface. Internet lore, for the kids to fathom about. In fact, I’ve met people who create things in deep web just to help the story out. They may claim to have weapons but they in fact have nothing. Furthermore, they’ve been contacted by the government about what the pages they created. There is no such thing as anonymous, and you’d be shocked what the government can see.
Hi,
Thanks for taking the time to write an excellent comment! I see your logic entirely and agree.
I think as the previous commenter points out there is a difference between the academic use of the term “deep web”, and the term “dark internet” which has been constructed and probably along the lines you mention.
Thanks again,
Gyp
I can’t accept stuff from someone who can’t spell ‘LOSE’
Ahh, well spotted I’ve always struggled with that one!
I agree that it’s a lot bigger than surface web is. However, most of it, is information that is not interesting for the majority of the surfers (like you said, intranet stuff, corp. stuff that wouldn’t change the MAJORITY of people’s lifes).
Although, industry spying gives you lots of money. Terrorists can learn how to be a terrorist. How to use, create and fire a gun. A bomb. A drug. If government can track you that much how come they don’t arrest HUGE ********** sites, forums and people? *********** forums with actually brutal *********** scenes? You can find out sites with “dad’s” archive from their daughters beach trip, exposing on a *********** subject. If they can trace your IP even with encrypting and dynamic proxys, and still don’t arrest people…man, world’s lost. My momma once said to me that governments needs evil and crimes, althought they need control over it. And from what I understand, Deep Web gives countries’ government no control at all.
Ideology opression is what we’ve been suffering of from the last years. And yes, if you make people think, they will think, and they will turn against opression, even if it’s not physical (which makes revolution processes faster). Now, if you assume that Deep Web really makes you anonymous (as I believe), truth and knowledge that “they” don’t want you to have, could be there yes, sir.
That was always my point of view. I hope you read it and answer it.
Sorry for my english mistakes if I made some.
Thanks for taking the time to write an excellent and constructive comment. Your English is good
I dare say that everything you’ve mentioned could be found on the surface web, if you knew where to look or how to look.
From this article and peoples comments I’ve got to thinking about the deep web more, and I suppose it could be argued that Tor has content that is aimed towards specific demographics and isn’t censored by search engines or webmasters. I can imagine some of this content could be gleaned from the surface web, but on the likes of Tor it’s given prominence hence it finds it’s audience easier. As you mention with oppression it’s the lack of censorship that makes Tor interesting from this perspective.
There are sites on the internet that help you look at Surface Web sites you may not come across before, try Stumble Upon as an example. Now I’m not saying that it is deep web, all I’m saying is it’s a way to explore an interest without search engine interference. Sometimes Stumble Upon some fascinating reading and I learn a lot using it. More I dare say than the couple of times I’ve been on Tor and Freenet.
Thanks again,
Gyp
I have been in the deep web. Mostly it is full of sick people, i call them sick only because normal individuals don’t behave that way. Those “sick” individuals might have there explanations which are completely logical to them and them only.
Now, we say there should be a freedom of information. Does it include *****? Does it include ***** stuffs? Does it include ***** and bizarre stuffs? Who wants to see a person under wheels? who wants to see a kid being *****? who wants to see a ***** or ***** video? So, you see Deep Web is just bullshit.
i doubt whether there is someone who can really vouch for the folklore that says deep web is the holy grail of information and all. If there is someone out there please do the honors show us noobs some real info which you assume to be true. I don’t want to learn how to make bombs or how to kill someone but i want to learn some true wisdom.
One thing about deep web that i personally like is that it gives an individual from a oppressive country to raise his voice which i think is very necessary.
There is huge information on the surface web itself let alone deep web.
Last but not the least good job GYP nice post.
Thanks for stopping by, and thanks for very useful and constructive insights
I agree, there should be freedom of information to an extent, but when that freedom encroaches on someone elses more important rights it should be curtailed. I’m not sure how the likes of Wikileaks fits into my definition though.
Thanks again,
Gyp
PS Sorry for the late response, I’ve been very busy.
Judging by the comments, most people are not realizing what you are getting at. If you go on the Hiddenwiki and follow some of their links, then yes, you can find a lot of cool, albeit useless information. But the “deepweb” isn’t an actual place, it just refers to sites that aren’t indexed in search engines (usually for good reason–because there is virtually no need to do so). Most of the deepweb is just junk that programmers purposefully left out in the search engine index; not to hide something, but just so it wouldn’t be in the way.
I’m guessing ‘Anonymous’ here is 16, a /b/tard, and has only been to a few sites on the Hiddenwiki. That comment was so absurd that I’m surprised you responded at all.
I agree entirely kgiro, there is the literal definition of “deep web” as you mention, and then there is the likes of Tor that can also be “deep web” but they don’t represent the whole definition.
As for Anon, well everyone is entitled to their opinions and I’m also willing to listen and see if I can learn more
Again some people believe that Tor is the whole deep web, but we know it’s not.
Thanks for stopping by
Gyp
We all get lulled to thinking that the big G rules, but not when the dark net is involved. My input: http://www.findthatfile.com, a file based search engine (vertical ‘deep web’ search) that searches contents, extracts, internal metadata and almost every major source and file type from over 300 million URLs.
What a great read, thanks for taking the time and the bother. I’ve just heard all this stuff abut the “deep web” recently and I’m really skeptical, it seems like a hoax or something. Not because of the existence of the web itself, but because of the content of the myths. Even so, I stumbled across articles such as this one
http://brightplanet.com/wp-content/uploads/2012/03/12550176481-deepwebwhitepaper1.pdf
wich seem fairly serious and claim that the the deep web is at least 400 bigger than the surface web, the quality of information is much greater, and the contents of pages are dynamically generated (wich doesn’t seem to be the case of coporations databases and the like, or am i wrong?) Is this all false, is this true and it applies to to the descrption you’ve posted, or what is it?
Sorry for the bad english
Hi Jose,
Thanks for stopping by
You mention the deep web being 400 times bigger than the surface web, I’d be tempted to agree. I think however it does include corporate databases and companies have much more information internally than they do on their websites. Hence my logic that the deep web will be bigger! Sharepoint is a big corporate product and it’s a website
Gyp
Hey guys!I have been exploring the deep web for a few days and i am in the position to mention that the things that exist there are really “heavy”. You will have to be prepared to see things that are “REAL” and not acceptable by society. You can see from drug,weapon dealers, hitmen, the worst scums of the planet like [redacted] and c[redacted], also atrocities like [redacted](!!!!),[redacted], [redacted] (and consider that even me i found it kinda “too much” and i am a [redacted] fan in surface web,being a member of any [redacted] site),but of course i realized that the “good” side of the deep web is much harder to be found. I ‘ve managed to find some great knowledge and I ‘m sure there is much more…I m not a hacker to go much deeper but I wish someone could help me to come across with the interesting part of the deep web and do not encounter all these hideous stuff each time I try to find something new.But anyway if you have a weak stomach or you are narrow minded never visit deep web..if you are seeking for knowledge and you are ready to put yourself in a challenge do it.I hope there were some people to track and hunt down to death all these child molesters…but usually nerds cant be vigilantes in real life…
Hey there,
Thanks for dropping by
Sorry for redacting some of your comment, I hope to have kept it’s meaning intact. Not words I want associated with my website.
As you mention I’m sure you can find some of that content on the surface web if you knew where to look. I think that they make it easier to find on the deep web, as opposed to hiding it on the surface web.
All the best for your future explorations!
Gyp