Crawling is the discovery process that search engines do by sending their team of bots known as crawlers, spiders or bots to find new and updated content. These bots can find different types of content it may be a web page, an image, a video, a PDF, etc. - but no matter what format the result (content) is it is detected using this process.
Googlebot starts by fetching a few web pages, and then follows the links within those pages, to find new URL's.
%20(1).jpg "What is search engine crawling?")
By moving between these links, the crawler can find new content and add it to the index called ( Caffeine ), which is a huge database containing the discovered (URL) addresses, and the crawler uses this index later to retrieve these addresses when the researcher requests a query or search for information that matches the The content on or closely related to these titles.
?What is the search engine index
Search engines process and store the information they find in their index. It is a huge database containing previously discovered content, and the search engine matches it with a web or search query to present it to the researcher or internet user.
Create and show results
Search engines generate results as soon as the user submits a web query or search, a process that begins with a query match check against all records stored in the index.
Keywords in search engines (SEO keywords)
Keywords to improve visibility in search engines, also known as keywords or key phrases, are terms that are added to the content that you publish online in order to improve the visibility of that content in search engines . Selected based on multiple criteria such as search rate, competition and commercial objective.
When you improve your content ( configuration ) by adding words and phrases that people are searching for, this can improve your ranking in search engines when a user queries or searches for those words.
Appearance of the site in search engines
Improving your ranking on the search engine results page increases the number of times your site appears and thus increases site visitors significantly
That is why the first step in any search engine optimization campaign or plan is to find the keywords that people are most searching for.
First: Search engines that base their work on the web crawler.
Crawler-based search engines use automated programs to scan and rank web pages, and these programs are called spiders, crawlers, or bots.
The spider will find the requested web page and download it, analyze the information displayed on it, and will add the page to the search engine database, when the searcher or user performs a search, the search engine will scan its database for the keywords that the user searched for to show the appropriate results to the user.
Results or a list of suggested links appear on the search results page based on their similarity to what the user has searched for and in the order the bot sees fit. Crawler-based search engines are constantly searching for new web pages to update their database.
\The directory uses human editors or programmers to decide which category each website belongs to. They sort websites into specific categories and store them in their database. The sorting process is done based on the information available on this website in general using a set of rules and criteria. Previously specified by the directory.
At the moment, these are the two most popular major guides:
Note: Since late 2002, Yahoo Maktoob has provided its search engine using crawler-based technology as well as its own directory.
Google search engine ranks first in the list of the most popular and popular search engines around the world.
Google is the giant of search engines, it has the most advanced algorithms, these algorithms include artificial intelligence AI, machine learning and RankBrain, and it has another algorithm that enables Google to modify the criteria for the order of appearance of pages on the results page based on user behavior and the quality of previous results, for this and since In 1998, more than 70% of Internet users performed searches on the Internet through Google.
Baidu
Baidu is the first search engine launched in China in 2000, it's like Google but it's Chinese, Baidu cooperates with companies like Microsoft , Qualcomm , Intel or Daimler on many artificial intelligence projects.
Similar to Google, it offers a lot of solutions such as cloud services, maps, social networking, image and video search, and many other solutions.
Bing
Microsoft launched its search engine in 2009 as a new project after several previous search engines launched such as ( MSN Search ) and ( Windows Live Search ).
The main goal was to develop a competitor to Google, but if we look at the issue globally, there is no room to compete with Google, but in the United States of America in particular, Bing is the second most popular search tool on the Internet.
Yahoo!
Yahoo was the most used email and search engine provider around the world, as the company grew significantly in the nineties, but after 2000 it began to lack innovation and modernization and lost its value, in 2017 it was bought by Verizon Communications .
Ask
Previously known as Ask Jeeves, which was launched in 1996 and was designed to answer questions sent by the user using the search form, and thanks to the Ask toolbar, it was able to compete with the major search engines such as Bing, Yahoo and Google, but unfortunately not much benefit from This toolbar is caused by the browser not being desirable.
Advantages of search engines
When your page appears in the first positions on the search results page, this will lead to the assumption that you are the most reliable option.
40% of search engine users click on the first pages that appear on the search results page, especially if they are looking for quick answers, the majority of users do not know what practices or tools you use to improve your visibility in search results to get to the top of the list, so they simply They trust you, and user trust will have a lasting impact even if you change your ranking on the web results page.
Increases the number of visitors to the site (Targeted Traffic)
People often use search engines to search for a specific solution to an immediate problem or need. To take advantage of this for commercial purposes and to attract more potential customers, it should appear in the first results on the search results page when the user searches for a service you provide.
While paid ads always appear at the top of the results, 70%-80% of searchers ignore those paid listings and go straight to the unpaid results. These users are exactly the audience you are looking for and who you want to attract to your business.
Disadvantages of search engines
Search engine optimization (SEO) results are not immediately visible; Since search engines take time to index the content you are trying to improve, it takes days or even weeks for your website to rank better.
This means that if you need to generate a large number of visitors to the site immediately you should try other methods and methods, here comes the role of Pay Per Click campaigns where your paid ads appear at the top of the search results page ranking ( SERP) quickly.
Difficulty obtaining competitive keywords
There is a high probability that your competitors in the market will use the same keywords that you are, and if you are competing with large and well-known companies, you will not be able to overcome their dominance with these general keywords, so you should devote more time and money to improving the appearance of those words or using keywords Your own and less competition by customizing it based on your geographic location.
Algorithms change
Search engines, and Google in particular, frequently update their algorithms to get more accurate results. While these updates are meant to help consumers get better results, even small changes in the algorithm can move your website from first place to somewhere on the third page. In the possibility that 75% of users or searchers only open the first page of results pages, you should ensure that it is always present on the first page by using paid ads to support your efforts to improve your visibility using keywords.
Reduces your advertising costs
SEO strategies benefit your business by lowering your advertising costs - and even your marketing expenses. Your company can always be on the first page in the search results page if you use a good and well-planned optimization strategy.
SEO increases your local marketing efforts
Even if you are a local business that relies on offline transactions, SEO benefits your business. With the increasing number of consumers who own smartphones, the popularity of local search is increasing. The value of local search is also huge, as 80 percent of local searches are converted into purchases, which means more revenue for your business.
Generates more leads and sales and increases your market share
One of the most mentioned benefits of SEO is the increase in sales, leads and market share of the company, you will notice the impact of this immediately in your daily operations as you will see shoppers visit your store or increase in purchase traffic from your store.
Search Engine Marketing (SEM)
Search engine marketing shows your ad to people who are searching for your product or service on the Internet.
The main advantage of this type of marketing is that your ad is shown at the moment a person is looking for your product, and since you skip many marketing processes, such as product and branding, this is usually the most profitable type of advertising. It is also considered "inbound marketing" because you only need to receive those who are actually interested in your product.
In search engine marketing (SEM ), you pay when someone clicks on your ad, so it's a type of pay-per-click (PPC), which also includes image, video, and other ad formats.