How can I use artificial intelligence to solve SEO problems

The search model must be capable of "self-calibration." That is, it should be able to take its own algorithms, their specific gravity and compare the simulated data with public search engines in order to identify the most accurate search engine that allows you to simulate any environment.

However, analyzing thousands of parameters, trying to find the best combination of them, is astronomically expensive in terms of computational processing, and also very difficult.

So how, then, to create a self-calibrating search model? It turns out that the only thing left for us is to turn to ... birds for help. Yes, yes, you heard right, to the birds!

Particle Swarm Optimization ( PSO)

It often happens that grandiose problems find the most unexpected solutions. For example, it is worth paying attention to optimization using a swarm of particles, which is an artificial intelligence method, first mentioned in 1995 and based on the socio-psychological behavioral model of the crowd. The technique is actually modeled on the concept of the behavior of birds in a flock.



In fact, all the algorithms that work on the rules that we created today still cannot be used to find at least approximate solutions to the most complex problems of numerical maximization or minimization. But using such a simple model as a flock of birds, you can immediately get the answer. We have repeatedly heard terrifying predictions about how one day artificial intelligence will take over our world. However, in this particular case, he just becomes our most valuable assistant.

Scientists were involved in the development and implementation of many projects dedicated to the Swarm Intelligence. So, in February 1998, the Millibot project, formerly known as Cyberscout, a program launched by the US Marine Corps, was launched. Cyberscout was essentially a legion of tiny robots that could infiltrate a building, spanning its entire territory. The ability of these high-tech crumbs to communicate and transmit information among themselves made it possible for the “swarm” of robots to act as a single whole organism, turning the very laborious task of exploring the whole building into a leisurely walk along the corridor (most of the robots had the opportunity to travel no more than a couple of meters).

Why does it work?

The really cool thing about PSO is that the methodology makes absolutely no assumptions about the problem you are trying to solve. It is a cross between a rule-based algorithm trying to come up with a solution and artificial intelligence neural networks that aim to investigate issues. Thus, this algorithm is a compromise between exploratory and exploitative behavior.

Without a research nature, this optimization approach, algorithm would undoubtedly turn into what statisticians call a “local maximum” (a solution that seems optimal, but is not really).

First of all, you start with a series of "swarms" or conjectures. In the search model, these can be various weighting coefficients of scoring algorithms. For example, having 7 different inputs, you will start with at least 7 different assumptions regarding these weights.



The idea of ​​PSO is to make each of these assumptions as far away as possible from the rest. Without going into 7-dimensional calculations, you can use several techniques to make sure your starting points are optimal.

After that, you will begin to develop your guesses. During this, you will imitate the behavior of birds in a flock in a situation where there was food near them. One of the random guesses (flocks) will be closer than the others, and each subsequent guess will be adjusted based on general information.

The visualization shown below clearly demonstrates this process.

Implementation

Fortunately, there are a number of possibilities for implementing this method in different programming languages. And the great thing about particle swarm optimization is that it's easy to make it real! The technique has a minimum of settings (it is an indicator of a strong algorithm) and a very small list of disadvantages.

Depending on your problem, the idea may turn out to be at a local minimum (not an optimal solution). You can easily fix this by introducing a neighborhood topology that quickly reduces the feedback loop to the best of the surrounding assumptions.

The bulk of your work will consist of developing an “adaptive function” or ranking algorithm that you will use to determine how close you are to the target correlation. In our case, with SEO, we will have to correlate the data with some given object like the results of Google or any other search engine.



If you have a working scoring system, your PSO algorithm will try to maximize performance through trillions of potential combinations. A scoring system can be as simple as performing Pearson correlation between your search model and the search results of network users. Or it can become as complex as the simultaneous use of these correlations and the assignment of points for each specific scenario.

Black Box Correlation

Recently, many SEOs have been trying to correlate with Google's black box. These efforts, of course, have the right to life, but are still quite useless. And that's why.

First, correlation does not always imply a causal relationship. Especially if the entry points to your black box are not too close to the exit points. Let's look at an example where entry points are very close to their respective exit points - the ice cream business. When it's warm outside, people buy more ice cream. It is easy to see here that the entry point (air temperature) is closely tied to the exit point (ice cream).

Unfortunately, most seo optimizers do not use the statistical proximity between their optimizations (inputs) and their corresponding search results (outputs).



Moreover, their inputs or optimizations are in front of crawl components in the search engine. In fact, typical optimization should go through 4 levels: crawling content, indexing, scoring, and, ultimately, the real-time query level. An attempt to correlate in this way cannot give anything but vain expectations.

In fact, Google provides a significant noise figure, similar to how the US government makes noise around its GPS network, which means that civilians are not able to receive the same accurate data as the military. This is called the real-time query level. And this layer is becoming a serious deterrent to SEO correlation tactics.



An example here is a garden hose. Being on the scoring layer of the search engine, you get the company's view of what is happening around. The water leaving the garden hose is organized and predictable - that is, you can change the position of the hose and predict the corresponding change in the movement of the water flow (search results).

In our case, the query layer sprays this water (search results) into millions of droplets (variations of the search results), depending on the user. Most algorithms that are changing today arise based on the level of queries in order to produce more variations of search results for the same number of users. Google’s Hummingbird Algorithm is one example. Query level shifts allow search engines to generate more marketplaces for their PPC ads.



The query level is the view of users, not the company, about what is happening. Therefore, correlations deduced in this way will rarely have causal relationships. And this is provided that you have one tool to find and model data. As a rule, seo optimizers use a number of input data, which will increase noise and reduce the likelihood of searching for a causal relationship.

Search for causal relationships in SEO

To get a correlation for working with a search engine model, you need to tighten the inputs and outputs as much as possible. In a search engine model, input or variable data must be in or above the scoring layer. How to do it? We need to break the black box of the search engine into key components, and then build a search engine model from scratch.

Optimizing the outputs is even more difficult due to the terrifying noise arising from the real-time query layer, which creates millions of variations for each user. At a minimum, we will need to make such inputs for our search engine model, which will be located in front of the usual layer with query variations. This ensures that at least one of the compared parties is stable.

By building a search engine model from scratch, we can display search results that come not from the query level, but directly from the scoring layer. This will give us a more stable and accurate relationship between the inputs and outputs that we are trying to relate. And then thanks to these strong and indicative relationships between inputs and outputs, the correlation will reflect a causal relationship. Focusing on one input, we get a direct connection with the results that we see. Then we can do a classic seo analysis to determine the optimization option that will be beneficial for the existing search engine model.

Summary

Situations when some simple thing in nature leads to scientific discoveries or technological breakthroughs cannot but delight. Having a search engine model that allows us to openly combine scoring entries with non-personalized search results, we can relate the correlation to a causal relationship.

Add to this particle swarm optimization, and you have a technical breakthrough - a self-calibrating search model.