Dynamic ontology. As Palantir engineers explain this to the CIA, the NSA, and the military
Palantir is Silicon Valley's fourth-highest privately held company (after Uber, Xiaomi, and Airbnb). While Palantir collects information about everything in the world, we collect information about him. Together with Edison, we continue to investigate the capabilities of the Palantir platform.
IT specialists figured out how to effectively “monetize mathematics and algorithms” (Segalovich, Bakunov), PayPal Mafia thought of how to monetizeFeanor’s gadgets philosophy (Palantir capitalization - $ 20 billion).
In a ten-minute lecture, an employee of the company Palantir will talk about the central concept of their system - a dynamic ontology.
0:00 Hi, I'm Asher Sinenski, Palantir Technology Deployment Engineer. I will talk about dynamic ontology.
0:08 Obviously, now, these two words look rather vague for you, I hope that by the end of the conversation you will understand what meaning we put into them.
0:17 Before I get down to business, I’ll explain: many people have problems with the word ontology. What do we mean by this word?
0:24 If you look at the roots of this word, it is formed from the Greek “ontos” (being) and “logic” (learning something). In essence, ontology is a categorization of the world.
0:34 There are many terms that people use to describe this: taxonomy, schematizer of a data model. But we use it, in a broader sense, as the idea that we are truly categorizing the world in some way.
0:43 The idea of building an ontology for exploring the world is not new. The first to approve this idea was a man named Plato. The idea of Platonic realism, basically, is that there are real things, but there is our idea of things.
1:02 In his model, things in the real world are a manifestation of certain patterns from the ideal world.
1:07 He talked about the universal and the particular; about ontology and being itself; about form and copy (perhaps an idea and an embodiment?). Computer scientists, however, have a class and an object.
1:15 In general, this is the idea that there is a concept of something and that it is something. And what makes it an apple brings us closer to what the concept of an apple is (correct me, people who are familiar with the philosophy of Plato).
1:29 Well, this is a cool philosophy, but the question now is how it can be used, how it will help us in creating a tool that can be useful in the information community.
1:36 A great example of a convenient ontological structure is the periodic table. We have the number ten here (pointing to arsenic, atomic mass), it says something about weight, about mass, but what we see here is not “ten,” in our real world, it’s conceptualizing what ten should mean.
1:56 Another well-known example of ontology, or rather taxonomy: it is a linear taxonomy with Latin names for animals, and it is a good example of hierarchical ontology.
2:06 Here you can move down to the genus panthera (lat. Panther), and you can use the division into species with Latin names: leo, tigris, pardus. You can create lion, tiger, and leopard objects.
2:20 Panthera pardus is obviously not the leopard itself, it is a concept of what the leopard is.
2:25 Well, and you see, there is a living leopard in the real world and panthera pardus in ontology.
2:29 These same concepts can be applied to areas that are more significant if we are talking about a person. For example, here is the model of the human ontology (person): we started with objects, moved on to entities, then to living entities, then to people.
2:43 If we want to characterize a person further, then we can add concepts such as: pilot, lawyer, and doctor — these concepts are united by the general idea of distinguishing subspecies. We will talk about this later.
2:53 So how can you model in ontology in practice? How would you structure your ontology?
2:58 People have tried several ways. I will list four of the most pronounced approaches.
3:03 Upper tied to objects. In object models, for example, animal taxonomy by Latin name, we have certain objects that are concepts that should correspond to real-world objects. You can add a little depth, saturation, adding features, and thus get objects with features.
3:21 For example, we can take the periodic table, where objects have, weight and number of protons.
3:28 What is missing from this table is a reference to the relationship. Some relationships can be understood by analyzing the structure of the table, but you cannot know, for example, whether these two substances form an alloy.
3:40 Another way you can model the world is with objects and relationships, that is, you have objects and how they relate to each other.
3:47 And here are the signs that were on the periodic table. You can get an idea of the signs if you rely on the relationship between two objects, for example, one of the objects with a data object, and if you want to say that Mike Fickrey has a (fictional villain from one of the first Palantir videos) blue eyes, you are creating a connection between the Mike Fickrey and blue eyes objects.
4:08 It will be a little strange, but you can get a fairly complete picture of the object.
4:13 Finally, the most revealing way is using objects, and relationships, and signs. As you can guess, this is the method we use in Palantir.
4:22 Let's look a little deeper at the Palantir ontology, how it is applied.
5:08 (Ontology pervades almost every function in the Palantir workspace. Therefore, having a properly designed ontology is critical for effective analysis.)
5:15 I said that Palantir is a dynamic ontology, let's talk about what I mean under the dynamics.
5:21 The first thing we mean when talking about dynamics is that the ontology in Palantir is not hardcoded. For example, the axial layer, the interpretation layer, the add-on above the databases, the analysis and user interaction layer are executed in this way.
5:33 There are some concepts that are hard coded: an object, a sign, a relationship. Objects are also rigidly divided into documents, entities and events. And that’s it, the whole ontology, given in advance. In fact, it is more than the ontological structure to which you are attached and which you are used to using.
5:50 Why soft coding? Why not code the ontology hard? I will talk about a couple of drawbacks of this approach.
6:00 The problem is that using hard-coded ontology, you inevitably end up in one of two camps.
6:06 You have either a very generalized ontology or a very special ontology, and you have to choose where you want to be.
6:12 In a generalized ontology, you are not actually modeling anything. You have objects, signs and relationships, and you can put anything inside such an ontology, but, in general, that's all, there is nothing to model there.
6:21 Some people try to create very specialized ontologies where you can model whatever you want. The problem here is that semantics become over-defined.
6:30 What do we mean by overdetermined semantics? This is a situation where it becomes excessively difficult to understand how something should be modeled.
6:35 You can enter the concept of “notice” (citation - extract, subpoena, mention), and then one will say: “Oh, notice, this is when I receive a parking ticket,” and the other that notification comes when a person receives a bonus.
6:48 If you need to model similar things in an overdetermined ontology using "notification", it becomes difficult to separate them.
6:54 If you use soft coding, a spectrum arises between our camps, and starting with a general model, you will reach a point that meets your goals (sweet spot).
7:11 Another significant advantage of a dynamic ontology is that it is a flexible system: you can model many different concepts, or one concept in many different ways.
7:19 Flexibility - Let's see how it works. We answer the question: “How to simulate a person’s occupation?”
7:28 On the slide it says: “A person’s working function can be classified through the definition of objects.”
First, take a look at the object model of human employment. We classified work functions as objects, and you have already seen this slide.
7:40 We have a man at the top and a pilot, a lawyer, a doctor at the bottom, so it looks like a model of animals with genera and species.
7:47 What are the advantages and disadvantages of this method? The disadvantage is that you cannot make a person a lawyer and a doctor at the same time (like a lion and a tiger), you have to choose.
7:56 The advantage is that you have many signs specific to the doctor or lawyer, and you can use these signs.
8:04 Medical specialization, obviously, differs from legal, and you can easily use this difference, working with objects.
8:12 On the slide it says: "The working functions of a person can be classified by the definition of signs of his employment."
You can use the signs to get an idea of the person’s employment.
8:17 We can say that a person can have the sign of “employment” and the different meanings of this sign.
8:24 You can be a doctor, a pilot, a lawyer, and if you have an idea about the multiplicity of a characteristic, you can choose several values. At Palantir, this is possible: a doctor and a pilot, a pilot and a lawyer, a lawyer and a cook.
8:35 The disadvantage here is that you have to try to add information to the lawyer that he is from such and such a university, without permanently linking this information to this attribute.
8:45 Finally, you can get an idea of a person’s career using relationships. The reason someone is called a doctor is not because he graduated from medical school, the reason is that this person treats patients.
8:58 So you can say that, person 1 is a doctor in relation to person 2, person 1 is a pilot because he controls the plane, person 1 is a lawyer because he provides legal services to person 3. So, in fact, it is determined career.
9:08 So what we see here are three concepts of career modeling. You cannot do this if you use hard-coded ontology, for example, the norms of society.
9:21 I talked about flexibility, soft coding, but not all of these things imply dynamics. So what is really the dynamism of the Palantir ontology?
9:31 The fact is that in Palantir, the ontology can be changed even after it has been deployed. Thus, an ontology can evolve along with how your organization's vision of the world develops.
9:41 A few examples of changes that you can make:
- any object, attribute or relationship can be deleted or added;
- the functionality of objects, signs and relationships can be changed, - and this is probably the most important.
9:54 This is important because the way that most people interact with ontology is to try to understand why there are various signs and so on.
10:05 Here on the slide, for example, is a list of some of the things you can change:
10:08 These changes have a significant impact on how people interact with the ontology and how the analysis will go.
10:15 Now let's quickly summarize. Using a dynamic ontology, you can model everything your organization needs in your area of interest. This ontology is not a hardcode, but a softcode.
(Special thanks to Alexei Vorsin, a Russian expert on the Palantir system, for helping prepare this article.)
More about Palantir:
IT specialists figured out how to effectively “monetize mathematics and algorithms” (Segalovich, Bakunov), PayPal Mafia thought of how to monetize
In a ten-minute lecture, an employee of the company Palantir will talk about the central concept of their system - a dynamic ontology.
0:00 Hi, I'm Asher Sinenski, Palantir Technology Deployment Engineer. I will talk about dynamic ontology.
0:08 Obviously, now, these two words look rather vague for you, I hope that by the end of the conversation you will understand what meaning we put into them.
0:17 Before I get down to business, I’ll explain: many people have problems with the word ontology. What do we mean by this word?
0:24 If you look at the roots of this word, it is formed from the Greek “ontos” (being) and “logic” (learning something). In essence, ontology is a categorization of the world.
0:34 There are many terms that people use to describe this: taxonomy, schematizer of a data model. But we use it, in a broader sense, as the idea that we are truly categorizing the world in some way.
0:43 The idea of building an ontology for exploring the world is not new. The first to approve this idea was a man named Plato. The idea of Platonic realism, basically, is that there are real things, but there is our idea of things.
1:02 In his model, things in the real world are a manifestation of certain patterns from the ideal world.
1:07 He talked about the universal and the particular; about ontology and being itself; about form and copy (perhaps an idea and an embodiment?). Computer scientists, however, have a class and an object.
1:15 In general, this is the idea that there is a concept of something and that it is something. And what makes it an apple brings us closer to what the concept of an apple is (correct me, people who are familiar with the philosophy of Plato).
1:29 Well, this is a cool philosophy, but the question now is how it can be used, how it will help us in creating a tool that can be useful in the information community.
1:36 A great example of a convenient ontological structure is the periodic table. We have the number ten here (pointing to arsenic, atomic mass), it says something about weight, about mass, but what we see here is not “ten,” in our real world, it’s conceptualizing what ten should mean.
1:56 Another well-known example of ontology, or rather taxonomy: it is a linear taxonomy with Latin names for animals, and it is a good example of hierarchical ontology.
2:06 Here you can move down to the genus panthera (lat. Panther), and you can use the division into species with Latin names: leo, tigris, pardus. You can create lion, tiger, and leopard objects.
2:20 Panthera pardus is obviously not the leopard itself, it is a concept of what the leopard is.
2:25 Well, and you see, there is a living leopard in the real world and panthera pardus in ontology.
2:29 These same concepts can be applied to areas that are more significant if we are talking about a person. For example, here is the model of the human ontology (person): we started with objects, moved on to entities, then to living entities, then to people.
2:43 If we want to characterize a person further, then we can add concepts such as: pilot, lawyer, and doctor — these concepts are united by the general idea of distinguishing subspecies. We will talk about this later.
2:53 So how can you model in ontology in practice? How would you structure your ontology?
2:58 People have tried several ways. I will list four of the most pronounced approaches.
3:03 Upper tied to objects. In object models, for example, animal taxonomy by Latin name, we have certain objects that are concepts that should correspond to real-world objects. You can add a little depth, saturation, adding features, and thus get objects with features.
3:21 For example, we can take the periodic table, where objects have, weight and number of protons.
3:28 What is missing from this table is a reference to the relationship. Some relationships can be understood by analyzing the structure of the table, but you cannot know, for example, whether these two substances form an alloy.
3:40 Another way you can model the world is with objects and relationships, that is, you have objects and how they relate to each other.
3:47 And here are the signs that were on the periodic table. You can get an idea of the signs if you rely on the relationship between two objects, for example, one of the objects with a data object, and if you want to say that Mike Fickrey has a (fictional villain from one of the first Palantir videos) blue eyes, you are creating a connection between the Mike Fickrey and blue eyes objects.
4:08 It will be a little strange, but you can get a fairly complete picture of the object.
4:13 Finally, the most revealing way is using objects, and relationships, and signs. As you can guess, this is the method we use in Palantir.
4:22 Let's look a little deeper at the Palantir ontology, how it is applied.
5:08 (Ontology pervades almost every function in the Palantir workspace. Therefore, having a properly designed ontology is critical for effective analysis.)
5:15 I said that Palantir is a dynamic ontology, let's talk about what I mean under the dynamics.
5:21 The first thing we mean when talking about dynamics is that the ontology in Palantir is not hardcoded. For example, the axial layer, the interpretation layer, the add-on above the databases, the analysis and user interaction layer are executed in this way.
5:33 There are some concepts that are hard coded: an object, a sign, a relationship. Objects are also rigidly divided into documents, entities and events. And that’s it, the whole ontology, given in advance. In fact, it is more than the ontological structure to which you are attached and which you are used to using.
5:50 Why soft coding? Why not code the ontology hard? I will talk about a couple of drawbacks of this approach.
6:00 The problem is that using hard-coded ontology, you inevitably end up in one of two camps.
6:06 You have either a very generalized ontology or a very special ontology, and you have to choose where you want to be.
6:12 In a generalized ontology, you are not actually modeling anything. You have objects, signs and relationships, and you can put anything inside such an ontology, but, in general, that's all, there is nothing to model there.
6:21 Some people try to create very specialized ontologies where you can model whatever you want. The problem here is that semantics become over-defined.
6:30 What do we mean by overdetermined semantics? This is a situation where it becomes excessively difficult to understand how something should be modeled.
6:35 You can enter the concept of “notice” (citation - extract, subpoena, mention), and then one will say: “Oh, notice, this is when I receive a parking ticket,” and the other that notification comes when a person receives a bonus.
6:48 If you need to model similar things in an overdetermined ontology using "notification", it becomes difficult to separate them.
6:54 If you use soft coding, a spectrum arises between our camps, and starting with a general model, you will reach a point that meets your goals (sweet spot).
7:11 Another significant advantage of a dynamic ontology is that it is a flexible system: you can model many different concepts, or one concept in many different ways.
7:19 Flexibility - Let's see how it works. We answer the question: “How to simulate a person’s occupation?”
7:28 On the slide it says: “A person’s working function can be classified through the definition of objects.”
First, take a look at the object model of human employment. We classified work functions as objects, and you have already seen this slide.
7:40 We have a man at the top and a pilot, a lawyer, a doctor at the bottom, so it looks like a model of animals with genera and species.
7:47 What are the advantages and disadvantages of this method? The disadvantage is that you cannot make a person a lawyer and a doctor at the same time (like a lion and a tiger), you have to choose.
7:56 The advantage is that you have many signs specific to the doctor or lawyer, and you can use these signs.
8:04 Medical specialization, obviously, differs from legal, and you can easily use this difference, working with objects.
8:12 On the slide it says: "The working functions of a person can be classified by the definition of signs of his employment."
You can use the signs to get an idea of the person’s employment.
8:17 We can say that a person can have the sign of “employment” and the different meanings of this sign.
8:24 You can be a doctor, a pilot, a lawyer, and if you have an idea about the multiplicity of a characteristic, you can choose several values. At Palantir, this is possible: a doctor and a pilot, a pilot and a lawyer, a lawyer and a cook.
8:35 The disadvantage here is that you have to try to add information to the lawyer that he is from such and such a university, without permanently linking this information to this attribute.
8:45 Finally, you can get an idea of a person’s career using relationships. The reason someone is called a doctor is not because he graduated from medical school, the reason is that this person treats patients.
8:58 So you can say that, person 1 is a doctor in relation to person 2, person 1 is a pilot because he controls the plane, person 1 is a lawyer because he provides legal services to person 3. So, in fact, it is determined career.
9:08 So what we see here are three concepts of career modeling. You cannot do this if you use hard-coded ontology, for example, the norms of society.
9:21 I talked about flexibility, soft coding, but not all of these things imply dynamics. So what is really the dynamism of the Palantir ontology?
9:31 The fact is that in Palantir, the ontology can be changed even after it has been deployed. Thus, an ontology can evolve along with how your organization's vision of the world develops.
9:41 A few examples of changes that you can make:
- any object, attribute or relationship can be deleted or added;
- the functionality of objects, signs and relationships can be changed, - and this is probably the most important.
9:54 This is important because the way that most people interact with ontology is to try to understand why there are various signs and so on.
10:05 Here on the slide, for example, is a list of some of the things you can change:
- add / change shortcut generators;
- add / change aliases;
- add / change parsers;
- add / change control parameters;
- add / change icons;
- add / change approximate values;
- add / change feature restrictions;
- add / change link restrictions.
10:08 These changes have a significant impact on how people interact with the ontology and how the analysis will go.
10:15 Now let's quickly summarize. Using a dynamic ontology, you can model everything your organization needs in your area of interest. This ontology is not a hardcode, but a softcode.
(Special thanks to Alexei Vorsin, a Russian expert on the Palantir system, for helping prepare this article.)
More about Palantir:
- Cyber Counterintelligence. How Palantir Can Catch Snowden
- Palantir: how to detect a botnet
- Palantir and money laundering
- Palantir: arms trade and pandemic spread
- Palantir, PayPal mafia, special services, world government
- Palantir 101. What is allowed to mere mortals to know about the second highest private company in Silicon Valley