Embedded Languages: Why Lua?
This material continues a series of publications based on the reports that we made at the Games Gathering 2017 conference last December. One of the reports touched upon the topic of choosing an embedded scripting language.
As already mentioned in the previous post of our blog , our company has its own engine. Today we will talk about how we were guided when choosing a scripting language for this engine.
Why is there a need for scripting languages at all? As the saying goes, “there are no difficult problems in the gaming industry, there is no need to demand complex solutions . In addition to experienced (and expensive!) Programmers who solve complex problems, we need people (many people!) Who will deal with, say, quests. And honestly, we would like these programmers to be not so expensive, and ideally not programmers at all, but game designers and scriptwriters themselves.
Thus, there is a need for a tool for describing simple, but nevertheless logic, without involving heavy artillery of programmers. We conclude - what is a scripting language for us? This is a tool that will make game development faster and cheaper.
The question immediately arises, why don't we just use something like XML? The fact is that for our purposes, we often need control constructs — branching and loops, while XML is a declarative description.
Another advantage of scripting languages is that scripts in a project can be both code and a resource. And, accordingly, it is possible to update the script part of the game not only together with the code, that is, during normal updates via the app store mechanisms. But together with resources - that is, along with graphic and other materials, using CDN.
We formulate the requirements for an ideal scripting language.
Let us analyze some popular programming languages that are used as scripting languages to meet these requirements.
Python is a dynamic language that is quite popular. It is characterized by a fairly flat learning curve, which is fairly easy to learn. However, it is not so easy to study it properly. As a result, good Python programmers are rare and expensive. This is contrary to our desire to speed up and cheapen the development of game logic.
Python has great features, good performance. His problem is inconsistent library system. Another of his problems, which plays an important role for us, is that it is, in fact, not an embedded language. This is the language from which it is convenient to call libraries written in C or C ++.
Regarding the possibilities for embedding Python, we can say that, for example, there is Maya, where Python is used. But the one who saw inside the Maya plugins written in Python would agree with us that they don't look very good.
As a result, we can say that Python, with all its strengths, does not suit us. Now consider JavaScript.
JavaScript is, without exaggeration, a great language that literally captured the world.
JavaScript is a popular dynamic language, featuring a gentle learning curve, with extensive capabilities, good performance, and an extensive set of libraries.
If we need a language interpreter to build the game engine, we can find a lot of such interpreters. In reality, you have to choose from two similar projects - V8 and WebKit. Both are large enough. As a result, if we are talking about board games, it would be possible to take a chance and include the whole interpreter in the game, but in the case of mobile games this option does not suit us.
SocialQuantum has its own JavaScript interpreter, which passes 98% of the tests, we plan to transfer this project to the category of open source.
As a result, it turns out that JavaScript looks like a strong candidate for the role of an embedded language, but it does not suit us either.
Here it should be noted that when the conversation about JavaScript comes, the following is usually remembered by Haxe. But, in fact, there is no point in talking about the possibility of using this language as an embedded one, since Haxe, in fact, is not so much a language as a trans compiler into other languages. And this is not what we need.
Maybe we will be satisfied with ActionScript or some other scripting language?
If we formally analyze ActionScript for compliance with the above requirements, it would seem that the ideal scripting language has been found. On its side is the dynamic nature, popularity, ease of learning, good features, performance, availability of libraries, ease of embedding. This language is loved and remembered in the gaming industry; a great number of wonderful Flash games are written on it. The main problem of ActionScript is that this language is almost dead. Therefore, he does not like us either.
In addition to ActionScript, there are many scripting languages, such as AngelScript, Squirrel, and others. Among them, one can find such that, formally, almost completely satisfy our requirements, but usually these are languages that are tied to their developer, they have some kind of long-standing problems that have not been fixed for years. Most likely, they are not too popular, they are not well documented, there are not enough teaching materials for them, they have not very large communities. One of the consequences of this state of affairs is the fact that they are difficult to study - if only because it is not completely clear what they are and how they work.
As you can see, we have not yet found the perfect embedded language. What if you create your own language?
It is possible that the language developed within the company will ideally meet its needs and will be easy to learn. But, most likely, such a language will not become popular. Such a language will either have a minimum number of libraries, or they will not exist at all. In addition, it is hard to believe that in modern conditions it is possible to create something that will work better, that it will have greater productivity and will be easier to integrate than something that is already on the market.
There are companies that develop and use their own languages, among them there are successful players in the gaming market, but, most likely, this is not a good idea.
Having considered the existing programming languages that claim to be embedded and discuss the idea of developing your own language, we turn to Lua.
Lua is a dynamic language. It is quite popular, there is a large community around it, especially in the field of game development. It has a very gentle learning curve. For example, in our company scripts for autotests are written on Lua. The standard introductory course for autotesters takes about two hours, after which a person is able to use this language. At the same time, Lua is a multi-paradigm language. It supports functional programming style and OOP. As a result, it is suitable not only for solving some simple tasks, but also for more serious tasks that professional programmers are engaged in.
Lua has good performance and has quite a few libraries. Not so much like JavaScript, but, nevertheless, on the LuaForge websiteYou can find almost everything you need. And finally, Lua is very easy to integrate; moreover, it was created to be used as an embeddable language.
For example, this is how our working environment based on JetBrains IDE CLion looks like . Here you can see the auto-completion mechanism we created for Lua, which is planned to be made open-source. Open source and we are going to make a debugger.
Work Environment
We chose Lua, but when it comes to using it as an embedded scripting language, we usually have to deal with roughly the same objections that we will now consider.
Nobody argues with the fact that Lua is an excellent embedded language. The main thing that they consider it a minus is that it was created for use with the C language, not C ++. Because of this, trying to use something in Lua that is in C ++ and not in C, we run into problems. However, here we must understand that these problems were solved by many pretty smart people. Luabind, Luabridge, toLua ++, SQLuaHost are among the tools that solve the problems of embedding Lua in C ++ projects. This is not a complete list. They have different advantages and disadvantages, but, nevertheless, most likely, all that you may need is already implemented in one of these solutions.
Consider, for example, SQLuaHost. This is a binding that is made inside the company SocialQuantum, and which is planned to be made open-source. This decision represents our vision of how Lua should be bind. Therefore, it is quite possible that if you did not find what you need in existing binders, you will find it in SQLuaHost.
We often have to deal with the opinion that Lua is a very slow language. Firstly, it is not. Lua is a stack machine, and there, in fact, there is simply nothing to slow down. In addition, we must understand that in the scripting language, we usually give the game logic, business logic, and not some really heavy things. As a result, if Lua-scripts make the game slow down, then the problem may lie in the non-optimal binding or irrational use of some sort of language function. We, for example, conducted synthetic tests on which LuaJIT runs faster than Mono. At the same time, no one bothers to write about such non-optimal code:
Here in each game tick a new texture is created and set as a background. Of course, such a construction will not work very quickly, but no one bothers to write such things here.
The next objection is that Lua is made in order to write some small things and something big cannot be written in this language. On the one hand it is true. But this language has high modularity. And from a variety of small blocks, you can make quite large and complex systems. And if you recall what has already been said about multi-paradigm and OOP, then it turns out that OOP pushes the developer to create small modules that can be used to create large and complex structures.
At the same time, some small modules are often written quickly on Lua, and in the gaming industry “faster” means “cheaper”.
Criticizing Lua, they say that the language is ancient, that it is called out of the box, does not support the PLO, that the numbering of the elements in its tables does not begin with 0, as one would expect from any decent language, but with one.
It is said that its disadvantage is that there is no ternary operator in it. In fact, there are quite a few such arguments against Lua, but we will not discuss them, since we believe that they, for the most part, belong to the habits and personal preferences of the developers.
Let's sum up. If your task is to acquire embedded language with minimal effort, take Lua. At the same time, if you have the time and resources to develop your own language or your own bindings - again - pay attention to Lua. Why do we recommend Lua in both the first and second cases?
In the first case, choosing Lua, you will select a language that is very easy to build and use. There is exactly one educational book.on this language, written by its author. There are no other books simply because the first one tells absolutely everything you need to know about Lua, and there's nothing more to talk about it. Lua is not an ideal and not the most common language in the world, but by the sum of the criteria, it is definitely one of the best languages to embed. He is the best we have right now. In addition, there are many standard tools for Lua, which greatly facilitate the lives of those who use it.
In the second case, if you have the resources to develop the tools, you will be able to really spend these resources by choosing Lua, since Lua, despite its popularity in the game development environment, is a very undervalued language. As a result, you will have the opportunity, taking Lua as a basis, to take into account your needs and get exactly what you need.
What are scripting languages and why?
As already mentioned in the previous post of our blog , our company has its own engine. Today we will talk about how we were guided when choosing a scripting language for this engine.
Why is there a need for scripting languages at all? As the saying goes, “there are no difficult problems in the gaming industry, there is no need to demand complex solutions . In addition to experienced (and expensive!) Programmers who solve complex problems, we need people (many people!) Who will deal with, say, quests. And honestly, we would like these programmers to be not so expensive, and ideally not programmers at all, but game designers and scriptwriters themselves.
Thus, there is a need for a tool for describing simple, but nevertheless logic, without involving heavy artillery of programmers. We conclude - what is a scripting language for us? This is a tool that will make game development faster and cheaper.
The question immediately arises, why don't we just use something like XML? The fact is that for our purposes, we often need control constructs — branching and loops, while XML is a declarative description.
Another advantage of scripting languages is that scripts in a project can be both code and a resource. And, accordingly, it is possible to update the script part of the game not only together with the code, that is, during normal updates via the app store mechanisms. But together with resources - that is, along with graphic and other materials, using CDN.
Requirements for an ideal scripting language
We formulate the requirements for an ideal scripting language.
- Dynamic. In our understanding, the ideal scripting language should be dynamic.
- Popularity. By popular language, we mean that it has a large enough community ready to answer questions on specialized resources like StackOverflow.
- A sloping learning curve. We want to take, relatively speaking, practically anyone, and quickly train him to a level that will allow this person to work productively on our tasks.
- Ample opportunities. The language must be powerful and have sufficient capabilities, must support different programming paradigms. A professional programmer who will be offered to write in such a language will be able to do it with comfort and pleasure.
- High performance. Performance is one of the cornerstones of the gaming industry.
- A large number of libraries. Very often we, in the course of solving the tasks before us, do not create a fundamentally new code, but use what has already been written by someone. The more stable, well-supported libraries we can use, using a certain language, the better.
- Easy embedding We are talking about embedded languages, so when choosing a scripting language, the possibility of embedding it plays an important role.
Let us analyze some popular programming languages that are used as scripting languages to meet these requirements.
Python
Python is a dynamic language that is quite popular. It is characterized by a fairly flat learning curve, which is fairly easy to learn. However, it is not so easy to study it properly. As a result, good Python programmers are rare and expensive. This is contrary to our desire to speed up and cheapen the development of game logic.
Python has great features, good performance. His problem is inconsistent library system. Another of his problems, which plays an important role for us, is that it is, in fact, not an embedded language. This is the language from which it is convenient to call libraries written in C or C ++.
Regarding the possibilities for embedding Python, we can say that, for example, there is Maya, where Python is used. But the one who saw inside the Maya plugins written in Python would agree with us that they don't look very good.
As a result, we can say that Python, with all its strengths, does not suit us. Now consider JavaScript.
Javascript
JavaScript is, without exaggeration, a great language that literally captured the world.
JavaScript is a popular dynamic language, featuring a gentle learning curve, with extensive capabilities, good performance, and an extensive set of libraries.
If we need a language interpreter to build the game engine, we can find a lot of such interpreters. In reality, you have to choose from two similar projects - V8 and WebKit. Both are large enough. As a result, if we are talking about board games, it would be possible to take a chance and include the whole interpreter in the game, but in the case of mobile games this option does not suit us.
SocialQuantum has its own JavaScript interpreter, which passes 98% of the tests, we plan to transfer this project to the category of open source.
As a result, it turns out that JavaScript looks like a strong candidate for the role of an embedded language, but it does not suit us either.
Haxe
Here it should be noted that when the conversation about JavaScript comes, the following is usually remembered by Haxe. But, in fact, there is no point in talking about the possibility of using this language as an embedded one, since Haxe, in fact, is not so much a language as a trans compiler into other languages. And this is not what we need.
Maybe we will be satisfied with ActionScript or some other scripting language?
ActionScript
If we formally analyze ActionScript for compliance with the above requirements, it would seem that the ideal scripting language has been found. On its side is the dynamic nature, popularity, ease of learning, good features, performance, availability of libraries, ease of embedding. This language is loved and remembered in the gaming industry; a great number of wonderful Flash games are written on it. The main problem of ActionScript is that this language is almost dead. Therefore, he does not like us either.
AngelScript, Squirrel and others
In addition to ActionScript, there are many scripting languages, such as AngelScript, Squirrel, and others. Among them, one can find such that, formally, almost completely satisfy our requirements, but usually these are languages that are tied to their developer, they have some kind of long-standing problems that have not been fixed for years. Most likely, they are not too popular, they are not well documented, there are not enough teaching materials for them, they have not very large communities. One of the consequences of this state of affairs is the fact that they are difficult to study - if only because it is not completely clear what they are and how they work.
As you can see, we have not yet found the perfect embedded language. What if you create your own language?
Creating your own language
It is possible that the language developed within the company will ideally meet its needs and will be easy to learn. But, most likely, such a language will not become popular. Such a language will either have a minimum number of libraries, or they will not exist at all. In addition, it is hard to believe that in modern conditions it is possible to create something that will work better, that it will have greater productivity and will be easier to integrate than something that is already on the market.
There are companies that develop and use their own languages, among them there are successful players in the gaming market, but, most likely, this is not a good idea.
Having considered the existing programming languages that claim to be embedded and discuss the idea of developing your own language, we turn to Lua.
Lua is the embedded language we have chosen.
Lua is a dynamic language. It is quite popular, there is a large community around it, especially in the field of game development. It has a very gentle learning curve. For example, in our company scripts for autotests are written on Lua. The standard introductory course for autotesters takes about two hours, after which a person is able to use this language. At the same time, Lua is a multi-paradigm language. It supports functional programming style and OOP. As a result, it is suitable not only for solving some simple tasks, but also for more serious tasks that professional programmers are engaged in.
Lua has good performance and has quite a few libraries. Not so much like JavaScript, but, nevertheless, on the LuaForge websiteYou can find almost everything you need. And finally, Lua is very easy to integrate; moreover, it was created to be used as an embeddable language.
For example, this is how our working environment based on JetBrains IDE CLion looks like . Here you can see the auto-completion mechanism we created for Lua, which is planned to be made open-source. Open source and we are going to make a debugger.
Work Environment
We chose Lua, but when it comes to using it as an embedded scripting language, we usually have to deal with roughly the same objections that we will now consider.
Objections to using Lua
Lua is intended for C and not for C ++.
Nobody argues with the fact that Lua is an excellent embedded language. The main thing that they consider it a minus is that it was created for use with the C language, not C ++. Because of this, trying to use something in Lua that is in C ++ and not in C, we run into problems. However, here we must understand that these problems were solved by many pretty smart people. Luabind, Luabridge, toLua ++, SQLuaHost are among the tools that solve the problems of embedding Lua in C ++ projects. This is not a complete list. They have different advantages and disadvantages, but, nevertheless, most likely, all that you may need is already implemented in one of these solutions.
Consider, for example, SQLuaHost. This is a binding that is made inside the company SocialQuantum, and which is planned to be made open-source. This decision represents our vision of how Lua should be bind. Therefore, it is quite possible that if you did not find what you need in existing binders, you will find it in SQLuaHost.
Lua is slow
We often have to deal with the opinion that Lua is a very slow language. Firstly, it is not. Lua is a stack machine, and there, in fact, there is simply nothing to slow down. In addition, we must understand that in the scripting language, we usually give the game logic, business logic, and not some really heavy things. As a result, if Lua-scripts make the game slow down, then the problem may lie in the non-optimal binding or irrational use of some sort of language function. We, for example, conducted synthetic tests on which LuaJIT runs faster than Mono. At the same time, no one bothers to write about such non-optimal code:
function myGame:onUpdate()
local tex = Texture::new(name)
self.background:setTexture(tex)
endHere in each game tick a new texture is created and set as a background. Of course, such a construction will not work very quickly, but no one bothers to write such things here.
Lua is only suitable for small projects.
The next objection is that Lua is made in order to write some small things and something big cannot be written in this language. On the one hand it is true. But this language has high modularity. And from a variety of small blocks, you can make quite large and complex systems. And if you recall what has already been said about multi-paradigm and OOP, then it turns out that OOP pushes the developer to create small modules that can be used to create large and complex structures.
At the same time, some small modules are often written quickly on Lua, and in the gaming industry “faster” means “cheaper”.
Other arguments against Lua
Criticizing Lua, they say that the language is ancient, that it is called out of the box, does not support the PLO, that the numbering of the elements in its tables does not begin with 0, as one would expect from any decent language, but with one.
It is said that its disadvantage is that there is no ternary operator in it. In fact, there are quite a few such arguments against Lua, but we will not discuss them, since we believe that they, for the most part, belong to the habits and personal preferences of the developers.
Results
Let's sum up. If your task is to acquire embedded language with minimal effort, take Lua. At the same time, if you have the time and resources to develop your own language or your own bindings - again - pay attention to Lua. Why do we recommend Lua in both the first and second cases?
In the first case, choosing Lua, you will select a language that is very easy to build and use. There is exactly one educational book.on this language, written by its author. There are no other books simply because the first one tells absolutely everything you need to know about Lua, and there's nothing more to talk about it. Lua is not an ideal and not the most common language in the world, but by the sum of the criteria, it is definitely one of the best languages to embed. He is the best we have right now. In addition, there are many standard tools for Lua, which greatly facilitate the lives of those who use it.
In the second case, if you have the resources to develop the tools, you will be able to really spend these resources by choosing Lua, since Lua, despite its popularity in the game development environment, is a very undervalued language. As a result, you will have the opportunity, taking Lua as a basis, to take into account your needs and get exactly what you need.