Remote control of Fceux emulator using Python
In the article I will describe how to make the NES emulator managed remotely, and the server for remote sending commands to it.
Some emulators of various gaming consoles, including Fceux , allow you to write and run custom scripts on Lua. But Lua is a bad language for writing serious programs. It is rather a language for calling functions written in C. Emulator authors use it only because of the lightness and ease of embedding. Accurate emulation requires a lot of processor resources, and previously the emulation speed was one of the main goals of the authors, and the possibility of scripting actions, if remembered, is far from being the first.
Now the average processor power is enough to emulate NES, why not use powerful scripting languages like emulator or javascript in emulators?
Unfortunately, none of the popular NES emulators have the ability to use these or other languages. I found only a little-known Nintaco project , which is also based on the Fceux core, which is for some reason rewritten in Java. Then I decided to add the ability to write scripts in Python to control the emulator myself.
My result is Proof-of-Concept emulator control capabilities, it does not pretend to speed or reliability, but it works. I did it for myself, but since the question of how to manage the emulator using scripts is quite common , I put the source code on the github .
The Fceux emulator already includes several Lua libraries included in it as compiled code . One of them is LuaSocket . It is poorly documented, but I managed to find an example of working code among the collection of Xkeeper0 scripts . He used sockets to control the emulator via Mirc. Actually, the code that opens the tcp-socket:
This is a low-level socket that receives and sends data by 1 byte.
In the Fceux emulator, the main loop of the Lua script looks like this:
And check data from the socket:
The code is quite simple - reading data from the socket is done, and if the next command was found, then it is parsed and executed. Parsing and execution are organized with the help of Corutin (coroutines) - this is a powerful concept of the Lua language for suspending and continuing code execution.
And one more important thing about Lua-scripting in Fceux - emulation can be temporarily stopped from the script. How to organize the continuation of the execution of Lua-code and re-launch it with a command obtained from the socket? That would be impossible, but there is a poorly documented opportunity to call the Lua code even when emulation is stopped (thanks to feos for bringing it):
Using it, you can stop and continue emulation inside passiveUpdate - this is how you can organize installation of the emulator break-points through a socket.
I use a very simple text based RPC protocol based on JSON. The server serializes the function name and arguments into a JSON string and sends it via a socket. Further, the code execution stops until the emulator responds with the command completion line. The response will contain the fields " FUNCTIONNAME_finished " and the result of the function.
The idea is implemented in the syncCall class :
Using this class, the Fceux emulator Lua methods can be wrapped in Python classes:
And then called verbatim the same way as from Lua:
You can register callbacks in Lua - functions that will be called when a certain condition is met. We can transfer this behavior to the server in Python using the following trick. First, we save the identifier of a function callback written in Python, and pass it to the Lua code:
The Lua code also saves this identifier and registers the usual Lua-Kolbek, which will transfer control to the Python code. Next, a separate thread is created in the Python code, which deals only with checking whether the call command from Lua was accepted:
The last step is that after the Python-kabek is executed, control returns to Lua using the " CALLBACKNAME_finished " command to inform the emulator that the callback is complete.
Everything, you can send commands from the Jupyter laptop in the browser directly to the Fceux emulator.
You can execute all the lines of the example notebook in sequence and observe the result of execution in the emulator.
Full example:
https://github.com/spiiin/fceux_luaserver/blob/master/FceuxPythonServer.py.ipynb
It contains simple functions like reading memory:
More complex examples with creating callbacks:
And a script for a particular game that allows you to move enemies from Super Mario Bros. using the mouse:
Laptop runtime video:
The script is not foolproof and is not optimized for execution speed - it would be better to use the binary RPC protocol instead of the text one and group the messages together, but my implementation does not need to be compiled. The script can switch execution contexts from Lua to Python and back 500-1000 times per second on my laptop. This is enough for almost any application, except for specific cases of pixel-by-pixel or line-by-line debugging of a video processor, but Fceux still does not allow performing such operations from Lua, so it does not matter.
Possible application ideas:
I used:
Fceux - www.fceux.com/web/home.html
This is a classic NES emulator, and most people use it. It has not been updated for a long time, and is not the best in features, but it remains the default emulator for many romhackers. Also, I chose it because the support for Lua sockets is integrated into it, and there is no need to connect it myself.
Json.lua - github.com/spiiin/json.lua
This is a JSON implementation on pure Lua. I chose it because I wanted to make an example that does not require compiling code. But I still had to fork the library, because some of the libraries built into Fceux overloaded the library function tostring and broke serialization (my rejectedpull request to the author of the original library).
Python 3 - www.python.org
Fceux Lua server opens a tcp socket and listens to commands received from it. The server that sends commands to the emulator can be implemented in any language. I chose Python for its “Battery included” philosophy - most modules are included in the standard library (work with sockets and JSON as well). Python also has a library of working with neural networks, and I want to try using them to create bots in NES games.
Jupyter Notebook - jupyter.org
Jupyter Notebook is a very cool environment for interactively executing Python code. With it, you can write and execute commands in a table editor inside the browser. It is also good for creating presentable examples.
Dexpot - www.dexpot.de
I used this virtual desktop manager to pin the emulator window on top of others. This is very useful when deploying a server in full screen for instantly tracking changes in the emulator window. Regular Windows tools do not allow you to organize the fastening of the window on top of others.
Actually, the project repository .
Nintaco - NES emulator in Java with the ability to remotely control the
Xkeeper0 emu-lua collection - a collection of various Lua scripts
Mesen - a modern NES emulator in C # with powerful scripting capabilities on Lua. So far, without the support of sockets and remote control.
CadEditor is my universal level editor project for NES and other platforms, as well as powerful tools for researching games. I use the script and server described in the post in order to explore the games and add them to the editor.
I would appreciate feedback, testing and attempts to use the script.
Why do you need it?
Some emulators of various gaming consoles, including Fceux , allow you to write and run custom scripts on Lua. But Lua is a bad language for writing serious programs. It is rather a language for calling functions written in C. Emulator authors use it only because of the lightness and ease of embedding. Accurate emulation requires a lot of processor resources, and previously the emulation speed was one of the main goals of the authors, and the possibility of scripting actions, if remembered, is far from being the first.
Now the average processor power is enough to emulate NES, why not use powerful scripting languages like emulator or javascript in emulators?
Unfortunately, none of the popular NES emulators have the ability to use these or other languages. I found only a little-known Nintaco project , which is also based on the Fceux core, which is for some reason rewritten in Java. Then I decided to add the ability to write scripts in Python to control the emulator myself.
My result is Proof-of-Concept emulator control capabilities, it does not pretend to speed or reliability, but it works. I did it for myself, but since the question of how to manage the emulator using scripts is quite common , I put the source code on the github .
How does it work
On the emulator side
The Fceux emulator already includes several Lua libraries included in it as compiled code . One of them is LuaSocket . It is poorly documented, but I managed to find an example of working code among the collection of Xkeeper0 scripts . He used sockets to control the emulator via Mirc. Actually, the code that opens the tcp-socket:
functionconnect(address, port, laddress, lport)local sock, err = socket.tcp()
ifnot sock thenreturnnil, err endif laddress thenlocal res, err = sock:bind(laddress, lport, -1)
ifnot res thenreturnnil, err endendlocal res, err = sock:connect(address, port)
ifnot res thenreturnnil, err endreturn sock
end
sock2, err2 = connect("127.0.0.1", 81)
sock2:settimeout(0) --it's our socket objectprint("Connected", sock2, err2)
This is a low-level socket that receives and sends data by 1 byte.
In the Fceux emulator, the main loop of the Lua script looks like this:
functionmain()whiletruedo--вечный цикл
passiveUpdate() --проверка, не пришли ли новые команды через сокет
emu.frameadvance() --передача управления эмулятору для отрисовки следующего кадраendendAnd check data from the socket:
functionpassiveUpdate()local message, err, part = sock2:receive("*all")
ifnot message then
message = part
endif message andstring.len(message)>0then--print(message)local recCommand = json.decode(message)
table.insert(commandsQueue, recCommand)
coroutine.resume(parseCommandCoroutine)
endendThe code is quite simple - reading data from the socket is done, and if the next command was found, then it is parsed and executed. Parsing and execution are organized with the help of Corutin (coroutines) - this is a powerful concept of the Lua language for suspending and continuing code execution.
And one more important thing about Lua-scripting in Fceux - emulation can be temporarily stopped from the script. How to organize the continuation of the execution of Lua-code and re-launch it with a command obtained from the socket? That would be impossible, but there is a poorly documented opportunity to call the Lua code even when emulation is stopped (thanks to feos for bringing it):
gui.register(passiveUpdate) --undocumented. this function will call even if emulator pausedUsing it, you can stop and continue emulation inside passiveUpdate - this is how you can organize installation of the emulator break-points through a socket.
Server side commands
I use a very simple text based RPC protocol based on JSON. The server serializes the function name and arguments into a JSON string and sends it via a socket. Further, the code execution stops until the emulator responds with the command completion line. The response will contain the fields " FUNCTIONNAME_finished " and the result of the function.
The idea is implemented in the syncCall class :
classsyncCall: @classmethoddefwaitUntil(cls, messageName):"""cycle for reading data from socket until needed message was read from it. All other messages will added in message queue"""whileTrue:
cmd = messages.parseMessages(asyncCall.waitAnswer(), [messageName])
#print(cmd)if cmd != None:
if len(cmd)>1:
return cmd[1]
return @classmethoddefcall(cls, *params):"""wrapper for sending [functionName, [param1, param2, ...]] to socket and wait until client return [functionName_finished, [result1,...]] answer"""
sender.send(*params)
funcName = params[0]
return syncCall.waitUntil(funcName + "_finished")
Using this class, the Fceux emulator Lua methods can be wrapped in Python classes:
classemu: @classmethoddefpoweron(cls):return syncCall.call("emu.poweron")
@classmethoddefpause(cls):return syncCall.call("emu.pause")
@classmethoddefunpause(cls):return syncCall.call("emu.unpause")
@classmethoddefmessage(cls, str):return syncCall.call("emu.message", str)
@classmethoddefsoftreset(cls):return syncCall.call("emu.softreset")
@classmethoddefspeedmode(cls, str):return syncCall.call("emu.speedmode", str)
And then called verbatim the same way as from Lua:
#Перезапуск игры:
emu.poweron()
Callback methods
You can register callbacks in Lua - functions that will be called when a certain condition is met. We can transfer this behavior to the server in Python using the following trick. First, we save the identifier of a function callback written in Python, and pass it to the Lua code:
classcallbacks:
functions = {}
callbackList = [
"emu.registerbefore_callback",
"emu.registerafter_callback",
"memory.registerexecute_callback",
"memory.registerwrite_callback",
]
@classmethoddefregisterfunction(cls, func):if func == None:
return0
hfunc = hash(func)
callbacks.functions[hfunc] = func
return hfunc
@classmethod deferror(cls, e):
emu.message("Python error: " + str(e))
@classmethoddefcheckAllCallbacks(cls, cmd):#print("check:", cmd)for callbackName in callbacks.callbackList:
if cmd[0] == callbackName:
hfunc = cmd[1]
#print("hfunc:", hfunc)
func = callbacks.functions.get(hfunc)
#print("func:", func)if func:
try:
func(*cmd[2:]) #skip function name and function hash and save others argumentsexcept Exception as e:
callbacks.error(e)
pass#TODO: thread locking
sender.send(callbackName + "_finished")
The Lua code also saves this identifier and registers the usual Lua-Kolbek, which will transfer control to the Python code. Next, a separate thread is created in the Python code, which deals only with checking whether the call command from Lua was accepted:
defcallbacksThread():
cycle = 0whileTrue:
cycle += 1try:
cmd = messages.parseMessages(asyncCall.waitAnswer(), callbacks.callbackList)
if cmd:
#print("Callback received:", cmd)
callbacks.checkAllCallbacks(cmd)
passexcept socket.timeout:
pass
time.sleep(0.001)
The last step is that after the Python-kabek is executed, control returns to Lua using the " CALLBACKNAME_finished " command to inform the emulator that the callback is complete.
How to run an example
- You must have working Python 3 and Jupyter Notebook in the system. You need to run jupyter with the command
jupyter notebook - Open the FceuxPythonServer.py.ipynb laptop and run the first line
- Now you have to run the Fceux emulator, open the ROM file in it (I use the game Castlevania (U) (PRG0) [!]. Nes in my example) and run a Lua script called fceux_listener.lua . It should connect to the server running on the Jupyter laptop.
These actions can be performed using the command line:fceux.exe -lua fceux_listener.lua "Castlevania (U) (PRG0) [!].nes" - Now switch back to Jupyter Notebook. You should see a message about successful connection with the emulator:
Everything, you can send commands from the Jupyter laptop in the browser directly to the Fceux emulator.
You can execute all the lines of the example notebook in sequence and observe the result of execution in the emulator.
Full example:
https://github.com/spiiin/fceux_luaserver/blob/master/FceuxPythonServer.py.ipynb
It contains simple functions like reading memory:
More complex examples with creating callbacks:
And a script for a particular game that allows you to move enemies from Super Mario Bros. using the mouse:
Laptop runtime video:
Limitations and Applications
The script is not foolproof and is not optimized for execution speed - it would be better to use the binary RPC protocol instead of the text one and group the messages together, but my implementation does not need to be compiled. The script can switch execution contexts from Lua to Python and back 500-1000 times per second on my laptop. This is enough for almost any application, except for specific cases of pixel-by-pixel or line-by-line debugging of a video processor, but Fceux still does not allow performing such operations from Lua, so it does not matter.
Possible application ideas:
- As an example of the implementation of similar control for other emulators and languages
- Study games
- Adding cheats or features to organize TAS passages
- Insert or retrieve data and code into games
- Empowering emulators - writing debuggers, writing scripts and viewing passages, script libraries, game editors
- Network game, game control using mobile devices, remote services, joypad or other control devices, saving and patches in cloud services
- Cross emulation features
- Using libraries of Python or other languages for data analysis and game management (creating bots)
Technology stack
I used:
Fceux - www.fceux.com/web/home.html
This is a classic NES emulator, and most people use it. It has not been updated for a long time, and is not the best in features, but it remains the default emulator for many romhackers. Also, I chose it because the support for Lua sockets is integrated into it, and there is no need to connect it myself.
Json.lua - github.com/spiiin/json.lua
This is a JSON implementation on pure Lua. I chose it because I wanted to make an example that does not require compiling code. But I still had to fork the library, because some of the libraries built into Fceux overloaded the library function tostring and broke serialization (my rejectedpull request to the author of the original library).
Python 3 - www.python.org
Fceux Lua server opens a tcp socket and listens to commands received from it. The server that sends commands to the emulator can be implemented in any language. I chose Python for its “Battery included” philosophy - most modules are included in the standard library (work with sockets and JSON as well). Python also has a library of working with neural networks, and I want to try using them to create bots in NES games.
Jupyter Notebook - jupyter.org
Jupyter Notebook is a very cool environment for interactively executing Python code. With it, you can write and execute commands in a table editor inside the browser. It is also good for creating presentable examples.
Dexpot - www.dexpot.de
I used this virtual desktop manager to pin the emulator window on top of others. This is very useful when deploying a server in full screen for instantly tracking changes in the emulator window. Regular Windows tools do not allow you to organize the fastening of the window on top of others.
Links
Actually, the project repository .
Nintaco - NES emulator in Java with the ability to remotely control the
Xkeeper0 emu-lua collection - a collection of various Lua scripts
Mesen - a modern NES emulator in C # with powerful scripting capabilities on Lua. So far, without the support of sockets and remote control.
CadEditor is my universal level editor project for NES and other platforms, as well as powerful tools for researching games. I use the script and server described in the post in order to explore the games and add them to the editor.
I would appreciate feedback, testing and attempts to use the script.