How we built the A320 flight simulator: the story in pictures (part 1)

    Hello everybody!

    In a previous post, I wrote about the invaluable experience gained in the construction of the Boeing B737NBG.

    Many different events have happened since that post, in particular, our Boeing survived a major overhaul in time pressure mode, and finally I got my hands on writing a post about Airbus A320.

    I will split the post into two - one will be mainly about IT, and the second - mainly about the stages of construction, with slides.

    Go.


    Introductory

    In August 2012, I joined the project that had just begun to build a new flight simulator. At that time, things were like this:
    • A movable platform and a cabin were ordered and made by us.
    • Sheathing was in production
    • The design of the power frame was ending
    • Projectors were purchased and were already in stock.


    The first thing, oddly enough, I set out to draw the IT infrastructure.

    IT infrastructure

    Everything was simple in the head: as a simulator, we have X-Plane 10 with its capabilities to distort the image to a cylindrical projection.

    You need scaling - it means there will be five servers: one calculates the dynamics, three - form an image for three projectors, the fourth - generates a picture for a video recording system for clients.

    Project Magenta components do just about everything else.

    Projectmagenta

    1. Drawing Pictures on Cabin Screens - Airbus Glass Cockpit, ABGC
    2. Airplane Performance Review - pmSystems
    3. Implementation of Flight management and guidance system - MCDU
    4. Autopilot. Implemented in Flight control unit, FCU
    5. Electro-remote aircraft control system, in our opinion, Fly-By-Wire. It also provides an interface with controls - side sticks, pedals, engine control knobs, etc. In addition, it is in this module that “laws” are laid down that prevent rolls that are too large, dangerous pitches, etc. - Flight Control Laws - pmFBW
    6. Sounds - where without them. pmSounds


    It seems to have forgotten nothing.

    System of checks and balances

    Infrastructure design came from a combination of Boeing’s limitations and experience.

    One of the problems from the experience with Boeing - the servers in the rack gather dust, then they start to overheat. So the most loaded computers - and this is X-Plane - should be in the server room.

    Pull USB from the cab - fire, then there must be on-board computers.

    If they are on board a mobile platform, then they are either with an SSD or diskless. The SSD in my memory has overheated twice, so we will learn about diskless loading. All "on-board" software we have under Windows, so it will be iSCSI.

    Download 6 computers over the network - you need bandwidth, which means we must have switches with trunk support. The choice fell on the D-Link DGS-1210-16.

    X-Plane will be several, and the scenarios on them must match, plus iSCSI, backups, etc., so there must be a file server, or rather, a NAS. In terms of price-quality ratio, Synology RS-812 + came up with a memory upgrade of up to 3GB. At the same time, it supports trunks.

    The server room must be clean, that is, the door to it must be closed, which means we maximize remote access, it must be IP KVM (in reality, it came in handy once, we mostly use VNC).

    In addition, she does not smile at all in climbing into the cockpit space where the on-board computers are located, so at least one must be able to turn them on and off. So you need a reinforced concrete working device that allows you to press the "Power" button. It was ordered by a familiar electronics engineer, after a series of iterations we got a device with Eth, which can “press” the power button.
    It looks like this: The


    top board is a “master”, and “slaves” are connected to it via RS-485. Each “slave” has 8 channels, each channel has two discrete inputs (for connecting power LEDs and HDD) and one output for the Power button.

    Our projectors are expensive and rare, you can’t turn them off without preliminary cooling, but if the input voltage has disappeared, then you need to turn off the projectors and on-board computers automatically, without relying on people. So we need UPSs with a network interface, projectors have it anyway, we just need to use it.

    In general, as a result of a couple of weeks of designing, we got such a scheme:


    On-board computers on our Celeron G540 2.5 GHz with 4 GB of memory and a Low profile video card on a Radeon HD 6450 chip with a gigabyte of memory.

    Graphics servers for X-Plane - Core i5-3470 with 8GB RAM and two 500GB drives in the mirror. For drawing pictures are GeForce GTX 670 c 2048MB. As an OS, we have Ubuntu 12.04 LTS 86_64, now the X-Plane is 64-bit.

    The server that calculates the dynamics - under Win7_32 on the Core i7-3770, uses the integrated video card. X-Plane is 32-bit here, and the server is under Windows because the plug-in for the interface with the mobile platform is only for 32-bit and for Windows.

    Cables and wires

    The super goal was to build a simulator so that only twisted pairs and power cables entered the cab. And so it happened in the end.
    A 220V power cable, two twisted pairs for connecting switches (trunk), three twisted pairs for HDMI extension cords, and another twisted pair for the ladder and platform control panel go to the cabin from the outside.

    Extenders I used Hama HDMI, they cope with their task.

    Twisted pair everywhere - CAT6 FTP.

    Separately, it would be worth writing how we decided to contrive and stretch our twisted pairs in an existing pipe between the cabinet and the movable platform. It was a stupid undertaking, even with the use of cable grease, besides, later on I still extended two more pipes.

    Electricity

    In Boeing, we were faced with the problem of an incorrect electrical network, so this time I sat down to draw a single-line diagram (I can not find it).
    The idea was that we protect all lines with automatic machines, plus we protect people from electric shock.

    Load calculations showed that 3000VA is enough for the entire cab (projectors, computers, input-output controllers, on-board ventilation, switch).
    On 5 servers, NAS, a switch and KVM also need about the same.

    The external cab fans, the instructor station and the ramp drive need less than 1500VA, so the old 1500 SmartUPS came in handy.Total
    UPSs cost four - two Ippon Smart Winner 3000 VA with an additional battery pack, and two 1500VA SmartUPS left over from one of the iterations of Boeing's improvement.
    A server room hangs on one three-kilowatt, on the other - the whole cabin. On one 1500 hangs the ventilation of the cabin and the instructor station, well, and one more - in stock.

    Given the availability of autonomous energy sources (UPSs), RCDs to protect people are after the UPS, i.e. Uninterruptibles are “tied” to our shield.



    Ventilation and air conditioning

    Our server room is very small, about 5 square meters, and 2 kW of heat from the rack and, probably, 1 kW from the platform control cabinet are allocated there. So we will hang the condo.

    Conde was chosen Samsung AQ12TSBN, it has a cold performance of about 3.5kW. I did the installation myself, the benefit of tools and equipment was purchased before that and was used for the installation of kondey in the country.
    Summer showed the right choice, the temperature in the server room fluctuated around the set 19 degrees.

    It was a stupid mistake to hang the indoor unit over an electric panel, so I had to pay special attention to drainage. Pah-pah, there were no excesses, but by next summer I will install a drainage pump, for sure.

    In a cabin with ventilation, this is not so simple.
    We have two volumes - the cockpit itself, in which people sit, and the cockpit space, in which there are irons, projectors and computers.
    From the very beginning, we decided that we would not do air conditioning - there are too many problems with condensate.

    In the cockpit space, inflow and exhaust were made by low-noise fans with a capacity of about 250 cubic meters per hour. The inflow was made to the place where the computers were installed. An


    extractor fan - near the projectors, at the top point (sorry for the quality of the photo)


    . Actually, it would not hurt to stick a fan into the hole in the “roof”, but temperature measurements do not show such a need.

    To circulate the air in the cabin, there is a large 2000 cc blower fan outside, and the hood is made by two successively standing silent blocks from under the feet:


    Initially, according to calculations, I made a mistake with the choice of fans, a 2000 cc supercharger appeared later, at first there were 250 cc silent pipes.
    Even with a 2000 fan in the cabin, it’s hot, so we’re thinking about how you can further improve the system.

    Crutches, crutches and crutches

    From the very beginning it was clear that even “shelf” products would need to be wrapped in numerous crutches.

    There were several problems for osteosynthesis:
    1. X-Plane Shared Data Sync
    2. Projector on / off automation
    3. Restarting failed Project Magenta components


    X-Plane: Sync and Launch

    To synchronize X-Plane data, rsync was finally selected from NAS.
    I tried to run directly from NFS balls, but considering that we have both Windows and Linux, and also the X-Plane directory structure, in my opinion, it was invented in a drunken delirium, I decided that each configuration has its own, and the scripts and general data synchronize over the network every time X-Plane starts or restarts.

    By the way, as I worked through the idea of ​​launching X-Plane directly from the balls, I went so far as to edit the binary so that all the files opened for writing were in the same directory. Here is a picture from this stage:

    Everything was successfully completed, and two days later an update came out. After that, I decided to do rsync.

    The startup script turned out like this:
    #!/bin/bash
    LOGFILE="/var/log/xplane-starter.log"
    log_msg () {
        DATE=`/bin/date`
        /bin/echo "$DATE: $@" >> $LOGFILE
    }
    log_msg "Starting xplane-startup.sh"
    while [ 1 ]; do 
     log_msg "Rsyncing"
     /root/bin/rsync_64.get >> $LOGFILE
     /bin/sed -i".bak" '/UNSAFE/d' "/xplane.64/Output/preferences/X-Plane Screen Res.prf"
      /usr/bin/pactl set-card-profile 'alsa_card.pci-0000_01_00.1' 'output:hdmi-stereo-extra1'
     /usr/bin/xinit  /root/bin/xinitrc.64 >> $LOGFILE 2>&1 &
     XINIT_PID=$!
     log_msg "xinit pid = $XINIT_PID"
     while [ 1 ]; do
      sleep 1
      # Check that the process is still alive
      if [ -e /proc/${XINIT_PID} -a /proc/${XINIT_PID}/exe ]; then
       sleep 1
      else
       log_msg "Process dead, restarting" 
       break
      fi  
     done
    done
    


    Since X-Plane absolutely does not need a window manager, the actual launch is done through xinit:
    /usr/bin/xsetroot -cursor_name top_left_arrow
    DISPLAY=:0.0 /usr/bin/X11/xset s 0
    DISPLAY=:0.0 /usr/bin/X11/xset s noblank
    /root/bin/shm_wipe.sh
    /usr/bin/x11vnc -display :0 -ncache 10 -many &
    umask 0000
    /xplane.64/X-Plane-x86_64 --no_crash_reporter
    /usr/bin/killall -9 x11vnc
    


    shm_wipe.sh is my laziness. Periodically, x11vnc ceases to start due to exhaustion of shared memory, so a recipe for searching and “untying” unused segments was found on the Internet.

    Additionally, through cron, X-Plane is rebooted in the early morning, because there are memory leaks in it. In addition, at night, just in case, a command is given to turn off the projectors.

    On Windows, not everything is so kosher, I do rsync synchronization periodically with my hands ...

    On-board computer monitoring

    To deal with the hanging components of ProjectMagenta, an entire PowerShell script was written on the on-board computer, and even with .NET elements.

    The script began with the detection of failed USB devices, and then it was already covered by everything else.

    Components are launched through this script to the heap, and checks to see if anyone has dropped out during the course of the play.

    I’ll tell you about EHID in a separate line.
    EHID is the interface between cab hardware (I / O controllers) and software components. Distributed under the NDA, I signed it and now I am the proud owner of the specification. I have applied it so far for the gasification of a bug with a tiller in X-Plane, which I will discuss below.
    The bottom line is that there is a component A320_EHID.exe (and accordingly B737_EHID.exe for a Boeing), which contains a "tree" of all all components of the cockpit - and the analog axes, and LEDs, and toggle switches and everything in general.
    Each item has its own name and 32-bit ID.

    Applications do not access hardware directly, but EHID. Communication - over TCP, Event-driven. Polling is all implemented in EHID, so this concept greatly simplifies life.

    The specification covers two levels - between hardware and EHID, and between EHID and applications.
    The lower level is presented including a description of the protocol on top of the USB HID, and is implemented also in the controller firmware.

    Update: painted a picture of the interaction of the components of the simulator with each other


    The trouble is that the “last mistake” was found at the interface between the Windows drivers and the EHID implementation in the firmware: when data from a large number of devices simultaneously arrives, something so magical happens inside the EHID, which leads to DoS. It’s easy to find out, thank God: EHID stops responding to application requests. This is what monitoring is based on.

    Govnokod - that one, do not kick.
    Import-Module c:\scripts\Release\DeviceManagement.psd1 
    [int] $Port = 21843
    $IP = "192.168.X.X"
    $Address = [system.net.IPAddress]::Parse($IP) 
    $LogDirs = @("C:\EHID","C:\MCDU","C:\RMCDU","C:\FBW","C:\pmSystems","C:\FCU","C:\xpuipc")
    Write-Host "Starting the supervisor script..."
    # Check if we just started
    $p = Get-Process -Name xpwideclient -ErrorAction silentlycontinue
    if (!($p))
     {
      Write-Host "Just started"
      # Find and remove logfiles
      foreach($logdir in $LogDirs)
       {
        del $logdir\*.log -ErrorAction SilentlyContinue
       }
      del R:\*.tmp  -ErrorAction SilentlyContinue
      # Set the routing for the goddamn windows shitty default gw to iSCSI target
      route delete 0.0.0.0
      route add 0.0.0.0 mask 0.0.0.0 192.168.X.X
      # Start XPUIPC
      Start-Process C:\xpuipc\xpwideclient.exe  -WorkingDirectory "C:\xpuipc" -WindowStyle Minimized
      Start-Process C:\EHID\A320_EHID.exe -ArgumentList "-run" -WorkingDirectory "C:\EHID" -WindowStyle Minimized
      Start-Sleep -Seconds 3
     }
    else
     {
      Write-Host "Already working"
      $startFlag=0
     }
    # Cycle forever
    while ( 2 -ge 1)
    {
     # Clear the flag
     $failFlag = 0
     # Get the list of the failed devices 
     $FailedDevices = Get-WmiObject Win32_USBControllerDevice |%{[wmi]($_.Dependent)} | Where-Object {$_.Status -ne "OK"}
     foreach($fDevice in $FailedDevices)
      {
       # Set the flag to restart the processes
       $failFlag = 1
       # Disable and then enable device
       $dDev = Get-Device | Where-Object {$_.InstanceID -eq $fDevice.DeviceID}
       Write-Warning "Failed device found:" $fDevice.DeviceID
       Disable-Device -TargetDevice $dDev -Verbose
       Enable-Device -TargetDevice $dDev -Verbose  
      }
     # Detect hang of A320_EHID 
     # Create IP Endpoint 
     $End = New-Object System.Net.IPEndPoint $address, $port 
     # Create Socket 
     $Saddrf = [System.Net.Sockets.AddressFamily]::InterNetwork 
     $Stype = [System.Net.Sockets.SocketType]::Stream 
     $Ptype = [System.Net.Sockets.ProtocolType]::TCP 
     # Create byte array  
     [Byte[]] $Message = 0x02,0x03,0x00,0x04
     $buffer = new-object System.Byte[] 8192    
     Try 
      {
       $Sock = New-Object System.Net.Sockets.Socket $saddrf, $stype, $ptype
       $Sock.TTL = 26 
       $Sock.ReceiveBufferSize = 8192;
       $Sock.ReceiveTimeout = 500;
       $Sock.Blocking = 1;
       # Connect to socket 
       $Sock.Connect($end)
       $Sent = $Sock.Send($Message)    
       Start-Sleep -m 30  
       $Received = $Sock.Receive($buffer)  
       $Sock.Close()
      } 
     Catch [Exception]
      {
       Write-Host "Error checking EHID"  $_.Exception.Message;
       $failFlag=1;
      }
     # Detect restart file-flag
     $FileExists = (Test-Path "R:\restart.txt" -PathType Leaf)
     if ($FileExists)
      {
       Write-Host "Restart file exists!"
       del R:\restart.txt
       $failFlag = 1
      }
     # Restart the processes
     if ($failFlag -eq 1) 
      {
       Write-Warning "Restarting processes"
       Write-Warning "Freezing..."
       Start-Process C:\scripts\sim_pause.exe -WorkingDirectory "C:\scripts" -WindowStyle Minimized
       Stop-Process -Name A320_EHID -force -ErrorAction silentlycontinue
       Stop-Process -Name pmFBW -force -ErrorAction SilentlyContinue
       Stop-Process -Name pmSystems -force -ErrorAction SilentlyContinue
       Stop-Process -Name MCDU -force -ErrorAction SilentlyContinue
       Stop-Process -Name RMCDU -force -ErrorAction SilentlyContinue
       Stop-Process -Name FCU -force -ErrorAction SilentlyContinue
       # Set start flag
       $startFlag=1
      }
     if ($startFlag -eq 1)
      { 
       Start-Process C:\EHID\A320_EHID.exe -ArgumentList "-run" -WorkingDirectory "C:\EHID" -WindowStyle Minimized
       Start-Sleep -Seconds 3
      }
      # FIXME Workaround for the falling PM processes
     $p = Get-Process -Name RMCDU -ErrorAction silentlycontinue 
     if (!($p))
      {
       Write-Host "RMCDU dead, restarting."
       Start-Process C:\RMCDU\RMCDU.exe -WorkingDirectory "C:\RMCDU" -WindowStyle Minimized
      }
     $p = Get-Process -Name FCU -ErrorAction silentlycontinue 
     if (!($p))
      {
       Write-Host "FCU dead, restarting."
       Start-Process C:\FCU\FCU.exe -WorkingDirectory "C:\FCU"
      }    
     $p = Get-Process -Name MCDU -ErrorAction silentlycontinue 
     if (!($p))
      {
       Write-Host "MCDU dead, restarting."
       Start-Process C:\MCDU\MCDU.exe -WorkingDirectory "C:\MCDU"
      }
     $p = Get-Process -Name pmFBW -ErrorAction silentlycontinue
     if (!($p))
      {
       Write-Host "pmFBW dead, restarting."
       Start-Process C:\FBW\pmFBW.exe -WorkingDirectory "C:\FBW"
      }
     $p = Get-Process -Name pmSystems -ErrorAction silentlycontinue
     if (!($p))
      {
       Write-Host "pmSystems dead, restarting."
       Start-Process C:\pmSystems\pmsystems.exe -WorkingDirectory "C:\pmSystems"
      }  
     if ($startFlag -eq 1)
      { 
       # Clear the flag and start the processes
       $startFlag = 0   
       Start-Sleep -Seconds 45	
       Write-Warning "Un-Freezing..."
       Start-Process C:\scripts\sim_unpause.exe -WorkingDirectory "C:\scripts" -WindowStyle Minimized
      }
     Start-Sleep -Seconds 5	
    }
    


    X-Plane, Tiller and Crutch

    Tiller is the front pillar head control handle. Needed for taxiing on the ground.
    You can steer in two ways - tiller and pedals. Upon reaching a certain speed, the tiller ceases to act, only the pedals remain.

    The problem is that in X-Plane (as acknowledged by its author) they “lost” the dataref, i.e. an internal variable where the tiller positions could be written.
    In addition, if a joystick is not connected to the X-Plane, then he believes that we are miserable, poor simmers who fly on the keyboard, and therefore we need to do a front desk turn on commands from the side-steering wheel.

    I had a very stormy correspondence on this topic with the author of X-Plane Austin Meyer (during which he showed himself to be a hysterical goat, sorry moi), who promised to fix this joint in one of the releases. We wait.

    While we wait, I wrote a crutch, which, on the one hand, clings to a virtual joystick (VJoy), and on the other, through a network and EHID takes the value of the position of the tiller handle.
    In X-Plane, I assigned a single axis - this same tiller, and, in the eyes of X-Plane, we ceased to be rogues, and therefore the side-turning functionality of the sidestick was disabled.
    There is nothing complicated or unusual in this crutch, so I see no reason to bring the source. Written in an hour on .NET in my opinion.

    Conclusion

    I’ll probably dwell on this, and so the post turned out to be very thick.
    In the next part there will be slides and the history of the construction of Airbas.
    For starters - a photo from this stage:


    Thank you for reading!

    Also popular now: