* * *

Translations for our friends from around the world...and beyond.

Click on banner for Tacview website


Author Topic: Tacview Server Side Recording Explained  (Read 45 times)

0 Members and 1 Guest are viewing this topic.

Offline Vyrtuoz

  • Moderator
  • *****
  • Posts: 36
  • Tacview Developer
    • Tacview The Universal Flight Analysis Tool
Tacview Server Side Recording Explained
« on: March 17, 2017, 02:50:41 PM »
I think that Dojo is explaining well how the new DCS World exporter features can be used in a multiplayer environment. Original post: https://forums.eagle.ru/showthread.php?t=183914

From Dojo (2017-03-01)

Hey folks, given that we were directly involved in the design and testing of the server side recording, I thought I'd offer an expanded explanation for reference, in the hopes that it will be useful to you.  Below I'll describe the feature, how it works, the performance implications (both client and server), and other tidbits.  Feel free to skip the origin, it's just background for future reference.

A little over a year ago, we noticed that Tacview was having a severe impact on frame rate in DCS.  The reduction in expected frame rate was anywhere from 40% to 130% within a given mission.  Yeah, that bad.  We use Tacview virtually after every mission to debrief, so turning if off was a non starter.

We provided the feedback to Vyrtuoz, who began to troubleshoot with us.  After evaluation, it was demonstrated that the size of our mission (specifically the number of objects), was directly related to the impact of Tacview.  If you're not aware, the 476th Nevada Training mission that is public, has thousands of objects in it, in order to reproduce the real world ranges on the DCS Nevada map.  In other words, it really stresses the export handling, although there are many missions which do the same. 

Proving the relationship between object count and increased performance impact with Tacview is easy:  Launch an empty mission (one plane on a runway at Nellis) on a given machine, with Tacview turned off, and look at the frame rate.  Let's say that was ~130FPS with VSync disabled on a given machine.  Launch the same empty mission with Tacview turned on, again 130FPS.  No difference as there's really nothing to export.

Now run the 476th NTTR mission with no Tacview.  Results: 70FPS.  So a substantial drop due to the object count, but still highly playable.  Next, re-launch that mission with Tacview on: Results ~35FPS.  So we have a significant problem.

Of course, our initial thought was "Hey Vyrtuoz, can't you just fix it please?".  After analysis on his side, he concluded that he could optimize his own export handling, but that much of the problem lied within the way ED handles the exports, so he could only do so much to address it, and he has.  In the above example, that machine's frame rate with Tacview is now ~45FPS after Tacview's optimization.  A substantial uptick (~29%), but far from ideal.

As I understand it, the inefficiencies in how DCS handles object exports has been reported to ED before, but thus far, optimizing the exporter hasn't been prioritized.

In summary, while the argument can be made "You only receive a performance loss with Tacview on, so isn't it Tacview's fault?", the real issue is that enabling Tacview is exposing inefficiencies in the DCS' export handling.  We have log files that demonstrate where the processing fault lies per frame (and by definition the frame rate loss), but for now, take my word for it.

So how do we get the benefits of Tacview with none of the frame rate loss?

Recording on the server

Tacview has always been able to record on any machine running it (client or server), so technically, you've always had the ability to record on the server instead of or in addition to the client.  However, this creates two problems:  If you were only running a single mission with everyone involved in that mission, for a reasonable amount of time (let's say 5-6 hours max), this isn't so bad.  However, if you ran a massive scenario, where players could join and disconnect at anytime, and after they disconnect they want to review the Tacview, well now there's isn't an elegant way to handle it.  Additionally, in previous versions of Tacview, compression wasn't built in, and a Tacview file would be literally hundreds of megabytes, making transfer a "process" to say the least.

The real issue for us, and many other multiplayer servers, is that the server is up for much longer than 5 to 6 hours, and as a pilot, I'm typically only interested in my flight, not the entire scenario. 

This is explicitly the case when a server such as ours is used:  It's a training environment, which means it's up all day, and pilots can join and leave any time.  Pilot 1 and Pilot 2 may join the server and go to one area of the NTTR to practice bombing.  Pilot 3 and 4 may join the server an hour later, and fly to the tanker, while pilot 5 and 6 can join later still and practice formation flying.  When they're done, the server is still running waiting for others to join and disconnect.

Why should any of them have to review a tacview file that might be a few days old, searching for their flight time?  Yes, they can run Tacview on their own machine, but considering the massive performance drop, that's not desired.

"Per Client" Recording
Enter the design proposal for server side user recording.  The idea is simple: Install Tacview on a server, and have it create a separate Tacview (ACMI) file per connected user.  Start recording when the user connects, stop recording when the user disconnects, and place the ACMI file in a folder with the pilot's name.

This means that a long running server, can have users connecting and disconnecting all day long, with each user getting complete ACMI files that just includes their own flight window.

When the server is put in this mode (called "Create one file for each client session" in the DCS Special options) it will not create an additional ACMI file for itself, just for the individual connected users.

Client Performance Benefits
This one's obvious, because Tacview doesn't need to run on the client anymore, there is zero impact from Tacview on the client's performance.  Calling back to my previous example, that machine runs at 70FPS plus now for the heavy mission.

Server Performance Implications
The story here is great too.  I'll spare you the gory test details, and cut to the executive summary: We conducted a lot of testing in Alpha, and a few Beta tests with 10+ pilots to evaluate this.  It should be noted that there is no distinct Tacview process running for recording, so Tacview's entire performance overhead  is captured within the DCS process itself, and has no noticeable impact whatsoever from a CPU or RAM perspective.

So of course the big question becomes disk performance, and the news here is great:

A rough summary: In multiple tests on our heavy mission with 10+ pilots, DCS disk write rate was ~9KB/s (note this includes Track data and Tacview), with a 450KB spike every ~5 seconds as Tacview empties its memory buffer to disk during our test scenario.  While that spike is below .5/MB, and well within the performance parameters of even 15 year old mechanical disks, the small block write size would likely stress them, and I wouldn't recommend anything less than a 4-5 year old SSD.

The performance impact scaled linearly in tests, so extrapolating our results to a 60 pilot connected server (the most I've personally seen), you're looking at ~3MB spikes at the buffer dump interval. 

Certainly a stress test would be prudent to prove my assertions above; I was unable (and unwilling ;) ) to coordinate a 50+ pilot test myself.

There is the question of network upload speed, but I'll come back to this at the end of the next session, as it's optional, depending on how you plan on "distributing" the server side ACMIs.

How did you set it up?
Our config is as follows: We enabled the server side client recording option, and advised our members that they can stop running Tacview locally.  We then used Google drive to create a share folder, and configured Tacview to save recordings to that folder.

The Google share was then made available to all of our members (they can read/download, not delete).  After a pilot is done flying and disconnects from the server, Google, within a few seconds, syncs the share.  Each pilot can then easily download their own Tacview and review it on their desktop.

For maintenance, we wrote a very basic server script, that will delete the ACMI files in the pilot folders after 24 hours.

**It's worth noting that the ACMI file production has been extremely optimized.  On average, a 3 hour mission for us produces a 4MB ACMI file size per pilot, so the transfer and sync are really fast.

Here's where the additional performance consideration is the Internet connection;  If you use Google Drive, Dropbox or any other network sync mechanism, this of course requires the server to constantly upload the ACMI files as they're being dumped to disk while Tacview is running.  Make sure your upload link can support regular Google Drive/Dropbox transfers if you go this route.

We're plenty happy, although we currently have a 100Mbit symmetrical link.  I've also done this with plenty of room to spare on a 25Mbit upload link during the test phase.
Additional Pros
  • Tacview will produce an ACMI file for a pilot whether he/she has Tacview installed or not, allowing admins to review pilot flights, even if the pilot isn't interested.
  • Local client recording can be disabled on the server, allowing the server to create an ACMI for the pilot, but the pilot won't be able to generate his own, even if he tries to record Tacview locally.  We don't do this in my group as we don't have cheating concerns internally, but public servers may use this as an alternative solution to the anti-cheat feature, because a pilot can be provided his server generated ACMI file at any future time period. (i.e. after the scenario completes)

Are there any cons?
Well, yes.  It would be accurate to say "you get most of the benefits of Tacview with none of the performance loss."  You don't get all of the benefits, because a remote host can not export "cockpit" data for a connected client.

For example, if a client runs Tacview locally, he will get the following additional data in Tacview:
  • Gear State
  • Flap State
  • Air Brakes State
  • Magnetic Heading (vs True)
  • *Accurate Indicated Air Speed (IAS will show, but it won't be accurate on a remote recording)

Since a server (or any remote host) can't export the above, the ACMI file will not be able to provide the data.  It's certainly my personal opinion that those are minor sacrifices for the huge performance gain.
« Last Edit: March 17, 2017, 02:54:16 PM by Vyrtuoz »


Official Tacview news feed