Audience Q&A
How does the FRR system determine how large a penalty should be assessed for collisions? How does it determine who's at fault, and what problems did you have to deal with while developing the FRR system?
(asked by SoraS1dewinder)
T10 OG: I'm going to start with the last question first actually. There were a lot of problems when we first started to develop an FRR system at the end of the support for FM7, and we implemented the first iteration of FRR. It was built by writing this scripting system that allowed our designers to write some simple rules to determine who is at fault. Our thought was that we could use this system to modify those rules through scripts and update the game without having to update the game code: without requiring an entire title update. So that's a great goal. But the scripting system wasn’t powerful or flexible enough to get it right. You could make one rule would be correct for some scenarios, and then it would get it wrong for other scenarios. We weren't anywhere near the accuracy that we're at now. So we knew that we had to attack the problem differently for our current title. It's a super difficult problem to solve, and we know that we still have more work to get it to where we're happy with it.
OK, so T10 LoungeToy sort of talked about this, but I think it's OK if we repeat ourselves a little bit. So how does the FRR system determine how large of a penalty should be addressed for a given collision? So the biggest factor with that is the size of the impact, which is determined by the relative velocity of the cars hitting each other. Then we have a couple modifiers on top of that. One is that if the car that you hit loses race position due to the collision, we'll modify the penalty based off of that. And the other is, if you're going over the optimal speed for your car at that position on the track when the collision occurred, if you're going way too fast into a corner or something like that, we modify it for that as well.
For the next release coming out in October, we're making a significant update to that, to how the system determines how big of a penalty you should get. And we're going to have our neural network also calculate this penalty time. I'll talk a little bit more about neural network in a little bit. And so we're going to have two systems determine the penalty time. And for the first iteration, we're just simply going to take the lesser of those two.
Last part, how does FRR determine who's at fault? So we use a neural network solution. I'm not going to get super deep into what a neural network is here today. In fact, you know, I'm not the expert on the team either. The expert happens to be sitting right next to me, being support on this Discord call for us. But it’s a neural networks machine learning technique. It's built to take a bunch of data as inputs that we all label, that T10 LoungeToy was talking about. And if properly trained, it can build these layers of nodes that recognize patterns and make smart decisions.
There's another question in a little bit where I'm going to get into one more layer of detail of the neural net and everything. But basically, you know, it's really cool tech allows the system to get smarter when given enough data. It's a little new to most of game development as well, right? It's this super fast evolving area in machine learning.
Will there be any change to how off track penalties are given or approached? I'd just like to add that they've gotten a lot better over time, but they're still very tedious to the point where some off-track penalties result in one or more seconds added when you don't gain that much time for slipping a tire off.
(asked by Ash)
T10 OG: That’s a really good question. Let me start with just how the system currently works, which is a little bit of a repeat again, but that's OK. For the off-track penalties specifically, we wanted to build a system that only gave penalties if you went off-track and gained a time advantage by doing so. If you go off-track and don't gain an advantage in time, we won't penalize you.
The way that we figure this out is, we use our driving AI system, and we look at the ideal racing line and the speed for your particular car, your particular upgrades, your setup. And so basically, if you go faster for that segment of the track where you're off-track, and our top-level AI thinks that it can’t, then we penalize you the amount of time that you gained plus a small scalar on top of that. So I'm pretty sure that [the player who asked this question] is pretty fast if they’re getting regular penalties when going off-track, because most people actually don't, because they're not going faster than our fastest AI. This is definitely something that we're aware of. It's becoming more important with the community, and specifically with some of our fastest drivers and people that play the game the most.
So it's important. I do, you know, I think we're going to need to make some modifications to the system. I can't say exactly when yet, but we do recognize this is an issue and needs work for some of our fastest drivers. So, yeah, thanks for that comment.
T10 LoungeToy: Yes, and, really quickly, we're definitely aware of some of the deficiencies that we have with the speed computer and how then it affects our off-track penalties. We are heavily, heavily focused on getting the collisions portion of FRR right first. So our priority right now is making sure that all the collisions are good to go and then we'll start addressing things like our off-track, just so you understand kind of where we are with our priorities.
What is your favorite part about what you do in your role at Turn 10 Studios?
(asked by PTG Jamie)
T10 LoungeToy: I've worked at a lot of different studios, made a lot of different games, and the greatest thing about this studio, Turn 10, is that I can show up every day and geek out and talk about cars and racing. There's no other studio that I've been able to do that at.
As a great example, I just blew an S54 engine and that's going to cost me a lot of money. It was a hard summer of racing and at the end of it, I was giving some people joy rides and I think we overdid a little bit, and that's going to cost me. There are a number of people at the studio who have not only worked on deep in the guts of the S54 engine, but raced it very hard over the years. So I immediately was able to come in and talk to all these experts on “hey, so what to do? What are the signs of this? What are my options here?” That incredible knowledge base of, like, where else in the world are you able to, outside of a mechanic’s garage, show up to work and say, “hey, I blew my S54? And how would you think about rebuilding it?” So again, I love thinking, I love talking, I love geeking out about cars and racing and there's no better place to do it than here.
T10 OG: OK, so kind of a two-part answer for me. First, for me, is just the people I get to work with. The people that I get to work with every day, you know, Turn 10, we have this ever evolving team: super smart and passionate game developers who enjoy coming into work, I'll say most days, maybe not every day, but most days! And a lot of people can't say that they like going into work most days. So that's pretty cool. Some of these people I've worked with for over 20 years, and they've become lifelong friends, practically family.
Second, for me, is just getting to work on really cool, challenging problems. In particular, in the simulation space, we have things like physics and FRR, AI, steering wheels, things like that. Things that I particularly enjoy. I learned how to program when I was about 10 years old and I've always loved it. You know, I'd make my friends these simple little games in Q Basic, which I know I'm dating myself by saying that, but oh well. It was always such a rush to have people play something that I made, and get joy out of it. And now I get to do the same thing and get paid for it. So yeah, not too shabby.
As someone who does machine learning research mostly applied to biology and aerospace, I'd love to maybe get an overview or understanding of the technical aspects behind FRR and its architecture. I've always been curious how these sorts of systems are deployed in a game setting and would love to know more.
(asked by PJK)
T10 OG: Machine learning in biology and aerospace sounds pretty cool! We’ll need to get a bit technical here. First off, we talked about the fact that we're using a neural network machine learning technique built to take a bunch of data, and if properly trained, can build layers of nodes, make smart decisions, etc. So, you know, that's kind of step one of the architecture of its neural network.
Let's talk about some of the inputs into the system. The inputs into the system is all the data that we think like a trained human brain, or like a race steward, would need to determine who's at fault in the collision and what types of penalties to hand out. So we capture roughly 7 seconds before and 7 seconds after the the first collision. We capture this data at 12 Hertz, so like 12 times a second.
We capture things like the position of your car and the nearest cars around you. We capture the car's orientation and how they're oriented compared to the direction of the track. We capture the speed of the cars, the optimal speed they should be going on that part of the track. The inputs of the car, are they braking? Are they on throttle? Are they steering into the corner or away from it? Are they steering into the collision or away from it? So all these things and then some: the points of the collision relative to the cars, the size of the impact.
OK, so how do we train the neural network? In the presentation we showed the clip of our FRR training tool. We capture all these inputs from the collisions in the retail version of the game. We store those on a server, and we have the ability to look at those in different types of collisions. So we have that simple visualizer tool that shows the cars on track from an overhead view. We allow the user to scrub forward, backward, look at the incident from all angles, and then label who's at fault.
We have tens of thousands of incidents labeled so far, soon to be hundreds of thousands. This is fed into this neural network training model. And once it has all the training data, it processes it, which does take many hours, and it creates that neural net model.
The last part is, once we have that model trained, how does it run? So, in-game we track your car and all of those inputs. When a car collides with another car, we have that data for the 7 seconds before and after. We pack all that data into the input vector, and that gets fed into the neural network runtime code. We actually run the code locally, on the box that you're playing on. We don't send it to a server to run, or anything like that.
So one of the tricky things for us is, our game is already pushing the CPU cores to their limits when the game is running with 24 cars and, and it does take a decent amount of CPU to process one particular incident. And so we decided to process the incident over the course of a few seconds, about 100 or 200 game frames. We run it on a lower priority thread on the CPU, which allows our gameplay core, our rendering core, audio core, and all the other code that's just currently running all the time to remain running at speed. Since you don't need an immediate result for FRR, we just allowed it to run a little bit slower over that time and get the result.
Would it be possible to introduce a strike system, and/or have you considered other systems beyond just the FRR system currently in place?
(asked by Artiick)
T10 LoungeToy: Have we considered other ways of doing a penalty system or adjudication system? Absolutely, yeah, absolutely we have. We've definitely looked at things like iRacing’s solution, and discussed at length about how we might do a similar thing, or how we might learn from that. I think of it as like, an insurance model or something like that: the strike system is a very similar kind of idea, suggesting maybe we move away from an at-fault determination type of system and that sort of thing. All really interesting points of discussion, and we can definitely think about how we might, in the future, if things don't work out with FRR as it currently is with our machine learning model, maybe we go down a different path.
What I can say is, right now we're not really considering a different solution for our adjudication, not yet. We're all in on FRR with the machine learning model, and I think we have all the right tools in place now and the right investments around it to get to where we need it to be. So that's what I'll say around all the different models. But right now, I think we have a really good chance of making FRR awesome.
Are there plans to add more telemetry data to the data-out feature? We are developing an app for Forza Motorsport, and it'll be great to be able to collect more telemetry data such as telemetry from other drivers, the gap between drivers, the type and the type of tires.
(asked by Anotherjez)
T10 LoungeToy: We're always looking at ways of adding data and making these third-party experiences just really awesome. I've seen some really good HUD elements out there based off the telemetry and the data-out stuff, that looks really good, and it’s inspired us as well. It's really just a matter of priorities and making us aware of what you're looking for and what kind of data-out you're looking for. Please add it to our Suggestions Hub.
Are we likely to see AI in private multiplayer?
(asked by Jess)
T10 LoungeToy: So even though my team works on aspects of AI, it doesn't necessarily cover how we talk about AI in private multiplayer in total, because it’s not only our team that’s needed to make it work. Every single person on the team wants to see AI in private multiplayer. It's just a matter of our priorities and making sure that we get this into our schedule. We have a lot of stuff on our plate right now, and we are making sure that all of the current investments and features and releases are going to go off smoothly. But yeah, this is definitely on our list. We talk about this literally every day. Hopefully we can light that up soon.