Fast, Secure Interactions in Latent Environments

Latency is an inevitable part of the internet; but it doesn’t have to destroy your ability to create compelling, timing sensitive game play in your designs. With the proper fusion of creative design, art, and programming can often all but eliminate the perception of latency in a gaming environment, as long as we work creatively within the bounds of our environment. This article will attempt to define some basic rules and techniques used to combat the sense of latency in what is, inevitably, a latent environment.


Rule #1: You cannot fight the latency; you must accept, and plan for, a certain amount of latency.


While minor improvements have been made over the years, mostly due to moving from analog phone lines to cable and DSL connections, ISP’s and the average user is more concerned about bandwidth than ping time. Thus, it is unlikely we’ll see the basic internet gaming environment improving any time soon. Even if this became a focus of major ISPs, the best we could hope for is a more consistent ping times, not faster ping times.

Additionally, the internet environment is constantly shifting. The path you take from one server to another may change at any moment, and the time it takes to send a message between two points will change constantly. Packets will be lost, or arrive out of order. This is just the nature of things. Accept it.


Rule #2: Consistent latency is better than lower, inconsistent latency.


In some cases it can be in your interest to increase latency. If we use rendering as an example, a game looks better with a slower but consistent frame rate than it does with a faster but inconsistent and jerky frame rate. If the user has a perception of the latency, then it is vital for them to be able to determine how much that latency will be. Providing a consistent but longer latency allows them to compensate for the latency in the environment. If our goal is to remove latency from users perception, then having a consistent but longer latency allows us to adjust our game design around these timings, and potentially remove the perception of latency completely using a number of techniques.


Client Autonomy


One of the most basic techniques involved in removing the perception of latency is client autonomy. The basic idea is well known; you simply let the client act immediately and have the server somehow resolve the event when it finds out about it. An example might be allowing the user’s avatar to move as soon as they press the forward button, but having the server verify the user’s movement and correct the character’s position if the user makes an invalid move. Clearly, this has issues. In the Asheron’s Call series, differences in the physics simulation on the client and server often caused small blips as the characters position was adjusted, creating a jarring experience for the user. A ‘maximum difference’ between the server and client was imposed, often causing an invisible wall when the server was strained. Other games opted for looser constraints or simply didn’t use physics to minimize these problems.

Client Autonomy must be resolved by the server to be secure; but there are often cases where you just don’t care. If the particle effect which went off was wrong, it rarely makes sense to try to correct it. But if the user hacks the client and moves through a wall, you want the server to correct this. Ideally, the client should never need to be corrected unless it has been hacked.

Pre and Post Padding


Another technique is what I refer to as pre and post padding. A pre-padded or post-padded action is one that waits for the server to decide on the result rather than attempting to predict that result. This doesn’t mean the user has to experience this wait as a pause, as we can creatively design around this time. Lets take a freeze spell as an example; and lets say our round trip ping time is a whopping 500ms. We can easily cover a half second with the spell cast animation; if we want that faster, we can consider a post-padded effect. In this scenario the spell goes off and appears to hit its target; the frost begins to form, and as the server returns the result, the monster either breaks free of his ice prison or freezes solid.

If the latency can be accounted for within the timing locks and flows of the game, it effectively exists within the design, and you can design things to help account for a constant amount of time. However, in general, pre-padded actions feel awkward compared to post padding them, due to the fact that they feel delayed inherently delayed. It is always better to press a button and get the immediate response of the action you expect, instead of a slow windup before the event really goes off.

Rule #3: Prefer post padding when possible.


Time Shifting


The above techniques are used in the majority of MMP titles. Now lets move on to something a little trickier and a little more advanced. When working with AIs, we can synchronize much faster interactions without resorting to padding out reaction times or using client autonomous guesses that are often wrong.

As an example, let’s use a real-time fighting system such as the ones found in games like Street Fighter II or Soul Caliber. These games require timings that are too tight for most internet connections. Personally, anything over about 30ms is probably going to cause a problem for this type of design. Users can execute and block individual moves on a per frame timing, and slowing the game down to compensate for large latencies won’t produce an acceptable result. Clearly, this type of design would be impossible for an MMP. Or would it?

First, we’ll need to establish a latency buffer that’s larger than our longest round trip ping time. For this example, we’ll set this buffer to 500ms; that should be plenty of time. Our server will have to record a cache of all actions being executed within this buffer time.

Our first case to solve is the user blocking an AI attack. To do this, the server has to synchronize the moment of attack with the users blocking state, but of course, if the server sends the attack message at time 0, the user won’t receive it until up to 250ms later. If the user presses the block button to react, the server won’t receive that message until another 250ms has passed.

So, the first thing we need to do is “move the latency onto the server”. Our server won’t get pissed at us if it deals with latency, but our users certainly will. Right now we have 250ms between the server’s attack and the user receiving that message. We have another 250ms until the server receives the clients response. If, instead, the server could have 500ms of latency, and the user 0, wouldn’t that be a better scenario for the user to experience?
We can accomplish this by changing our thinking about how the AI and server work. Instead of sending a message to the client based on the current time, we post date them into the future. We send a message to the client that effectively says “At time 500ms, the AI will be at this position, and will attack in this direction”. This means that the server is making decisions 500ms in advance, effectively giving it 500ms of latency in its reactions.
This message gets sent and arrives at the client up to 250ms later, and the client begins the simulation of the attack at 250ms. At 450ms, the user presses the block key responding to the attack. At 500ms, the client autonomously plays the block effects and assumes the attack has been blocked. It can do this because it knows exactly what the server is doing during that time frame; the client is not running behind the server, as in the traditional case, because it is receiving the servers instructions well in advance of the actual actions begin time. The client sends of a message that effectively says “I block at this time, with this position and facing”.

By 700ms, well after the attack was blocked on the client, the server receives the message that the client pressed the block key at 450ms after the start of the event. It rewinds the simulation back to the time of the attack and verifies the simulation. Yep, the monster was attacking at 500ms, and the player was blocking during that time, so the user blocked it.

So what does this effectively do? It moves the problems of latency onto the AI programmer instead of the user. The AI programmer now has to think in forward time; the AI will always be viewing a simulation that’s 500ms behind the client because it has to make it’s decisions in advance, but the server will be able to fully verify the results against cheating clients, and the client will be able to autonomous predict the result without error. While this gives the client a distinct advantage, it’s usually much easier to make the AI harder than it is to convince the user that lag feels good. And any move the AI does can be effectively countered with very tight timings, allowing for things like real-time blocking of individual attacks.

Astute observers will quickly realize that this doesn’t work in reverse. Because the server is viewing time 250ms behind the client, while making decisions based on data that’s 250ms before that; it can’t react to user input in less than 500ms. Or can it?

Let’s say we want the AI to block the client’s attacks. The server won’t know about the attack until 250ms after it’s happened on the client (assuming no pre or post padding is used), and won’t be able to get a message back to the client for another 250ms. That just won’t do. Instead, we again need to change the way we think about the AI, and go for a latency friendly model. Instead of thinking “if the user attacks, block it”, we should effectively be thinking “between time 250ms and 750ms, any user attack will be blocked”. This allows the server to tell the client in advance, and the client already knows the answer when the user begins his attack; the monster will block this move. Again, the server can rewind time, notice that the monster is in the “block” state and correctly synchronize the simulation.

You have to design for this kind of a model; each move using a specific synchronization technique. The types of moves available for an AI will likely be different than those available for a player, as certain actions will only work in one direction. Interrupting a player’s action is easy, but interrupting a monster’s action is far trickier due to the fact that the server has already sent the actions to the other clients and must somehow correct the results. The server correcting things is always bad! However, you can always fall back to pre or post padding an event which goes against the stream of the latency model, and use a design or lore method to cover up the latency.


Synchronized Random


Another technique is to pre-send and synchronize various random rolls within a system. Puzzle Pirates does this with its sword fighting pieces, sending you a small batch of upcoming pieces every few turns so they’ll always be there in time. You could also consider trying to synchronize the actual random number generators between the client and server, though that’s a bit trickier.

Statistic Simulation


Many game feedback elements are not observed as discreet units by the user. For instance, in Guitar Hero the hand animation stops when you miss a note; but it actually stops the animation one note later, and restarts it one note later. This is because the animation has to start before the system knows if you’ve hit the note or not. Did anyone ever notice that it’s latent? Doubtful. Can you tell that your party member in WoW hit twice, then missed, then hit again? Or is a statistical average of hitting 75% of the time just as good? Do you really need an exact model, or is synchronizing to the rough result fine?

Sometimes, you don’t need to represent the data exactly as it appears, and can instead create a statistical approximation. You not only save yourself headaches, but a ton of net data as well.


Summary


I hope this has given you some insight into various techniques that can be used to fight the perception of latency. All of these techniques have limitations, and a mix of techniques combined with a willingness to adapt the design to what works for the internet environment is the clearest path to success. There is no silver bullet for latency, but with a bit of smart design and engineering it is entirely possible to have extremely timing sensitive game play in high latency environments like those found in MMPs. As you work with techniques like these, you’ll constantly find small ramifications that need to be accounted for in your design, or discover things you can do that you wouldn’t have expected. Go with it and embrace the platform you’re working on rather than fighting the inevitable, or thinking that slow, boring combat is an MMP convention we decided on. The truth is, the original MMP combat systems were all designed the way they were because we didn’t have enough understanding of the latency problem to risk doing anything else. We shouldn’t be constrained to these outdated concepts any longer.

I can’t, in good conscious, mention any of these techniques without mentioning Dan Ogles, who is as much if not more responsible for some of these approaches than I am.

slashdotted...

So my last post got slashdotted. Crazy. And here I thought no one was really listening. Anyway, a few clarifications:

1. I have the original arcade machine about 5 feet away from my XBox360, so those claiming it's a perfect port are clearly mistaken. There is a significant speed increase between the two, and the AI is completely re-written.

2. The actual speed increases I posted were not from any kind of measurements taken. I don't have any factual data on exactly how much each version was sped up, but the point was not the increase, but the fact that increasing the game speed further effectively makes the lag worse.

3. Anyone claiming they played lag free obviously does not understand the finer points of SF2. I'm sorry if you take that as an offense, but it's true. When playing at a pro level, you'd notice a single frames worth of lag; because you operate at the frame level of timings; and I don't know of anyone with a sustained 16ms ping time.

4. I tried to stay away from the controller and connection issues in my writeup, since both can potentially be fixed. Adding an emulation speed option would help reduce the inherent lag, but will never remove it completely.

I think it's time I wrote up some info on combating lag with both game design and programing techniques. I had originally wanted to do this as a GDC session, but didn't get the spot last year and I've been procrastinating ever since.

XBLA SF2

Anyone who knows me well knows I’m a die-hard Street Fighter II addict; more specifically, a Street Fighter II turbo edition addict. The Turbo (or Hyper Fighting, as it’s also called) edition, for me, was the ultimate achievement in game complexity and balance.

I was at the beach one summer and wondered into the arcade; I hadn’t been inside an arcade for a year or more, simply because the industry hadn’t produced anything interesting in a long time. I played a few rounds against the AI and left. A few days later, I decided to return and found a small group of people trying to figure out the controls. We quickly became obsessed with the game, and I’d bump into the same people every day at the same machine. Over the next year or so we began to unravel the various moves and combinations within with no instructions or internet to speak of. I think it was a solid year before anyone could do the uppercut reliably.

After a while, we got to know each other pretty well; we started venturing to other arcades, where we heard there was someone who could play a certain character really well. I went off to college and continued to learn new subtleties of the game; such as how to crash the machine with a certain combo, or other tricks for those that have mastered pretty much the entire game. I met the only person who could regularly beat me at the game, as he played a nearly flawless Dhalsim, pointing and laughing each time you made a mistake.

For me, this magical thing happens when you play a really good match of SF2. It’s as if time slows down and you’re looking directly into the mechanical, moving parts of your opponent’s brain. You dance around this clockwork in an effort to prevent your own behavior from following any strict patterns. To be truly good at the game, you have to remain loose and playful and free your mind from following any routine. If you fail to read your opponents mind, time suddenly speeds up and you’re caught in a whirlwind of confusion.

Editions after the Turbo edition didn’t work for me. Part of this was that they re-drew the art and changed the frame rate, destroying my mental map of the game’s finer details. While I’m sure this came from an amicable goal (make the game look better), I don’t think the developers realized exactly how it affected the players perception of the game. Extremely tight timings were harder to read due to the extra frames, and because they were all re-drawn, you had to rememorize the entire map of the game’s moves. I’m not talking about big details here, but subtle ones that many people don’t realize exist, such as the exact frame and spacing needed to kick someone’s foot after they miss a sweep. If you don’t play SF2 really well, you probably don’t even realize how precise the game actually is. I cooled off on the game for a few years as they continued to destroy the franchise.

About 6 years ago, we were playing SF2 on emulation. It sucked. So I figure, hey, I’m in the industry, I should just buy the real thing. I found a place in Philly selling the machine for $500, and it just happened that my co-worker’s roommate was in Philly driving a van with no seats in the back. It was destiny, and I began a regular schooling of all those in the office that would oppose me.

About a year later I was in an arcade in Japan, and noticed a man playing SF2 Turbo. Watching him for a moment, I noticed he actually knew how to play, so I jumped in for some matched. We had some great games, with me choosing some particularly uphill battles and coming through at the end. I don’t speak any Japanese, and he didn’t really speak English, but I was inside this guys head the same as anyone else. By the end of those matches, we knew each other’s minds better than many old friends do. He finally got up to leave, turned around and mustered up his only words in English to me: “You are…… very strong…”

I miss these types of matches; I miss the challenge and community once created by the game. So when they announced SF2 for Xbox360 with online play, I was intrigued. Now, having worked in the industry for so long, and having worked on latency management solutions, I knew there was no way the experience would be the same. But still, I wanted to see what it would be like, so when it was released yesterday I figured I’d give it a whirl.

Boy did they drop the ball on this one. First off, the game is about 20-30% faster than the original. This makes the game feel frantic, and leads me to believe there using one of the standard emulators as the base for the game, since they also run a bit faster than the original. They also did a bunch of annoying things, like changing the sounds around, which really serves no purpose as they don’t sound any better; just different.

Now, where the game really falls down is online. It’s bad enough playing SF2 on the 360 controller, but with lag it’s nearly impossible. The funny thing is, the lag really isn’t that bad from a technical standpoint. Most matches feel as if I’m getting a reasonable ping time and response. But SF2 turbo is a fast game, and at that speed, you just can’t play, and can’t compensate enough for the lag. If you press an attack button while leaving the ground on a jump, it’ll probably go off as its landing.

This is, of course, compounded by the fast that the game is sped up. What should have been implemented was an emulation speed option allowing you to slow the game down. The turbo edition is about 40% faster than the original SF2; meaning that this version is about 80% faster than the original SF2. If there was an option to run the emulation at the original game’s speed, this would cut the lag by almost half when compared to game time. For me, that would actually make the game playable. I can compensate for a bit of lag.

I’m sure many gamers are going to complain that the lag is the fault of bad programming; but no programming can overcome lag of the type of lag we have here. What exists here is a design problem, not a technical one. It requires a design solution, and the least invasive on such a master design is to slow the game down. While the lag isn’t reduced, the perception of lag is.