Seeing stuff like this on here fills me with nostalgia.
I basically learned to code thanks to a few helpful and extremely patient IRC members in the Spring RTS engine[1] community (some of whom I spot in these comment sections on occasion - hi!), and deterministic lockstep simulation for online play was one of the first ideas that totally blew my mind when I learned about it ("b-but that means all the "random" numbers on every computer need to be the same! and so do all the floating point calculations across different hardware!"). Of course, to me at the time, that was deep magic that I thought I'd never be able to fully understand.
That was more or less the case for Starcraft 1 as well.
It made for incredibly funny replay of matches if you happened to own a newer version of the map, as the game starts to diverge from reality and enter some parallel universe where players do nonsense actions and movements.
Just showing my appreciation for Army of Two. It was a cheesy, weird game, but my friend and I loved playing it regardless. The online multiplayer worked great!
Yes - and this can be AWESOME. In a deterministic engine, anything not affected by player input can be forecast accurately - by the computer, not the human user.
For NBA Inside Drive 2000, I developed all the ball physics to be deterministic and "reasonably" accurate (25 years ago, in a world without physics engines). This meant that after a player made a shot and the ball left their hands, the trajectory, scoring or bouncing off the rim, was purely controlled by deterministic physics. This in turn meant that we could foresee what was going to happen, and feed the player AI with that while the ball was in flight.
Obvious use was to move characters with high rebounds stats towards the place where the ball would bounce after not scoring, so they're more likely to pick the rebound.
[edit: for shooting the ball, the instant the ball would leave the player's hands I computed the perfect trajectory to hit a shot, and then based on the player stats, I'd nudge the trajectory off center. It was then purely up to the physics to determine if the ball went in or what]
Generally (I hope I get this explanation right), the PRNGs on all players' computers are the same algorithm, with the same seed, executing the same player commands in the same order. So as soon as the game starts, you can predict what all future calls to random() will return.
However, you can't predict the other players' actions, and those actions can insert an arbitrary (and unpredictable) number of random() calls between now and when your next action executes, so practically there is no gameplay advantage you can get from that avenue (AFAIK).
I'm gonna take an uneducated guess that though the simulation is in lockstep, the order of inputs from all players is not known ahead of time so you can't peek too far.
Though maybe a cheat engine could simulate a few branches of possible inputs a few steps ahead and adjust inputs when it discovers split-second advantages?
Everyone waits for your client's next step. Your client can basically pause the game whenever it wants, giving you time to do something.
But there are bigger exploits. IIRC, the client sends separate actions for buying a unit/tech/building and paying resources, and the other clients don't enforce that they go together. This allows malicious clients to buy without paying. It's been seen a few times even in the ranked mode of the Definitive Edition remake. https://www.reddit.com/r/aoe2/comments/fjjudp/cheaters_are_a...
This way of doing things made sense for the time, and the execution was impressive. It was pretty reliable, nobody wanted to cheat anyway, and yes lag was a big problem but there wasn't another good way to do it back then. Nowadays they should have something else IMO.
That problem doesn't seem inherent to the model, it's a flaw in the selection of client steps. Is there any reason transactions like building and paying couldn't be atomic?
The problem is simply the lack of validation -- I detailed it a bit below in my top-level comment.
The only exploits that are inherently applicable in the case of a simultaneous deterministic simulation model are those of the information-revealing class (e.g, "maphack"). This is where a standard AH system comes in.
I don't know why it's that way. My guess would be complexity. Yeah, they could fix that later, but they can't fix the fog of war cheat, which would also be pretty powerful.
Deterministic multiplayer has an assortment of cheating problems that weren't really a big deal in the early days of online multiplayer, but certainly are now, in the days of ranking systems and very competitive gaming.
The obvious one is that the client knows where everything is on the map, even when hidden by fog-of-war - so client-side hacks can reveal that.
A hacked client can easily desync the game too, or stall it and cause it to time out if the player is losing.
Not to any meaningful degree -- inputs (from humans) permutate the state. You can, however, compute say the initial world state, revealing the map/resources/ect. Ofcourse in this case you'd may aswell just look for some sort of maphack.
Hedgewars also uses deterministic lockstep. But due to the floating point problem, the main developer elected to write custom fixed point math instead. It works since the game doesn't use a lot of physics.
You are not by chance the guy responsible for the commando-sneak widget? Computes the view circles of know units, sneaks a whole army single file into enemy territory by chosing a out of sight route?
No but that sounds fun! The best one I made was a really hacky script called followmouse that would constantly tell selected units to move towards the cursor. It was for microing around defenses and whatnot.
The random numbers I understand, because I’ve used a similar principle to make tests deterministic. The trick is to make the same number of calls to the generator and use the same seed. The floats are really thorny though, differing subtly across CPUs. Did they just rely strategically on rounding? Make heavier use of integers?
Don't use floats (it wasn't a 3D game) or if you have to, make sure every float is derived from an integer that's kept in sync. And don't do anything that affects simulation state based on those floats.
Age of Empires 2: Definitive Edition consistently de-syncs when playing across a M1 Mac (using Windows 11 Arm or Crossover) and an x86 machine, and I suspect the difference in floating point behavior described here is the culprit: https://developer.apple.com/documentation/apple-silicon/addr...
LHC@Home had big issues[1] with AMD and Intel machines giving very different results to the same work unit. They traced it down to the exp() function behaving different, and ended up using a library[2] for anything more fancy than basic arithmetic.
Another reason for me to stay clear of the M1 for now. It’s really fascinating, but it’s not quite there yet. I’ll let others work the rough edges away before I give it a try.
The id of the node acts as the deterministic seed. Really handy when building views that slice and dice data, it feels like you’re interacting with a database.
IRL it was not simple. The game usually worked, but occasionally things would go out of sync.
The biggest downside is mentioned in the article: If one person lags, everyone lags. In an 8-player game, the chance that at least one person lags was pretty high. I used to play on the Mac version of GameRanger, a small community. That was 2008, when reliable internet wasn't as common. Players had personal reputations, and those who lagged too much were often excluded from games.
That's not intrinsic to this style of netcode, though. Company of Heroes, for instance, just keeps the game running and your input is delayed until you can communicate it to the server.
You'd generally want an authoritative server to accomplish this. Having it also execute the inputs prevents most cheating, too.
It is, in fact, inherent to this model. While there will be some tolerance threshold as to how far behind in the simulation a machine may fall, it still must be rather conservative (generally a few seconds), as elsewise the context (state-) sensitive inputs start frequently failing -- if I issue a command given at world-state time t, will it be meaningful/valid at state t2? The answer is where you get your threshold and it's always just a conservative approximation.
I've built and licensed a time traveling variant of this model (GGPO style) and spent years with thousands of concurrent users playing my games. And they operated just as I describe.
Company of Heroes is more of a traditional AoE style and will happily queue up unit orders for a minute or more while it attempts to reconnect or catch up.
You could probably build something better than what the AoE2 remake has now. I don't know if it'd be viable in 1998, though. Wasn't like today's constantly self-updating games; each patch risked fragmenting the community, and they only released like 2 patches across 15 years of activity. They chose something simple that'd work well enough without having to tweak it. And old PC specs, and no dedicated servers. You said it's better to disadvantage laggers than to punish everyone, but that requires some balancing to ensure it's fair enough. AoE model is always fair and can be lag-free.
Yes it relies on the players trusting each other a little to avoid extreme lag, but there are plenty of ways to ruin this kind of game besides lagging. It's not like CoD where a few useless teammates won't make a big difference, while it didn't have automated ranked play like Csgo. (but the recent DE remake does, and yes it needs better netcode)
Time-traveling netcode... I know Slippi does this, and it works great.
Are these players interacting in any way? If they aren't then it could be more tolerable, but one player being more than several seconds behind others is going to lead to a rather undesirable experience, especially in the middle of a battle (and having most of their commands dropped due to being invalid equally so)
Giving the undesirable experience to one player is strictly superior to giving it to all of them.
There just aren't many modern multiplayer games where slow players can negatively impact the experience of others, and that's because the industry has learned how terrible the experience is.
And that's for players playing in good faith on bad connections - abuse by inducing latency was a big thing in the past. Think a player that runs out the clock when they're a move away from being checkmated.
Basically the only time you'd ever want to operate the system to pause if a player fell behind is during a tournament, where fairness is more important than player experience and they know that going in. And even that, IMO, is questionable.
I'm of the opinion that it's an acceptable concession that a brief pause occurs in gameplay to permit another player to recover from transient issues (network hiccups, brief machine resource starvation, or generally: stalling) in favor of fairness rather than punishing the player too harshly for factors that are frequently out of their control.
Consider for instance the situation where-in one player's simulation enters the spiral-of-death, under the policy that you're proposing this player is effectively removed from the game; I would say this is a substantially worse trade-off -- I know that I would personally prefer some mild annoyance of brief pausing as opposed to a player leaving the game (and in session-based games, as RTS's generally are, a team-mate leaving often ruins the entire game)
Abuse is indeed a problem that most games I've seen of this nature don't address, and while there's no perfect solution, it can be quite effectively curtailed. In the systems I've designed, the basic principle is that each participating player has an allocated amount of wait time which serves as a quota that is charged while-ever the game can progress/run, but cannot, due to them being in a wait-state (connecting/loading/reconnecting/stalling/explicit pausing/whatever). There's some extra logic for handling the oscillation of wait-state.
The spiral of death trade-off does not happen as you describe.
While a client is catching up, you run the sim in a loop and don't render the game.
The deterministic simulation code is only a small part of the overall game logic, and you can run it much faster than real time.
If a client is slow enough to genuinely experience a spiral of death, the game will not be playable for them, and "mild annoyance of brief pausing" is not remotely what the other players will experience if they're forced to stay in sync.
Wait time is the common solution in older games to prevent abuse, yes. But it doesn't prevent somebody with high ping making a game run at half speed.
Go play Company of Heroes 2 with a packet loss simulator. Even as the bad client, it's totally playable. For everyone else, it's silky smooth.
I don't think you'll come back and tell me you prefer the implementation in AoE II Definitive Edition.
The simulation is, rather obviously, what's being referred to with the spiral-of-death -- it's not unusual for instance in Age of Empires II to encounter transient periods where-in steps take an order of magnitude or more than the period they're simulating on less-capable machines (due to some O(n*n)'s in the AI system).
In the end this is still a decision regarding how to handle player's falling behind; too far and they're effectively taken out of the game as they're no longer interacting within a sufficient approximation of the current state.
Under what design is "high ping" going to cause the game to run at lower "speed"? The only possibility I can see is where-in the turn period is, for some bizarre reason, also the granularity of the simulation step and you're also adapting the turn based on player latency (in aid of fairness, usually)
Having implemented such a system myself, a few times now, it has in practice performed sufficiently well.
DE is garbage in general so it's not really a fair comparison.
Transient periods is the point - for a client to enter a spiral of death, the simulation is running slowly enough that a client can't keep up, even if it's all they're doing. If a client enters this state, it's simply not going to be able to run the rest of the game, and it's out of the match regardless of what you do unless the game slows down permanently for everyone.
If you could simply wait a few seconds for them to catch up, they'd also be able to catch up without a global pause.
In any case, we're going in circles. I came to this thread to claim that "If one person lags, everyone lags." is not intrinsic to this kind of net code, and you disagreed. Company of Heroes and my own games are existence proof of my claim. I'm not really interested in arguing about it any more - millions of players can't be wrong.
> You'd generally want an authoritative server to accomplish this
That makes it very different from this model in my eyes. You wouldn't have every client running the game in sync. The AoE devs had this dilemma and chose the other path.
Don't you have to handle the situation where one client falls behind? That means asking for the latest state and figuring out what happens to the stale inputs it sent. It's a nontrivial difference.
Definitive and HD Edition both seem to use nearly the same netcode as the original game. And the whole thing lags if one player lags. Yes it handles situations where all players leave, so there's a server doing something, but the glitches in Elo changes hint to some weird edge cases. I even found an easily triggered bug in HD Edition where both players would gain Elo at the end of a ranked game.
I was writing a real-time multiplayer game a couple years ago.
The netcode was written so that players only send their inputs along with a timestamp (The "timestamp" actually being a frame counter). The server would then relay those inputs with the timestamp to other players. The client maintains snapshots of the game state and would then re-run the game logic from that timestamp given the new input.
I handled clients falling behind by making the server reject any inputs that were more than 500ms (Really, 30 frames) behind the current time. The server would send a message to the client that essentially said "You're lagging, here's the current frame counter and current state of the game"
Granted, this approach really only worked for my game because the entire game state could be squished to ~300 bytes. But it had the added benefit that it prevented players from cheating using a lag switch. They couldn't have a malicious client that sat and listened to what other players were doing and then send a bunch of back-dated inputs to put itself in an advantageous position.
You definitely can build out functionality in the authoritative server, like drop in. I did that with my games.
But you could keep it strictly as cheat prevention, which keeps it very close to the AoE model.
I wouldn't start a game in 2023 trusting any input from players, including win reporting - but I understand if their hands are tied by a legacy system, and the players vote on outcomes.
Dropping in a server like another client might be what they did in HD/DE remakes to allow the game to continue without a player host present. It doesn't help with cheating, cause other players are already interested in enforcing the rules. And doesn't help with lag ofc.
Yes, the remakes should've had new netcode. They had full freedom to redo it, no legacy compatibility to deal with. Maybe they wanted to totally emulate the old experience, but ya know, I can just play the old game. DE at least seems to be working out the kinks better; HD was a total mess even after years of patching.
(Want to add, I'm not complaining about the engineering. This simple approach totally made sense for a late-90s RTS without dedicated servers, continuous patches, or high system requirements. And it held up impressively well considering all that.)
Man, this brings back memories. People fighting over char in the lobbies, some derivative of “laggy” used a slur to try and get someone you don’t like to leave the game before enough people joined, etc. fun times!
I was always confused about that taunt. It sounds like, blame your poor play on your ISP, but this is one of the few games where laggy internet doesn't affect your moves.
In AoE2 custom scenario lobbies like CBA, people would often see the file transfer as an indicator that you're a n00b since you didn't already have the scenario, and you'd get kicked.
I think this is a “must-read” for anyone, even if you aren’t in game dev or doing networking stuff. There’s just so many mini “eureka!” nuggets to enjoy.
I think that internet multiplayer is possibly one of the hardest things in game dev (use an engine that gives it to you!!!) despite feeling like some little sideshow.
I imagine Blizzard and Westwood have similar stories from the 90s.
> Cheating to reveal information locally was still possible, but these few leaks were relatively easy to secure in subsequent patches and revisions.
I don't understand this. Running the whole simulation locally on both ends means that a modified client would have access to the whole game state, and I don't really see how you could patch that out.
Anyone have any idea what they actually did? Try to detect modified clients? Obfuscate the game state to make it harder to interpret?
In age II, it was CRC checks of game state that prevented cheating! Games would get out of sync if there was a mismatch, which could happen for various reasons.
I worked at MacSoft during the Age2 and Age3 days. The ports were faithful to the windows versions, but cross platform hadn’t been solved at the time because of this problem.
It also made long game play sessions longer because the calculated state kept getting more complicated.
There was one particular Mac OS X update that broke math interoperability for multiplayer because they changed how math worked on the OS. This meant we had to bundle a common math library to ensure the game the game states would line up, preventing unnecessary CRC checks.
Do you still remember what actual toolchain you used to build Age2? I have been looking into these games for some time, and the Mac build has been helpful, but has some oddities, like very weird floating point argument passing to functions, or unpredictable vtable placement ... knowing which exact compiler toolchain was used, would be incredibly helpful! I know it's probably some VisualAge C++ version because of the name mangling, but not exactly which.
In the Definitive Edition, there has been observed cheating in online ranked mode where players give themselves infinite resources. Evidently there's a way to do that without going OoS. This was probably the same in the original. This is besides the less complex cheat, removing fog of war. Seems the only thing stopping most people from cheating is, they don't want to.
Also, I used to play the Mac version a lot. It was great! Shame they made the remakes Windows-only.
You can call the command creation functions yourself, making the game think it's part of the simulation [1]. These types of cheat have been around ever since the original game, and the HD edition [2].
I was thinking of reverse-engineering the game's calls out of curiosity, but I figured someone else already did it and wrote a nice article about it... and there it is.
Sorta. You have to mess with the DLLs and Proton settings to make multiplayer work, and even then it's laggy. It also works in Wine on Mac in some cases, but for me it always had problems.
But CRC checks wouldn't prevent things like revealing units through fog of war right? The presentation of the game state couldn't be CRCd because each player has a different view of it. And the cheat client doesn't have to modify the actual game state to get that information out.
The game actually tracks this too, what is visible for each player, which gets slightly complicated with diplomacy changing. Of course this doesn't prevent you from just patching other parts handling this, but you cannot just simply modify this value, it will desync as well.
One of the unofficial solutions used in Warcraft 3 was to spawn an illegal 3D model object in the corner of the map by a trigger as soon as the map begins, or during random spot checks during the map gameplay.
The model would crash the game (and world editor, that's why we have to spawn it during runtime) when displayed, but it wouldn't get displayed when under fog of war, so you'd put it in a place that is impossible to be seen by a player under normal circumstances. But if someone uses a fog of war cheat or a maphack, it'd crash for them.
Of course it won't prevent you from more advanced hacks which e.g. modify the client and display an overlay of the enemy units rather than just revealing the fog of war.
Similarly, a common technique used within notably the DotA community (of which's map didn't have such a tripwire) was to analyze the replay for what were termed "fog clicks", since for whatever reason object selection is part of the command stream and those using maphack would often, intentionally or inadvertently, select objects otherwise under fog.
That's cool. AoE2 scenarios sometimes had a different kind of anti-cheat, preventing players from deleting buildings that would otherwise count as points to enemies who raze them. That relied on a lot of complex triggers that I think involved spawning birds to keep count of things.
As long as users are running the game on their own computers, preventing that type of read-only cheating is not possible. "Solutions" to this problem come in the form of invasive spyware, such as Warden (Blizzard), Easy Anti-Cheat (Epic), Vanguard (Riot), etc. These are programs that run with the highest possible priviledge, inspect all memory/storage/devices/input, and report what they find to a server.
So you're saying that in order to avoid the "elitist" nightmare world of "just play with your friends, and people you trust" and "normal people are never going to want to be able to run their own servers!" all we have to do is hand over unlimited powers of surveillance to games corporations and succumb to their daylight banditry, paying a premium price for the privilege of being able to play call of duty with the teenage edgelords you may know from such online wonders as "youtube comments"? Sounds like a bargain! Sign me up! :)
Ok, so option 1) is to make internet multiplayer run on a game server, not peer to peer, but that's not exactly running the game on their own computers.
Another option would be for the peers to only send data that the other should know (fog of war), but that's a lot trickier. Because you then need to figure out how to validate the unseen data wasn't cheating, too. You might be able to do something with this today, because storage, computation, and bandwdith has grown so much.
Maybe store all the local state changes, and when a block becomes visible, send its current state first, and then stream the history as time permits; the other side would accept the state initialy, until it could fully validate it.
You would need some way to keep the randomizers in sync and fair, too.
But that only protects from seeing what should be invisible; you could still have computer enhanced movement and maybe enhanced display of data attributes that weren't supposed to be human visible.
Yes, not having the entire game state on the user's computer in the first place is a solid choice. And for many games, that's exactly how multiplayer works. They run all the logic on a server, "never trust the client" (that is, validate all the input that clients send to the server), and only send data the user needs to know.
However for something like an RTS, the amount of data in game state updates can be prohibitively large to transmit to clients. The deterministic lockstep networking model described in the article is a solution to that problem. In that system, the only data transmitted is input, and each client updates locally, so it does require them to have a copy of the entire game state.
Run entire game on server, send stuff to clients, get inputs from clients. Or full-on stream the game like Stadia (RIP). Which is hard with low-spec hardware.
I still regularly have dreams about it 20 years later.
I take it as a sign that I'm in a good healthy place mentally when the most stressful and anxiety inducing thing my brain can conjure up is me being rushed by 3 units in the dark age.
What I find interesting about the lockstep approach described here is it seems a lot of modern RTS games still use the same approach, even though the bandwidth calculations have changed radically. When developing my own RTS game I was able to fairly straightforwardly get 1000 units in combat to live stream using about 50 KiB/s bandwidth, which is nothing these days: https://www.construct.net/en/blogs/ashleys-blog-2/rts-devlog...
So as much as this is a fascinating piece of history and an impressive technical solution to the constraints of the time, I think modern games ought to move past it.
> So as much as this is a fascinating piece of history and an impressive technical solution to the constraints of the time, I think modern games ought to move past it.
But why? As far as I know mega-hits like Warcraft III and it's new, updated, version, "Warcraft III: Reforged" which came out 20 years later still use that technique.
The benefits do go well beyond being able to "send" hundreds of units across the wire: a deterministic game engine allows to create tiny replay files and, very importantly, allows to find and smash bugs way quicker.
Having the next game state being a deterministic function of the current game state + player inputs is great.
What would RTS games win by "moving past" that? To do what instead? How would you then implement the replay functionality? You'd also invariably run into a class of bugs which would be hard to reproduce but which would be trivial to reproduce using a deterministic engine.
From a latency point of view you're not gaining anything either: you need to receive the other player's units position anyway. So what's the difference between receiving the hundreds of unit's position or receiving the player input that created these unit's position? Just compute them, deterministically, as soon as you get the player's input.
Everything I've read about the lock-step approach is that it is a total nightmare to develop - keeping games deterministic is really hard and a de-sync bug that happens out the blue after 1 hour of 4-player multiplayer is the kind of thing that is extremely difficult to get to the bottom of. Streaming everything is by comparison much easier to develop in my view since that class of problems disappears. It also allows late-joining, including spectators, as it syncs the full state of the game periodically, and it makes it resistant to brief network outages, such as going through a tunnel on a train while on cell data.
I think it's also worth sometimes revisiting the assumptions made for the current algorithms and approaches in use. The original design was for dial-up modems. The networking landscape is completely different now. Maybe some of the original assumptions are no longer valid and a different set of tradeoffs is worthwhile.
Did those units shoot? What happens when there is a few thousand projectiles in the air, fired by those 1000 units? What about map deformations? What about map wide physic simulations, like springs tsunami water sim or the lava flows of a volcano, changing directions?
None of this should matter as long as the algorithm determining randomness is deterministic.
The bottleneck is player input which is the most overestimated bandwidth stat in gaming. It's mouse movements and a couple of keys strokes per second. Top Starcraft players are in the 300 actions per minute range, that's still just 5 per second.
You don't need to track every projectile, you just need to know that player 1's unit 33 started shooting at player 2's unit 45.
You can encode that very compactly. 2 bytes for each unit ID, source and destination. So 1000 units would be just around 4K, if they all start shooting at the same time.
After that, you can rely on that in most RTS games how units shoot is deterministic.
IIRC, and I've got nearly all my emails since the nineties until now so I could find out, I sent an email to the author of that article back when I read it, I think, still IIRC, on the Gamasutra website. And he answered me: we chit-chatted about deterministic game engines, for I wrote one in... 1991! (not for a networked-game though).
The subject already came up here on HN and some posted about games using deterministic engines before AoE.
The reason I did it is I had a bug which happened ultra-rarely and couldn't figure it out so I thought a long time about this and realized a could make the game deterministic and that would maybe allow me to record the bug happening and then be able to replay it. I found that by myself: back in 1991 I had never heard of anyone writing a deterministic engine back then. So I took a few days and rewrote the engine to be fully deterministic. Sure enough it eventually caught the bug: some case where the hero would clear a level after having fired two shots at once, which was an extra (by default it only had one shot at any time). The shot still on the previous level would continue to "live", invisible, in the following level, and would corrupt the memory. Classic.
Yup, this post is how I found out that they changed their name.
> Game Developer was first founded in 1997 as Gamasutra and has strived since its inception to be a leading resource and reference for game development and industry knowledge. Following the shift from Gamasutra to Game Developer in August 2021, the site has maintained that mission while embracing the in-depth content its namesake Game Developer magazine is known for.
Shame that. Sutra is a word all on its own. Gamasutra was a nifty name. It works.
One day sex positive triple-breasted feminist whores of Eroticon Six are all the rage the next you're sunsetting harmless Sanskrit puns. Y'all strange.
Archer micro is a huge part of the competitive AoE2 scene. AoE4 had auto tracking arrows and I think folks didn't like it as much as it lowered the skill ceiling
I understand what was said. I'm saying that sucks. I don't really care about the tiny minority of people who care about the skill ceiling within a super low level micro.
RTS entices people into a strategy game. But the games are so similar that everyone but the top 1% of players will do better by simply following a predefined strategy and executing it as tightly as they possible can. And that sucks. Nudging guys a few inches to the side is the opposite of what makes the game fun for most people.
I'd like to see RTS games that randomize game parameters each time, requiring you to actually strategize.
> I don't really care about the tiny minority of people who care about the skill ceiling within a super low level micro.
I'd argue that most people that kept playing aoe2 online and kept it "alive" all care about that.
> I'd like to see RTS games that randomize game parameters each time, requiring you to actually strategize.
aoe2 already has randomised maps, I think that's why you actually get less strategy (in a sense).
In games with fixed maps, you see much more interesting one-off strategies (people like to call it cheese[0]) because you can plan everything down to the second. You'd plan & practice strategies on specific maps in specific matchups against the "standard" build orders.
AoE2 has these types of strategies too, but since the maps are always slightly different, you can't plan your building placement etc ahead of time.
There's also MegaRandom [1] which takes the randomisation to the next level. But I'd argue micro matters significantly more in that map, since you can end up in "unfair" situations where you need to outplay with micro to stand a chance.
> Nudging guys a few inches to the side is the opposite of what makes the game fun for most people.
That's how these games stay popular for 20 years. It's like complaining about strafe jumping in quake, it's what kept the game alive all this time.
> aoe2 already has randomised maps, I think that's why you actually get less strategy (in a sense).
I'm more of a fan of the in-game, on the spot strategizing that randomized maps force you to do, than the out-of-game meta-strategizing of fixed maps, where you do comparatively limited in-game strategizing.
> I'd argue that most people that kept playing aoe2 online and kept it "alive" all care about that.
I didn't say they didn't?
> aoe2 already has randomised maps, I think that's why you actually get less strategy (in a sense). In games with fixed maps, you see much more interesting one-off strategies (people like to call it cheese[0]) because you can plan everything down to the second. You'd plan & practice strategies on specific maps in specific matchups against the "standard" build orders. AoE2 has these types of strategies too, but since the maps are always slightly different, you can't plan your building placement etc ahead of time. There's also MegaRandom [1] which takes the randomisation to the next level. But I'd argue micro matters significantly more in that map, since you can end up in "unfair" situations where you need to outplay with micro to stand a chance.
The random maps are random within a very narrow set of parameters. They don't change the strategy. Practicing standard build orders is what 99% of players will need to do. I agree, at the very top level there is a level of strategy again. But in that grind to the top, the way you do better is by simply narrowing the time to get your first rush online. There's very little chance to actually change your gameplan.
> That's how these games stay popular for 20 years. It's like complaining about strafe jumping in quake, it's what kept the game alive all this time.
These games have not stayed popular. RTS has not done very well as a genre. You've got Starcraft and AoE2. And that's kind of it. AoM, AoE3, the shittier spinoffs, and AoE4 haven't made particularly big splashes. Nor have other franchises.
The thing is, these games are really fun to play casually, or in single player. But playing competitively where people are going to do stuff that works is so different and not compatible with what people liked when they were introduced to the series.
Consider a comparison to fighting games, where it's also a hard requirement to be obsessively deep with the game to make any progress. Fighting games have still seen a ton of success with many popular new titles.
edit: I was super deep on the AoM scene and mildly deep on the AoE2 scene; dropping off well past the title's prime but sometime before AoE2:DE
edit edit: I can also recall that the majority of online players back in the day would like the setting but seem to lose interest in playing normal maps and eventually settle on silly scenarios. It's not surprising that MOBA games have really taken off as that's what a lot of these scenarios were in some form; but worse.
If you get as chance give AOE2:DE a shot, I feel like it's what you are describing. It's impressive how many randomized game parameters it has... they even included a randomize button.
Additional you can add mods such as 'Random Costs' where units and buildings have random costs so standard build orders are impossible. Because the game is still being updated, people are making new maps and mods, there are often new strategies being discovered frequently.
But to me, the best part of the game is playing with a group that are all a similar skill level. Fortunately the game has a match making system letting my mates and I get paired up with others.
The randomize button was added in UserPatch, which was the de-facto version people played (on Voobly) until DE2 came around, it just took that over from there.
Nah, micro can definitely be exciting for even more casual players. They'll be worse at it, obviously, just like casual FPS players are worse at headshots, but it's not like they can't do those things at all.
The data we have suggests that people with your opinion -- "people want RTSes just about the strategy, not about control" -- are dead wrong. The most enduringly popular RTSes are mechanically demanding games: StarCraft 2, Age of Empires 2, StarCraft 1.
If you remove the control aspects of an RTS, you get something that's closer to turn-based strategy. Nothing wrong with turn-based, but that's just not what most people come to RTS for.
> The data we have suggests that people with your opinion -- "people want RTSes just about the strategy, not about control" -- are dead wrong. The most enduringly popular RTSes are mechanically demanding games: StarCraft 2, Age of Empires 2, StarCraft 1.
I think this data suggests the opposite. These titles are all more than a decade old. They have endured, but the genre has died. People often find it really compelling to get into these games and do the singleplayer stuff. The multiplayer experience didn't live up to that. Limited strategic choices and a demanding micro are not what most people liked when they first entered the genre.
There are some indie titles that seem to understand this.
> I think this data suggests the opposite. These titles are all more than a decade old. They have endured, but the genre has died.
What you're missing is that there have been many titles that have reduced the mechanical demands in pursuit of "greater accessibility", and have been even less popular. That's been most RTSes over the last 15 years, really. They're always talking about how they wanna get rid of the clickiness or base building or what have you. "We're not gonna be APM heavy like Starcraft!" they crow, as they quickly fall into obscurity while Starcraft remains.
It's not wrong to want the genre to be more accessible or fun, or less frustrating, but getting rid of technical and tactical options by reducing control of your units is generally a bad way to go about that.
> People often find it really compelling to get into these games and do the singleplayer stuff. The multiplayer experience didn't live up to that. Limited strategic choices and a demanding micro are not what most people liked when they first entered the genre.
Nah, what points the way forward is SC2's co-op mode. That was extremely popular among more casual players -- being more popular than ranked ladder at least for a time -- despite getting only limited support as a new feature in LotV.
And the way SC2 co-op worked is that it had the same mechanical inputs of course, but it actually had less strategy, not more; similar to campaign AI's, the AI in co-op mode is largely scripted and predictable, no scouting and very little reacting required. People loved it!
> What you're missing is that there have been many titles that have reduced the mechanical demands in pursuit of "greater accessibility", and have been even less popular. That's been most RTSes over the last 15 years, really. They're always talking about how they wanna get rid of the clickiness or base building or what have you.
I'm not convinced this is accurate. I'm of the opinion that by the time AoE 3 came out the fire was gone from the genre because people did not like playing the multiplayer.
Excessive micro sucks, but that's not really the main thesis of my argument. It's more so that the traditional "Random Map Battle" mode is really devoid of strategy for the majority of players because you have so few real choices to make. You don't get to do much strategy. You probably have a build order for several minutes. And then want to rush. That's almost always the optimal move until you're very, very talented. I can't think of any competitive multiplayer focused RTS games that tried to address that successfully.
> Nah, what points the way forward is SC2's co-op mode. That was extremely popular among more casual players -- being more popular than ranked ladder at least for a time -- despite getting only limited support as a new feature in LotV.
> And the way SC2 co-op worked is that it had the same mechanical inputs of course, but it actually had less strategy, not more; similar to campaign AI's, the AI in co-op mode is largely scripted and predictable, no scouting and very little reacting required. People loved it!
I'm amused because I feel that was MY point. Single player (and naturally also co-op) content against scripted scenarios was the best part of RTS games. It's where the genre was born. You get to ask yourself "What am I going to do" and come up with a plan. My point is that you DON'T get to do that with Random Map Battle RTS games in competitive multiplayer. Instead you largely just get punished for not doing the meta plan; or worse, get punished because your execution of the meta plan is worse.
Well, it is. They've been a lot less successful. "The fire was gone from the genre" isn't how things work. SC2 co-op is a good example of this -- if something is good and polished, people will play it, have fun, spread it around, etc.
In contrast, you could look at the CoH series, which is one of the many games that went for being less APM heavy (albeit there's still a fair amount of micro involved). CoH2 is decently popular, sure, but much less so than AoE2 or SC2. The CoH/DoW branch is one subgenre of RTS that's less demanding, but you could look at the C&C and TA-like branches the same way. And yeah, they're moderately popular, but less so than the more demanding games. Why?
> It's more so that the traditional "Random Map Battle" mode is really devoid of strategy for the majority of players because you have so few real choices to make. You don't get to do much strategy. You probably have a build order for several minutes. And then want to rush. That's almost always the optimal move until you're very, very talented. I can't think of any competitive multiplayer focused RTS games that tried to address that successfully.
This is just an argument from ignorance. The truth is that even APM-heavy games like AoE2 or Starcraft have plenty of strategy. Well, maybe not the really cheesy games where someone immediately goes all-in, but the ones that go beyond a few minutes, yes.
Like, these words
> You probably have a build order for several minutes. And then want to rush.
don't even make any sense. In the case of Starcraft, if the game is several minutes in, attacking is no longer a "rush". It's just...an attack. Which you'd expect people to do in an RTS. And even early attacks aren't necessarily rushes.
The words you've put in here is like stumbling into a discussion about Counterstrike and claiming that the game is nothing but people using the AWP forever and ever, or that you can win every game of PUBG or Fortnite by doing nothing but hiding. You're simply wrong, and advertising that you don't really know what you're talking about.
> I'm amused because I feel that was MY point. Single player (and naturally also co-op) content against scripted scenarios was the best part of RTS games. It's where the genre was born.
If that's what you like most, that's fine, but it's true that these involve less strategy, rather than more. You don't have to outsmart someone, you don't have to scout them out and react, all the while they're scouting you out and reacting to you. The computer's on rails and there's usually a dozen different ways to exploit how dumb the AI is. That can be good fun, just like any other single player game, but it's not very strategic at all, because your opponent isn't actually strategizing against you.
> My point is that you DON'T get to do that with Random Map Battle RTS games in competitive multiplayer. Instead you largely just get punished for not doing the meta plan; or worse, get punished because your execution of the meta plan is worse.
People get 'punished' if they have poor strategies or poor execution of strategies, which is as it should be, yes. There's nothing special about being "on meta", what's meta is usually just what people have found to be effective -- at lower levels, the professional meta tends to be markedly less dominant, with all kinds of other strategies being effective that wouldn't be at higher levels. I'm considerably higher ranked than average, but I still lose to plenty of stuff that would never work at a pro level.
It doesn't lower the skill ceiling, it moves the emphasis on which skills contribute to winning.
It's relatively easy to spot when a game has a problem with low skill ceiling: the very best players have difficulty distinguishing themselves from other top players. So if the top player plays the 100th best player, in a game with low skill ceiling the top player might have a 55% chance of winning or so.
This is very much not the case in AoE4. And if AoE2 would add auto tracking arrows tomorrow, you wouldn't see the skill ceiling drop appreciably. You'd see players with strong micro but weaker macro and strategy fall in the rankings, and slower players with better macro and strategy rise.
Yes. Most people back in 2001 were not using fibre. Dial-up, DSL, etc are all much higher latency. In Dial-up you are easily talking at least 100ms slower, if not a lot more.
You bet. Mangonel micro, archer kiting, cavalry bluff charges, deer pushing etc are all things you regularly see today that wouldn't have been feasible in the old versions. (in online play)
Yes: dialup modem latency was on the order of 100-250ms, DSL 10-70ms, and cable modems 5-40ms.
Sitting on my couch using Wi-Fi over cable, I have 18ms to Google.com – call it a full order of magnitude better than a disk up user despite Wi-Fi adding 7ms.
Last mile latency has been reduced a lot as people moved to broadband, but also on longer distance routing where there are more paths from A to B than twenty years ago, and some of them are physically shorter. Higher bandwidth interconnects also mean less time in transit; it's not a lot, but it is some.
Also, some early broadband networks backhauled too much traffic to central locations. I recall commonly seeing traceroutes from my home in orange county, ca going to kansas, and then coming back to servers in los angeles. I still see some traffic that leaves my metro area only to come back, but it's less frequent, and it's usually only going one or two states instead of halfway across the country.
As someone who fixes up old games in my spare time, and Age of Empires II being one of them, I'll provide a bit of trivia about the game's internals:
- The AI system is not part of the deterministic simulation. This was surprising to me, and after contacting one of the original programmers it was explained that it was due to a desynchronization bug that the "AI and network programmers weren't able to fix it in time".
A consequence of this design regression is that, due to the AI now being authoritatively run by the designated host player, network congestion issues arose which lead to a clear series of progressively more aggressive optimizations to reduce egress traffic. This primarily consisted of a very simple filter (mentioned in the article) which dropped duplicate commands in the common submission path, meaning it applied to both local user and AI commands, along with batching of AI-submitted commands which would be flushed at rather arbitrary times.
I'll note that I've restored AI being deterministic in my project.
- A rather obscure determinism bug resulted from their compiler's implementation of a few CRT routines, namely fsin/fcos and a few others, which leveraged the specialized ISA instructions of the same name. The problem being that these transcendental functions are beyond the scope of the IEEE 754 spec. and ergo are hardware implementation-dependent. In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin (as confirmed by an Intel engineer on a thread I came across while researching).
- The game employs a dirty-update system for rendering, not only that it's at a scanline granularity. This was something I was very pleased to see, as it's a exceedingly rare to see such an important optimization in games of this era (although common in earlier eras)
- There's some "interesting" naming conventions, one being prefixing member variables names with "value" -- there's even a "valueValue". Very little consistency in general in this regard, a reflection of independence between teams working on different components.
- While there are attempts at validation of input into the simulation (albeit woefully inadequate), it relies on ad hoc inclusion of PID (player ID) fields within commands. This is entirely useless, as this information is not authoritative and controlled by the players, permitting them to "spoof" the contextual information required for validation.
This is one of the more perplexing aspects of the engine, especially given the necessary information about the origin of a command is ofcourse available.
(As this information was later publicly published by other individuals I don't see a problem with elaborating on it here as I have)
- An example of missing the wood for the trees: session information goes through an ad hoc compression for its wire form (just bitpacks fields) to conserve bandwidth, however the architectural choice is to synchronize this session state by just having the host broadcast the state --dirty or not-- every 200ms, flooding the network pointlessly (atleast in the 18.8k days)
- While Age of Empires is somewhat notorious for its poor multiplayer performance (notably when contrasted with say, the recent "Definitive Edition"), and while its implementation of this synchronization model is certainly rather juvenile, it should be understood that the final MP gameplay issues are primarily due to the choice of a peer-to-peer topology over the public internet, which at the time was the most reasonable.
While superficially P2P may seem like it should achieve the lowest latencies for instance, the reality is that the primary determinant is the characteristics --jitter, delays and packet-loss, reordering, ect-- of the path between two hosts and a P2P architecture means there's n*n paths (network egress/ingress paths are typically asymmetric). In contrast, a server/client model you not only have far fewer routes, but datacenters are located at critical points in the network, roughly analogous to comparing travelling A->B along a freeway versus via the maze of residential streets.
- The state checksum "algorithms" involved are bordering on useless, atleast for state-tracing (such as when debugging desync. bugs.). They appear to have been devised by way of believing doing "a bunch of random bitwise ops" constitutes sufficient mixing -- a quick test demonstrated that csum collisions were not just possible, but occured sometimes for over 80% of inputs.
--
There's a lot more that could be said but I feel that's enough for now.
Sending state updates every 200ms made my spidey sense tingle. Most industrial control systems of the 90s would work on continuous 200ms polling through the network. Mostly RS485 links at 9600 baud.
The thing is that this is perfectly reasonable if your network infrastructure is known. If you have fixed bandwidth and deterministic packet sizes, then you can do math and know what the behavior will be. Determinism is good! Also this assumes the network is single purpose. Which it was! For games in the 90s it was! This isn’t bad design, it was good design for the network infrastructure people would have had at the time!
Certainly true for local and/or controlled networks, however over the public internet where you're competing for bandwidth with many others over limited (and sometimes already congested-) links it's a rather questionable choice.
Absolute left field question. Have you touched the other RTS contemporary of the time, Red Alert 2 in any detail?
I've been told the source code for it leaked some long time ago and has floated around for the past two decades or so. While most people contend that EA had lost the source code to the Command & Conquer series games many years ago when the studio was closed down and assets were shipped to "DICE" in Stockholm
RTS-wise it's just been Blizzard games otherwise; Starcraft and Warcraft III, the latter of which I did the most work with, mainly multiplayer optimizations and writing a server implementation.
However, I have been entertaining the idea of supporting other deterministic (RTS-) games, adapting them the same way --rearchitecting the multiplayer system/code-- as I have done for Age of Empires II, providing a unified platform for these games. The first candidate that came to mind was the Red Alert series.
I remember using some kind of resource explorer, looking into the scripts powering the AIs. If I remember correctly, it didn't get "smarter" at higher levels, it just cheated and gave itself more resources every minute, lol.
I've heard this repeated quite a bit, and while I haven't checked the AI scripts themselves (although I don't recall/imagine there being a script facility to give oneself resources..) I can say that, engine-wise, the only handicap behavior for AI that I've noticed is:
- On the highest difficulty ("Hardest"), AIs are bestowed 500 to all resources on next-age research completion.
- Work rates for researches are skewed to be slower for the lowest-difficulty setting.
> I'll note that I've restored AI being deterministic in my project.
Are you the OpenAge author, or maybe Voobly? I've never read such detailed info on AoE's internals online. Thanks for the post.
> In practice, contemporary Intel/AMD chip families produce bitwise the same result, however those around the time of AoE are known to diverge on results to some small margin
That little bit of validation I needed for why I avoid AMD CPUs. (not entirely serious)
No, I'm quite the outsider to the community having only been involved in the game for a few years (besides as a teenager when it was released) and working alone.
The project in question --a platform for AoE2 multiplayer /w general game improvements-- is currently under development, although the core game/MP has been functional for over a year now and our little beta-testing group plays regularly.
> There's some "interesting" naming conventions, one being prefixing member variables names with "value" -- there's even a "valueValue". Very little consistency in general in this regard, a reflection of independence between teams working on different components.
I enjoyed reading how they solved synch problem: schedule this moments commands for 2 turns from now, and the turn length is dynamic to smooth out experience.
I actually remembered a tidbit that will show my age as well. The first 28.8Kbps modem in the house was bought by my dad and he told me not to tell my mom that he got it because it cost around $500.
As early as the 2nd of January this year guys were commenting "Can we add (2022) to the title?" for articles that were authored only a couple of days before. It doesn't matter in this case if the article was written in 2001, 2011 or 2021 or even yesterday - the content and the context is clear, I cannot see anyone being confused or tricked by the original title.
Ah, I remember playing Quake death match at night in the late '90s on a university connection, against scores of opponents on dialup. It usually ended with somebody shouting "Everyone, go after that LPB" (Low-ping b**d). Then finally there was some real challenge :)
That's crazy I see this post. Haven't read it yet, but I just played a 3v3 game out of nostalgia. People still play this game years later, and it works great.
An RTS we developed just recently (Iron Harvest) is still based on the same basic principle. But instead of peer to peer we use a "smart" proxy server that receives the commands of all clients and sends the entire sequence back to the client. That way if one client has a high ping the other players latency isn't affected.
For those who don’t want to suffer from eye-bleeds due to ads, interstitial scrolling ads, more ads, and awful formatting, Googling the title yields this PDF:
I basically learned to code thanks to a few helpful and extremely patient IRC members in the Spring RTS engine[1] community (some of whom I spot in these comment sections on occasion - hi!), and deterministic lockstep simulation for online play was one of the first ideas that totally blew my mind when I learned about it ("b-but that means all the "random" numbers on every computer need to be the same! and so do all the floating point calculations across different hardware!"). Of course, to me at the time, that was deep magic that I thought I'd never be able to fully understand.
[1] see https://springrts.com/ - or check out http://zero-k.info/ and https://www.beyondallreason.info/ for games on the engine that are still pretty active