Client side prediction rigidbody player desyncing

I have a rigid body player that is a rolling ball. As i roll the player around using applied physics velocities the client starts to drift from the location on the server. I have attempted to reconcile the position, rotation, velocity, and angular velocity, but doing so is creating jittering.

I feel i am missing something, or doing something below incorrectly.

below is my current configuration :

Here is my code for reconciliation.

I’d be interested in finding out the solution to this too :thinking:

Hey happymask!

I think this is a pretty good setup, it might just need some slight adjustments. Here are the changes I would consider.

First, the client is predicting, and that means the value received from the Simulator is actually from the past. For example, if the client starts moving on frame #1, the input for this movement might arrive on the Simulator at frame #3. Simulator applies that input and moves the player to the same spot as predicted, then sends the position sample to the client. When that sample arrives on the client, it might already be frame #6! Our prediction however was correct, it’s just that we should compare it with the historical state at frame #1, not the current predicted state as of frame #6.

In order to achieve that client needs to keep historical states. It could be as simple as:

struct PlayerState {
  public long SimulationFrame;
  public Vector3 Position;
  public Quaternion Rotation;
  public Vector3 Velocity;
  public Vector3 AngularVelocity;
}

List<PlayerState> historicalStates = new();

At the end of every frame we then create the PlayerState and add it to the list with the current simulation frame.

Given that, we’re still left with one small problem. That is, when receiving a sample, we have the simulation frame that tells us when this change was produced on the Simulator. That would be frame #3 from our example. What we need to know however, is what was the input simulation frame that was used to produce the sample (input from frame #1 in our sample).

One way to handle this would be to have long InputSimulationFrame as a synced variable on the simulated object. When Simulator applies input, it could read the frame of that input and assign it to the InputSimulationFrame. That way together with position, rotation, and others we’ll receive the frame of the input that produced this change.

Now we can finally put all of this together. After receiving new samples, we check the InputSimulationFrame. We search historicalStates looking for a PlayerState with a matching SimulationFrame. When we find one, we compare it with the received state. If it matches, the prediction was correct and we move on. If not, then we can start reconciling.

Some things to keep in mind:

  • When reconciling remember that the current state is most likely ahead of the frame at which the misprediction happened, so you need to resimulate things based on the mispredicted frame
  • The resimulated state doesn’t have to be applied immediately. You can reconcile smoothly by setting a “target” state and interpolating over multiple frames between the old, mispredicted state and the new resimulated state
  • When receiving samples through callbacks, not all samples could be applied at this point (e.g. we might have the new position but old velocity). It’s probably safest to wait for all the samples and then do a single misprediction check

Hope that helps!

1 Like

Cheers for the very detailed response! i appreciate you putting the time in :slight_smile:

Quick noob question. when i am waiting for all the samples before doing my misprediction check, is it a case of checking incoming OnNetworkSampleReceived callbacks until I have all data for the same simulationFrame? Or is there a more elegant way I can do so?

You could use the OnValueSynced attribute to get notified when the InputSimulationFrame changes. That is guaranteed to execute after all the data gets deserialized and so you can be sure to have the latest state of your object at that point.

The reason I would use this on the InputSimulationFrame field from the example is, other properties could stay the same if player is not moving, but the InputSimulationFrame should keep incrementing even if Simulator applies “empty” input.

Hey! another question. You mentioned " you need to resimulate things based on the mispredicted frame". How would you recommend I do this for my player once i determine a misprediction has occurred for a given frame?

to note as well. This is how i am moving the player. it is pure rigidbody physics.

private void RotateBall(MoveData md)
{
Vector3 ballDirection = new Vector3(md.Vertical, 0, -md.Horizontal);
cachedRigidbody.AddTorque(ballDirection * Speed, ForceMode.VelocityChange);
}

Hey Filip :slight_smile: I have been experimenting with your advice. but feel i am missing something in terms of reconciling. my result is still jittery and does not come to rest. this is likely because i am not using the correct flow or method for reconciling.

I also attempted to do a kinematic test, but the reconcile for that is causing my kinematic player to reconcile very visually, and also spiral out of control under network stress and rapid movements.

I’ll attach my project below where i am testing coherence if you want to take a look. this includes both my dynamic and kinematic test :

code for my dynamic player ball that is jittery :

using System.Collections.Generic;
using System.Linq;
using Coherence.Toolkit;
using Coherence.Toolkit.Bindings.TransformBindings;
using Coherence.Toolkit.Bindings.ValueBindings;
using Unity.Mathematics;
using UnityEngine;

namespace Samples.CSP.Ball
{
    struct RigidBodyState
    {
        public long SimulationFrame;
        public Vector3 Position;
        public Quaternion Rotation;
        public Vector3 Velocity;
        public Vector3 AngularVelocity;

        public RigidBodyState(Rigidbody rb, long simulationFrame)
        {
            SimulationFrame = simulationFrame;
            Position = rb.position;
            Rotation = rb.rotation;
            Velocity = rb.velocity;
            AngularVelocity = rb.angularVelocity;
        }

        public bool NaiveTestForReconcile(RigidBodyState other)
        {
            return math.distance(Position, other.Position) > 0.1f;
        }

        public void ApplyTo(Rigidbody rb)
        {
            rb.position = Position;
            rb.rotation = Rotation;
            rb.transform.position = Position;
            rb.transform.rotation = Rotation;
            rb.velocity = Velocity;
            rb.angularVelocity = AngularVelocity;
        }
    }

    public class PlayerBall : MonoBehaviour
    {
        private struct MoveData
        {
            public float Horizontal;
            public float Vertical;

            private const float MoveThreshold = 0.01f;
            public bool IsMoving => math.abs(Horizontal) >= MoveThreshold || math.abs(Vertical) >= MoveThreshold;

            public void CopyTo(ref CoherenceInput coInput)
            {
                coInput.SetAxis("Horizontal", Horizontal);
                coInput.SetAxis("Vertical", Vertical);
            }

            public void CopyFrom(CoherenceInput coInput)
            {
                Horizontal = coInput.GetAxis("Horizontal");
                Vertical = coInput.GetAxis("Vertical");
            }
        }

        public float Speed = 10f;

        /*
         *  Should should be called after all other relevent info has been synced. if what we are reconciling becomes more complex, we may not be able to rely on this be called after all other info has been synced.
         *
         *  "You could use the OnValueSynced attribute to get notified when the InputSimulationFrame changes. That is guaranteed to execute after all the data gets deserialized and so you can be sure to have the latest state of your object at that point.
         *  The reason I would use this on the InputSimulationFrame field from the example is, other properties could stay the same if player is not moving, but the InputSimulationFrame should keep incrementing even if Simulator applies “empty” input." - Filip - https://community.coherence.io/t/client-side-prediction-rigidbody-player-desyncing/458/4
         */
        [OnValueSynced(nameof(DetectReconcile))] 
        public long InputSimulationFrame;

        private const int MAX_HISTORY_LENGTH = 1000;
        private List<RigidBodyState> historicalStates = new();
        private List<MoveData> historicalMoveData = new();
        private List<float> historicalDeltaTimes = new();

        private Rigidbody cachedRigidbody;

        private CoherenceInput coInput;
        private CoherenceSync coSync;
        CoherenceBridge bridge;

        public Vector3 syncedTestPosition;
        public Vector3 SyncedTestVelocity;
        public Vector3 SyncedTestAngularVelocity;
        public Quaternion SyncedTestRotation;

        private void Awake()
        {
            cachedRigidbody = GetComponent<Rigidbody>();
            coSync = GetComponent<CoherenceSync>();
            coInput = GetComponent<CoherenceInput>();
            bridge = GameObject.FindObjectOfType<CoherenceBridge>();

            if (coSync == null && coInput == null)
            {
                Debug.LogError("Missing Sync and/or Input components");
                gameObject.SetActive(false);
                return;
            }

            // set up bindings for syncing position, rotation, velocity, and angular velocity from the simulator
            var positionBinding = coSync.Bindings.First(b => b is PositionBinding) as PositionBinding;
            positionBinding.OnNetworkSampleReceived += (sample,_, _) => syncedTestPosition = (Vector3)sample;

            var velocityBinding = coSync.Bindings.First(b => b is Vector3Binding) as Vector3Binding;
            velocityBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestVelocity = (Vector3)sample;

            var angularVelocityBinding = coSync.Bindings.First(b => b is Vector3Binding) as Vector3Binding;
            angularVelocityBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestAngularVelocity = (Vector3)sample;

            var rotationBinding = coSync.Bindings.First(b => b is RotationBinding) as RotationBinding;
            rotationBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestRotation = (Quaternion)sample;

        }

void FixedUpdate()
        {
            MoveData movement = default;

            if (coSync.HasInputAuthority)
            {
                movement = GetMoveData();
                movement.CopyTo(ref coInput);
            }

            if (coSync.HasStateAuthority || coSync.HasInputAuthority)
            {
                if (!coSync.HasInputAuthority)
                {
                    movement.CopyFrom(coInput);
                    InputSimulationFrame = coInput.CurrentSimulationFrame;
                }

                if (movement.IsMoving)
                {
                    RotateBall(movement);
                }
            }

            CacheState(movement, Time.fixedDeltaTime);
        }

        private MoveData GetMoveData()
        {
            return new MoveData()
            {
                Horizontal = Input.GetAxis("Horizontal"),
                Vertical = Input.GetAxis("Vertical")
            };
        }

        private void RotateBall(MoveData md)
        {
            ApplyMoveData(cachedRigidbody, md);
        }

        private void ApplyMoveData(Rigidbody rb, MoveData md)
        {
            Vector3 ballDirection = new Vector3(md.Vertical, 0, -md.Horizontal);

            rb.AddTorque(ballDirection * Speed, ForceMode.VelocityChange);
        }

        private void CacheState(MoveData inMoveData, float dt)
        {
            //historicalStates.Add(new RigidBodyState(cachedRigidbody, coInput.CurrentSimulationFrame)); // cant find this frame when i use coInput.CurrentSimulationFrame.
            historicalStates.Add(new RigidBodyState(cachedRigidbody, bridge.NetworkTime.ClientSimulationFrame));

            historicalMoveData.Add(inMoveData);
            historicalDeltaTimes.Add(dt);

            if (historicalStates.Count > MAX_HISTORY_LENGTH)
            {
                historicalStates.RemoveAt(0);
                historicalMoveData.RemoveAt(0);
                historicalDeltaTimes.RemoveAt(0);
            }
        }

        private bool GetStateAtFrame(long simulationFrame, out RigidBodyState state, out int index)
        {
            for (int i = historicalStates.Count - 1; i >= 0; i--)
            {
                if (historicalStates[i].SimulationFrame == simulationFrame)
                {
                    state = historicalStates[i];
                    index = i;
                    return true;
                }
            }

            state = default;
            index = -1;
            return false;
        }

        private void DetectReconcile(long oldInputSimulationFrame, long newInputSimulationFrame)
        {
            Debug.Log("Test for reconcile");

            InputSimulationFrame = newInputSimulationFrame;

            if (!GetStateAtFrame(InputSimulationFrame, out RigidBodyState clientSideState, out int index))
            {
                Debug.Log("Test for reconcile : failed, could not find frame");
                return;
            }

            // construct a state from the synced data at the synced frame
            RigidBodyState simulationSideState = new RigidBodyState()
            {
                Position = syncedTestPosition,
                Rotation = SyncedTestRotation,
                Velocity = SyncedTestVelocity,
                AngularVelocity = SyncedTestAngularVelocity,
                SimulationFrame = InputSimulationFrame
            };

            // compare the cliend side state from the same frame with the state constructed from the synced data. if we detect a difference, we need to reconcile.
            if (clientSideState.NaiveTestForReconcile(simulationSideState))
            {
                // correct the client side state to match the synced state
                historicalStates[index] = simulationSideState;

                // reconcile from the frame we detected the difference
                Reconcile(index);
            }
        }

        private void Reconcile(int fromIndex)
        {
            Debug.Log("Reconciling");

            RigidBodyState historicalState = historicalStates[fromIndex];

            // apply the historical state to the rigidbody, so we are in the same state as we were at the frame we are reconciling from.
            historicalState.ApplyTo(cachedRigidbody);

            // loop through the historical states and apply the move data to the rigidbody, then simulate the physics to the frame we are reconciling to.
            for (int i = fromIndex; i < historicalStates.Count; i++)
            {
                MoveData historicalMoveData = this.historicalMoveData[i];
                ApplyMoveData(cachedRigidbody, historicalMoveData);

                Physics.Simulate(historicalDeltaTimes[i]);

                historicalStates[i] = new RigidBodyState(cachedRigidbody, historicalStates[i].SimulationFrame);
            }
        }
    }
}

i have been experimenting in my kinematic test scene. I am drawing the frame positions for my character and comparing between the client, the server, and the reconciled position. In the pic below i have:

  • blue being the marked position of the client locally for a frame,

  • red for the server position for a given frame

  • yellow for reconciled frames

  • red lines connect client frames to reconciled and server frames

the left hand side is the connected client, the right hand side is the server.

in the picture i went straight down and then turned left, this should have been a perfect 90 degree angle. But instead what i am getting is the server turning much later than the client, and the client being reconciled multiple times as the server continues down, while the client tries to travel left. I feel as if i have something not configured correctly to be making the server overshoot its inputs in the manner that i am seeing.

Hey!

What I think might be missing is some input buffering on the Simulator side. In order for the simulation to flow nicely on both sides both Client and the Simulator must execute the same stream of inputs.

Both Client and the Simulator are trying to stay as close to the Replication Server’s frame as possible. That in turn means, that if Client sends input at frame #10, this input might arrive on the Simulator at frame #12. Input sent at frame #11 however might arrive at frame #14 creating a gap.

In order to mitigate this issue, Simulator shouldn’t immediately use all the inputs that arrive from the client. Instead, it should buffer some inputs so when there’s a gap, it can still proceed without guessing. How many inputs to buffer depends on the latency and stability of the client’s connection, usually it’s 1-2 when the connection is good.

In essence it means that the Simulator should use coInput.GetAxis("Horizontal", currentSimulationFrame - numInputsToBuffer). Or better yet, use the coInput.LastReceivedFrame to track what was the last available input form the client, and use that to determine which input to use. In the absence of inputs the Simulator can just use the old input, i.e. assume that the input didn’t change (it must then however skip the next received input so the total number of inputs produced and consumed stays the same).

2 Likes

Hey Filip :slight_smile: So i tried what you suggested. as it seems to give the same effect as the ‘initial input delay’ on the coherence input component. The best result i can get is with.

initial input delay = 3

and a 3 frame delay on the reconcile frame.


        private void Reconcile(long oldInputSimulationFrame, long newInputSimulationFrame)
        {
            const long frameDelay = 3;
            InputSimulationFrame = newInputSimulationFrame - frameDelay;

            InputData cachedInputData = GetInputState(InputSimulationFrame);
            SimulationState cachedStateData = GetSimulationState(InputSimulationFrame);

However the moment i add network stress it reverts back to being jittery. is what i am showing here a lead to a possible solution? in terms of delaying the reconcile frame.

Here is my full code for my kinematic solution.

using Coherence.Toolkit;
using Coherence.Toolkit.Bindings.TransformBindings;
using System.Linq;
using Unity.Mathematics;
using UnityEngine;

namespace Samples.CSP.BasicKinematicPlayer
{

    public class SimulationState
    {
        public long SimulationFrame;
        public Vector3 Position;

        public void ApplyTo(Transform transform)
        {
            transform.position = Position;
        }

        public bool NaiveTestForReconcile(SimulationState other)
        {
            return math.distance(Position, other.Position) > 0.01f;
        }
    }


    public class InputData
    {
        public float Horizontal;
        public float Vertical;

        private const float MoveThreshold = 0.01f;
        public bool IsMoving => math.abs(Horizontal) >= MoveThreshold || math.abs(Vertical) >= MoveThreshold;

        public void CopyTo(ref CoherenceInput coInput)
        {
            coInput.SetAxis("Horizontal", Horizontal);
            coInput.SetAxis("Vertical", Vertical);
        }

        public void CopyFrom(CoherenceInput coInput)
        {
            const int numInputsToBuffer = 0;
            Horizontal = coInput.GetAxis("Horizontal", coInput.CurrentSimulationFrame - numInputsToBuffer);
            Vertical = coInput.GetAxis("Vertical", coInput.CurrentSimulationFrame - numInputsToBuffer);
        }
    }

    public class BasicKinematicPlayer : MonoBehaviour
    {
        private const int STATE_CACHE_SIZE = 1024;
        private SimulationState[] simulationHistory = new SimulationState[STATE_CACHE_SIZE];
        private InputData[] inputHistory = new InputData[STATE_CACHE_SIZE];

        private CoherenceInput coInput;
        private CoherenceSync coSync;

        private InputData cachedMovement;

        [OnValueSynced(nameof(Reconcile))]
        public long InputSimulationFrame;

        public long LastCorrectFrame;

        public float Speed = 1f;
        public Vector3 syncedTestPosition;

        void Awake()
        {
            coSync = GetComponent<CoherenceSync>();
            coInput = GetComponent<CoherenceInput>();

            var positionBinding = coSync.Bindings.First(b => b is PositionBinding) as PositionBinding;
            positionBinding.OnNetworkSampleReceived += (sample, _, _) => syncedTestPosition = (Vector3)sample;
        }

        private void Update()
        {
            if (coSync.HasInputAuthority)
            {
                cachedMovement = GetMoveData();
                cachedMovement.CopyTo(ref coInput);
            }
        }

        private InputData GetMoveData()
        {
            return new InputData()
            {
                Horizontal = Input.GetKey(KeyCode.A) ? -1 : Input.GetKey(KeyCode.D) ? 1 : 0,
                Vertical = Input.GetKey(KeyCode.S) ? -1 : Input.GetKey(KeyCode.W) ? 1 : 0
            };
        }

        private void FixedUpdate()
        {
            InputData movement = cachedMovement == null ? new InputData() : cachedMovement;

            long simulationFrame = coInput.CurrentSimulationFrame;

            if (coSync.HasStateAuthority || coSync.HasInputAuthority)
            {
                if (!coSync.HasInputAuthority)
                {
                    movement.CopyFrom(coInput);
                    InputSimulationFrame = simulationFrame;
                }

                MovePlayer(movement);
            }

            if (coSync.HasInputAuthority)
            {
                CacheSimulationState(CurrentSimulationState(simulationFrame), simulationFrame);
                CacheInputState(movement, simulationFrame);
            }
        }

        public SimulationState CurrentSimulationState(long inSimulationFrame)
        {
            return new SimulationState
            {
                Position = transform.position,
                SimulationFrame = inSimulationFrame
            };
        }

        public SimulationState CurrentSimulationState()
        {
            return new SimulationState
            {
                Position = transform.position,
                SimulationFrame = 0
            };
        }

        private void CacheSimulationState(SimulationState state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            simulationHistory[cacheIndex] = state;
        }
        private void CacheInputState(InputData state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            inputHistory[cacheIndex] = state;
        }

        private SimulationState GetSimulationState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return simulationHistory[cacheIndex];
        }

        private InputData GetInputState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return inputHistory[cacheIndex];
        }

        private void MovePlayer(InputData md)
        {
            Vector3 movement = new Vector3(md.Horizontal, 0, md.Vertical);
            transform.position += movement * Speed;
        }

        private void Reconcile(long oldInputSimulationFrame, long newInputSimulationFrame)
        {
            const long frameDelay = 3;
            InputSimulationFrame = newInputSimulationFrame - frameDelay;

            InputData cachedInputData = GetInputState(InputSimulationFrame);
            SimulationState cachedStateData = GetSimulationState(InputSimulationFrame);

            if (cachedInputData == null || cachedStateData == null)
            {
                transform.position = syncedTestPosition;
                LastCorrectFrame = InputSimulationFrame;
                return;
            }

            SimulationState serverState = new SimulationState
            {
                Position = syncedTestPosition,
                SimulationFrame = InputSimulationFrame
            };

            if (cachedStateData.NaiveTestForReconcile(serverState))
            {
                transform.position = serverState.Position;

                long rewindFrame = InputSimulationFrame;
                long simulationFrame = coInput.CurrentSimulationFrame;

                while (rewindFrame < simulationFrame)
                {
                    // Determine the cache index 
                    long rewindCacheIndex = rewindFrame % STATE_CACHE_SIZE;

                    // Obtain the cached input and simulation states.
                    InputData rewindCachedInputState = GetInputState(rewindCacheIndex);
                    SimulationState rewindCachedSimulationState = GetSimulationState(rewindCacheIndex);

                    // If there's no state to simulate, for whatever reason, 
                    // increment the rewindFrame and continue.
                    if (rewindCachedInputState == null || rewindCachedSimulationState == null)
                    {
                        ++rewindFrame;
                        continue;
                    }

                    // Process the cached inputs. 
                    MovePlayer(rewindCachedInputState);

                    // Replace the simulationStateCache index with the new value.
                    SimulationState rewoundSimulationState = CurrentSimulationState();
                    rewoundSimulationState.SimulationFrame = rewindFrame;
                    simulationHistory[rewindCacheIndex] = rewoundSimulationState;

                    // Increase the amount of frames that we've rewound.
                    ++rewindFrame;
                }
            }

            LastCorrectFrame = InputSimulationFrame;
        }
    }
}

Hey!

You’re right about the numInputsToBuffer, it’s very similar to the InputDelay. There is a small difference in that it is applied only on the Simulator and so does not induce any input lag on the client. It can also be dynamic to cover the bad connection scenario. In the local test scenario however the InputDelay should be good enough so there definitely is another issue.

I tried to use the project that you’ve provided earlier, but things don’t seem to work out of the box, so I started setting up a new project with the provided code (perhaps I missed something?). If it were possible to get a simple working sample that would help a lot!

I’m swamped with a lot of things right now, but know that I’m pushing this issue whenever I can, so you’re not left alone.

Hey Filip :slight_smile: cheers for getting back to me.

And it is not a problem, i have put a pin in this for now. But if you want to take a look, ill upload my entire project, and not just the assets, package, and project folders. I have tidied up my code as well for the dynamic and kinematic test. I’ll post everything below if you want to have a crack at it.

My samples are under :
\Assets\Samples\Dynamic and \Assets\Samples\Kinematic

Dynamic character code -

using System.Collections.Generic;
using System.Linq;
using Coherence.Toolkit;
using Coherence.Toolkit.Bindings.TransformBindings;
using Coherence.Toolkit.Bindings.ValueBindings;
using Unity.Mathematics;
using UnityEngine;

namespace Samples.CSP.BasicDynamicPlayer
{
    public class SimulationState
    {
        public long SimulationFrame;
        public Vector3 Position;
        public Quaternion Rotation;
        public Vector3 Velocity;
        public Vector3 AngularVelocity;

        public bool NaiveTestForReconcile(SimulationState other)
        {
            return math.distance(Position, other.Position) > 0.1f;
        }

        public void ApplyTo(Rigidbody rb)
        {
            rb.position = Position;
            rb.rotation = Rotation;
            rb.transform.position = Position;
            rb.transform.rotation = Rotation;
            rb.velocity = Velocity;
            rb.angularVelocity = AngularVelocity;
        }
    }

    public class InputData
    {
        public float Horizontal;
        public float Vertical;

        private const float MoveThreshold = 0.01f;
        public bool IsMoving => math.abs(Horizontal) >= MoveThreshold || math.abs(Vertical) >= MoveThreshold;

        public void CopyTo(ref CoherenceInput coInput)
        {
            coInput.SetAxis("Horizontal", Horizontal);
            coInput.SetAxis("Vertical", Vertical);
        }

        public void CopyFrom(CoherenceInput coInput)
        {
            Horizontal = coInput.GetAxis("Horizontal");
            Vertical = coInput.GetAxis("Vertical");
        }
    }

    public class BasicDynamicPlayer : MonoBehaviour
    {
        private const int STATE_CACHE_SIZE = 1024;
        private SimulationState[] simulationHistory = new SimulationState[STATE_CACHE_SIZE];
        private InputData[] inputHistory = new InputData[STATE_CACHE_SIZE];

        private Rigidbody cachedRigidbody;

        private CoherenceInput coInput;
        private CoherenceSync coSync;

        private InputData cachedMovement;

        public Vector3 syncedTestPosition;
        public Vector3 SyncedTestVelocity;
        public Vector3 SyncedTestAngularVelocity;
        public Quaternion SyncedTestRotation;

        public float Speed = 10f;

        [OnValueSynced(nameof(Reconcile))]
        public long InputSimulationFrame;
        public long LastCorrectFrame;


        private void Awake()
        {
            cachedRigidbody = GetComponent<Rigidbody>();
            coSync = GetComponent<CoherenceSync>();
            coInput = GetComponent<CoherenceInput>();

            if (coSync == null && coInput == null)
            {
                Debug.LogError("Missing Sync and/or Input components");
                gameObject.SetActive(false);
                return;
            }

            // set up bindings for syncing position, rotation, velocity, and angular velocity from the simulator
            var positionBinding = coSync.Bindings.First(b => b is PositionBinding) as PositionBinding;
            positionBinding.OnNetworkSampleReceived += (sample,_, _) => syncedTestPosition = (Vector3)sample;

            var velocityBinding = coSync.Bindings.First(b => b is Vector3Binding) as Vector3Binding;
            velocityBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestVelocity = (Vector3)sample;

            var angularVelocityBinding = coSync.Bindings.First(b => b is Vector3Binding) as Vector3Binding;
            angularVelocityBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestAngularVelocity = (Vector3)sample;

            var rotationBinding = coSync.Bindings.First(b => b is RotationBinding) as RotationBinding;
            rotationBinding.OnNetworkSampleReceived += (sample, _, _) => SyncedTestRotation = (Quaternion)sample;

        }
        private void Update()
        {
            if (coSync.HasInputAuthority)
            {
                cachedMovement = GetMoveData();
                cachedMovement.CopyTo(ref coInput);
            }
        }

        private void FixedUpdate()
        {
            InputData movement = cachedMovement == null ? new InputData() : cachedMovement;

            long simulationFrame = coInput.CurrentSimulationFrame;

            if (coSync.HasStateAuthority || coSync.HasInputAuthority)
            {
                if (!coSync.HasInputAuthority)
                {
                    movement.CopyFrom(coInput);
                    InputSimulationFrame = simulationFrame;
                }

                MovePlayer(cachedRigidbody, movement);
            }

            if (coSync.HasInputAuthority)
            {
                CacheSimulationState(CurrentSimulationState(simulationFrame), simulationFrame);
                CacheInputState(movement, simulationFrame);
            }
        }

        private InputData GetMoveData()
        {
            return new InputData()
            {
                Horizontal = Input.GetKey(KeyCode.A) ? -1 : Input.GetKey(KeyCode.D) ? 1 : 0,
                Vertical = Input.GetKey(KeyCode.S) ? -1 : Input.GetKey(KeyCode.W) ? 1 : 0
            };
        }

        private void MovePlayer(Rigidbody rb, InputData md)
        {
            Vector3 ballDirection = new Vector3(md.Vertical, 0, -md.Horizontal);

            rb.AddTorque(ballDirection * Speed, ForceMode.VelocityChange);
        }

        private void CacheSimulationState(SimulationState state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            simulationHistory[cacheIndex] = state;
        }

        private void CacheInputState(InputData state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            inputHistory[cacheIndex] = state;
        }

        public SimulationState CurrentSimulationState(long inSimulationFrame)
        {
            return new SimulationState
            {
                Position = transform.position,
                Rotation = transform.rotation,
                Velocity = cachedRigidbody.velocity,
                AngularVelocity = cachedRigidbody.angularVelocity,
                SimulationFrame = inSimulationFrame
            };
        }

        public SimulationState CurrentSimulationState()
        {
            return new SimulationState
            {
                Position = transform.position,
                Rotation = transform.rotation,
                Velocity = cachedRigidbody.velocity,
                AngularVelocity = cachedRigidbody.angularVelocity,
                SimulationFrame = 0
            };
        }

        private SimulationState GetSimulationState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return simulationHistory[cacheIndex];
        }

        private InputData GetInputState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return inputHistory[cacheIndex];
        }

        private void Reconcile(long oldInputSimulationFrame, long newInputSimulationFrame)
        {
            InputSimulationFrame = newInputSimulationFrame;

            InputData cachedInputData = GetInputState(InputSimulationFrame);
            SimulationState cachedStateData = GetSimulationState(InputSimulationFrame);

            if (cachedInputData == null || cachedStateData == null)
            {
                transform.position = syncedTestPosition;
                LastCorrectFrame = InputSimulationFrame;
                return;
            }

            SimulationState serverState = new SimulationState()
            {
                Position = syncedTestPosition,
                Rotation = SyncedTestRotation,
                Velocity = SyncedTestVelocity,
                AngularVelocity = SyncedTestAngularVelocity,
                SimulationFrame = InputSimulationFrame
            };

            if (cachedStateData.NaiveTestForReconcile(serverState))
            {
                transform.position = serverState.Position;

                long rewindFrame = InputSimulationFrame;
                long simulationFrame = coInput.CurrentSimulationFrame;

                while (rewindFrame <= simulationFrame)
                {
                    // Determine the cache index 
                    long rewindCacheIndex = rewindFrame % STATE_CACHE_SIZE;

                    // Obtain the cached input and simulation states.
                    InputData rewindCachedInputState = GetInputState(rewindCacheIndex);
                    SimulationState rewindCachedSimulationState = GetSimulationState(rewindCacheIndex);

                    // If there's no state to simulate, for whatever reason, 
                    // increment the rewindFrame and continue.
                    if (rewindCachedInputState == null || rewindCachedSimulationState == null)
                    {
                        ++rewindFrame;
                        continue;
                    }

                    // Process the cached inputs. 
                    MovePlayer(cachedRigidbody, rewindCachedInputState);

                    Physics.Simulate(Time.fixedDeltaTime);

                    // Replace the simulationStateCache index with the new value.
                    SimulationState rewoundSimulationState = CurrentSimulationState();
                    rewoundSimulationState.SimulationFrame = rewindFrame;
                    simulationHistory[rewindCacheIndex] = rewoundSimulationState;

                    // Increase the amount of frames that we've rewound.
                    ++rewindFrame;
                }
            }

            LastCorrectFrame = InputSimulationFrame;
        }
    }
}

Kinematic character code -

using Coherence.Toolkit;
using Coherence.Toolkit.Bindings.TransformBindings;
using System.Linq;
using Unity.Mathematics;
using UnityEngine;

namespace Samples.CSP.BasicKinematicPlayer
{

    public class SimulationState
    {
        public long SimulationFrame;
        public Vector3 Position;

        public void ApplyTo(Transform transform)
        {
            transform.position = Position;
        }

        public bool NaiveTestForReconcile(SimulationState other)
        {
            return math.distance(Position, other.Position) > 0.01f;
        }
    }


    public class InputData
    {
        public float Horizontal;
        public float Vertical;

        private const float MoveThreshold = 0.01f;
        public bool IsMoving => math.abs(Horizontal) >= MoveThreshold || math.abs(Vertical) >= MoveThreshold;

        public void CopyTo(ref CoherenceInput coInput)
        {
            coInput.SetAxis("Horizontal", Horizontal);
            coInput.SetAxis("Vertical", Vertical);
        }

        public void CopyFrom(CoherenceInput coInput)
        {
            const int numInputsToBuffer = 0;
            Horizontal = coInput.GetAxis("Horizontal", coInput.CurrentSimulationFrame - numInputsToBuffer);
            Vertical = coInput.GetAxis("Vertical", coInput.CurrentSimulationFrame - numInputsToBuffer);
        }
    }

    public class BasicKinematicPlayer : MonoBehaviour
    {
        private const int STATE_CACHE_SIZE = 1024;
        private SimulationState[] simulationHistory = new SimulationState[STATE_CACHE_SIZE];
        private InputData[] inputHistory = new InputData[STATE_CACHE_SIZE];

        private CoherenceInput coInput;
        private CoherenceSync coSync;

        private InputData cachedMovement;

        [OnValueSynced(nameof(Reconcile))]
        public long InputSimulationFrame;

        public long LastCorrectFrame;

        public float Speed = 1f;
        public Vector3 syncedTestPosition;

        void Awake()
        {
            coSync = GetComponent<CoherenceSync>();
            coInput = GetComponent<CoherenceInput>();

            var positionBinding = coSync.Bindings.First(b => b is PositionBinding) as PositionBinding;
            positionBinding.OnNetworkSampleReceived += (sample, _, _) => syncedTestPosition = (Vector3)sample;
        }

        private void Update()
        {
            if (coSync.HasInputAuthority)
            {
                cachedMovement = GetMoveData();
                cachedMovement.CopyTo(ref coInput);
            }
        }

        private InputData GetMoveData()
        {
            return new InputData()
            {
                Horizontal = Input.GetKey(KeyCode.A) ? -1 : Input.GetKey(KeyCode.D) ? 1 : 0,
                Vertical = Input.GetKey(KeyCode.S) ? -1 : Input.GetKey(KeyCode.W) ? 1 : 0
            };
        }

        private void FixedUpdate()
        {
            InputData movement = cachedMovement == null ? new InputData() : cachedMovement;

            long simulationFrame = coInput.CurrentSimulationFrame;

            if (coSync.HasStateAuthority || coSync.HasInputAuthority)
            {
                if (!coSync.HasInputAuthority)
                {
                    movement.CopyFrom(coInput);
                    InputSimulationFrame = simulationFrame;
                }

                MovePlayer(movement);
            }

            if (coSync.HasInputAuthority)
            {
                CacheSimulationState(CurrentSimulationState(simulationFrame), simulationFrame);
                CacheInputState(movement, simulationFrame);
            }
        }

        public SimulationState CurrentSimulationState(long inSimulationFrame)
        {
            return new SimulationState
            {
                Position = transform.position,
                SimulationFrame = inSimulationFrame
            };
        }

        public SimulationState CurrentSimulationState()
        {
            return new SimulationState
            {
                Position = transform.position,
                SimulationFrame = 0
            };
        }

        private void CacheSimulationState(SimulationState state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            simulationHistory[cacheIndex] = state;
        }
        private void CacheInputState(InputData state, long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            inputHistory[cacheIndex] = state;
        }

        private SimulationState GetSimulationState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return simulationHistory[cacheIndex];
        }

        private InputData GetInputState(long simulationFrame)
        {
            long cacheIndex = simulationFrame % STATE_CACHE_SIZE;
            return inputHistory[cacheIndex];
        }

        private void MovePlayer(InputData md)
        {
            Vector3 movement = new Vector3(md.Horizontal, 0, md.Vertical);
            transform.position += movement * Speed;
        }

        private void Reconcile(long oldInputSimulationFrame, long newInputSimulationFrame)
        {
            InputSimulationFrame = newInputSimulationFrame;

            InputData cachedInputData = GetInputState(InputSimulationFrame);
            SimulationState cachedStateData = GetSimulationState(InputSimulationFrame);

            if (cachedInputData == null || cachedStateData == null)
            {
                transform.position = syncedTestPosition;
                LastCorrectFrame = InputSimulationFrame;
                return;
            }

            SimulationState serverState = new SimulationState
            {
                Position = syncedTestPosition,
                SimulationFrame = InputSimulationFrame
            };

            if (cachedStateData.NaiveTestForReconcile(serverState))
            {
                transform.position = serverState.Position;

                long rewindFrame = InputSimulationFrame;
                long simulationFrame = coInput.CurrentSimulationFrame;

                while (rewindFrame <= simulationFrame)
                {
                    // Determine the cache index 
                    long rewindCacheIndex = rewindFrame % STATE_CACHE_SIZE;

                    // Obtain the cached input and simulation states.
                    InputData rewindCachedInputState = GetInputState(rewindCacheIndex);
                    SimulationState rewindCachedSimulationState = GetSimulationState(rewindCacheIndex);

                    // If there's no state to simulate, for whatever reason, 
                    // increment the rewindFrame and continue.
                    if (rewindCachedInputState == null || rewindCachedSimulationState == null)
                    {
                        ++rewindFrame;
                        continue;
                    }

                    // Process the cached inputs. 
                    MovePlayer(rewindCachedInputState);

                    // Replace the simulationStateCache index with the new value.
                    SimulationState rewoundSimulationState = CurrentSimulationState();
                    rewoundSimulationState.SimulationFrame = rewindFrame;
                    simulationHistory[rewindCacheIndex] = rewoundSimulationState;

                    // Increase the amount of frames that we've rewound.
                    ++rewindFrame;
                }
            }

            LastCorrectFrame = InputSimulationFrame;
        }
    }
}

Hey!

I’m sorry it took so long. I’ve made all the changes required to make this work. I’ve also simplified the project a little bit to make it easier for me to navigate it. None of the removed code was causing issues so it can be safely reintroduced.

There were problems with reconciliation due to a handful of small issues:

  • Movement using direct input data instead of CoherenceInput data when the delay was turned on
  • Reconciliation happening immediately as part of the “sample received” callback
  • Input frame getting shifted due to the FixedUpdate not aligning with FixedNetworkUpdate
  • Interpolation enabled on the Position binding (that shouldn’t matter with prediction turned on so we’ll be investigating this)

Project files

Notes on the project:

  • Fixes were applied only to the kinematic setup. I imagine the dynamic one will be similar
  • I’ve set it up to test things in “split screen”. Just load both BasicKinematicPlayer_Client and BasicKinematicPlayer_Simulator scenes at the same time (left side is client, right side is simulator)
    image
  • Reconciliation triggers a warning
  • You can enable movement logs by setting the Log Movement to true. This will print an info log every time the client and simulator move the player object
    image
  • You can tweak the Inputs Buffered to make the simulator delay applying inputs even further (allowing for a more unstable connection) at the cost of the player’s experience

Hope that helps!