See books on Spice for a full explanation. If I have simplified too much, please send me an email.
The accuracy of a cursor readout of a waveform in a graph is a separate issue. But it affects our perception of Spice's accuracy so I talk about it at the end.
There is no method to directly solve a system of nonlinear circuit equations, so Spice uses an iterative process at each step in time of the Transient simulation. At each time step, it starts with the solution from the previous time step.
Spice's RELTOL parameter affects accuracy in two ways: convergence error and tracking error. It is commonly set in a range of 1E-2 to 1E-4 (default 1E-3) but could be set smaller.
RELTOL is used as a relative tolerance on the amount of change in node voltages from one iteration to the next. When the change of every node voltage is less than RELTOL * abs(new_value - old_value), the solution is considered to have converged to a numerically acceptable value and iteration stops. Tighter RELTOL = more iterations at each time point = more computational accuracy at that point.
Tracking (truncation) Error
There is error because the second order integration methods used in Transient analysis can exactly track a parabola but not circuit waveforms (an exponential or sine wave cannot be exactly represented by a second order polynomial). Spice calculates an estimate of this error for all capacitor currents and inductor voltages.
Spice uses RELTOL along with TRTOL and Spice's estimate of maximum tracking error to choose the size of the time step. If tracking error is too high, it shortens the time step and recomputes the solution. Tighter RELTOL = simulated waveform points have less tracking error from the ideal waveform.
Why not make RELTOL much tighter?
Some nonlinear circuits will fail to converge. Oops.
Slow simulation of nonlinear circuits.
Does not affect accuracy of fully settled portions (flat regions) of waveform.
We are talking accuracy when circuit node voltages are changing. If the node voltages have settled and are not changing, then the accuracy of simulation results will be orders of magnitude higher. Exception: a circuit with an integrator will store previous error.
a surprising Conclusion
Since RELTOL is a relative tolerance and since some circuits converge in just a few interations while others take many more, we can't know how close to final values we are when iteration stops! We just know the change from the last iteration was within RELTOL. So our basic calculating accuracy is somewhat suspect.
Additionally theory says Spice's integrating solution method cannot know how far it's waveform tracking is from what an engineer would call ideal. Mathematically we only know how to calculate a rough upper bound on the size of any error.
Finally circuits such as an integrator can accumulate these errors over time.
So there is no way to derive an exact accuracy number for our results from the Spice settings. (!)
One useful check: half the time step and see if results change.
Accuracy in a graph
Simulated points are connected with straight lines to make a graph. Consider a sine wave: the accuracy between simulated points will be poor due to the straight line connecting them.
In 5Spice, to get more closely spaced simulated points: enable the FINE dynamic time step, or shorten the "max time step", or use the Precision option (RELTOL=1E-4).
Default Spice accuracy settings
Spice defaults to a RELTOL setting of .001 and a TRTOL setting of 7.
Several Spice simulators, including 5Spice, default TRTOL to 1. TRTOL=1 gives shorter time steps with changing waveforms which makes sine waves look better (but not perfect).
5Spice's takeout style
5Spice provides checkbox style convergence options to modify the values of RELTOL and TRTOL. And a wizard to set VNTOL and ABSTOL to match the circuit.