UQ-Loc: Teaching LiDAR Localisation to Say How Sure It Is

Paper: 2608.06307 Authors: Jacek Komorowski Categories: cs.CV

The Gap

Scene Coordinate Regression (SCR) is the “no map database” branch of localisation. Instead of retrieving a nearby keyframe and matching features, you train a network that eats sensor data and directly spits out, for each input element, where that element lives in the global scene frame. You then solve for the 6-DoF pose from those 2D-3D or 3D-3D correspondences. In images this lineage runs DSAC / DSAC* to ACE to GLACE. On LiDAR it runs SGLoc to LightLoc, which is the direct parent of this paper: a sparse-convolutional backbone that maps each voxel of a point cloud to a global 3D coordinate, then hands the correspondence set to a registration solver like SC2-PCR.

The boundary these methods stop at is this: every predicted coordinate is a bare point. No error bars. That has two costs. First, the solver is forced to treat a voxel on a distant, sparsely sampled façade exactly like a voxel on a nearby, richly sampled kerb — RANSAC-style inlier tests use a single global distance threshold in metres, which is a statement that all errors are identically distributed and isotropic. They are not. LiDAR error is famously anisotropic: uncertainty along the ray differs from uncertainty across it, and a voxel sitting on a large planar wall is well-constrained perpendicular to the wall but nearly free to slide within it. Second, downstream consumers (a Kalman filter, a pose graph, a planner deciding whether to trust the localiser) want a covariance, and SCR gives them nothing to fuse.

The obvious patch — attach a learned scalar confidence — has been tried in the visual SCR world and it throws away exactly the structure that matters. UQ-Loc’s claim is that the right object is a full anisotropic covariance, and that once you have it you should push it all the way through the solver rather than just reporting it.

[Problem] LiDAR SCR predicts bare 3D points per voxel
     |     no error bars -> solver treats all voxels alike
     |     downstream filters get nothing to fuse
     v
[Assumption] per-voxel error is (a) anisotropic, shaped by
     |       local geometry, and (b) spatially smooth
     v
[Method] add 3x3 covariance head (Cholesky-parameterised)
     |   train with Gaussian NLL + kNN smoothness prior
     |
     +--> [Solver] SC2-PCR with uncertainty-weighted seed
     |             scoring + Mahalanobis inlier test
     v
[Evidence] lower 6-DoF pose error  AND  low ECE
     |     (accuracy gain + calibration, not a trade)
     v
[Conclusion] uncertainty is not a bolt-on report; it is
             an input to geometric estimation

The Increment

One sentence: Before, a LiDAR SCR network told you where each voxel is and the solver had to guess who to believe; after, it tells you where each voxel is *and the shape of its ignorance, and the solver uses that shape to decide who to believe.

Core Mechanism

Architecturally it is a second head. The LightLoc sparse-conv backbone already produces a per-voxel feature; the coordinate head turns that into a 3-vector mu. UQ-Loc adds a parallel head producing 6 numbers, which are assembled into a lower-triangular matrix L (diagonal passed through something positive like softplus or exp), and the covariance is Sigma = L L^T plus a small ridge for numerical safety. This Cholesky trick is what makes the output guaranteed positive-definite no matter what the network emits — you never have to project or clamp an invalid matrix.

Training swaps the L1/L2 coordinate loss for the multivariate Gaussian negative log-likelihood: roughly d^T Sigma^-1 d + log det Sigma, where d = mu - ground truth. The first term says “your ellipsoid must cover your error”; the second says “but don’t just inflate it to infinity”. The equilibrium is honesty. The catch is that raw NLL is unstable and permissive — the network can shrink loss by declaring confusing voxels hopeless, and there is nothing in the per-voxel objective tying neighbouring voxels together. So the paper adds a kNN spatial smoothness regulariser: find each voxel’s nearest neighbours in the point cloud and penalise disagreement between their predicted covariances. Physically this encodes that uncertainty is a property of local surface geometry, which does not flip character between adjacent voxels on the same wall.

At inference the covariance stops being decoration and becomes solver input. SC2-PCR normally picks “seed” correspondences by a compatibility score and then counts inliers with a fixed metric radius. UQ-Loc changes both: seed scoring is weighted so that tight-covariance voxels are preferred as anchors, and the inlier test becomes a Mahalanobis test — residuals are measured in units of that voxel’s own predicted standard deviation, in the direction they actually occur. A 30 cm residual perpendicular to a wall is a hard reject; the same 30 cm sliding along the wall may be entirely consistent. Evaluation then uses Expected Calibration Error, i.e. checking that the predicted 90% ellipsoid really does contain the truth about 90% of the time.

   raw point cloud
        |
        v
  +----------------------------+
  |  sparse conv backbone      |   (LightLoc)
  +----------------------------+
        |                 |
        v                 v
  [coord head]        [covariance head]
     mu (3)              6 params
        |                 |
        |                 v
        |            L = lower-tri, diag > 0
        |            Sigma = L L^T + eps*I
        |                 |
        +--------+--------+
                 |
   TRAIN:        v
     NLL:  d^T Sigma^-1 d + log det Sigma
     plus  kNN smoothness: || Sigma_i - Sigma_j ||
           over spatial neighbours j of i
                 |
   INFER:        v
  +----------------------------------------+
  |  SC2-PCR*                              |
  |   seed score  ~  compat / spread(Sigma) |
  |   inlier test :  d^T Sigma^-1 d < tau   |
  +----------------------------------------+
                 |
                 v
        6-DoF pose  (+ usable covariance)

The metaphor: think of a surveying crew staking out a building on a foggy site.

Each voxel is one surveyor. In old LightLoc, every surveyor drives a stake into the ground and says “the corner is here”, full stop. The foreman then measures how far apart the stakes are and discards anyone more than 30 cm from consensus — the same 30 cm for the surveyor standing on solid ground three metres away and the one squinting through fog at eighty metres.

UQ-Loc gives each surveyor a chalk ellipse to draw around their stake: “the corner is here, and my confidence is this shape — I’m certain within 2 cm side-to-side but could be off half a metre in depth.” The covariance head is the chalk ellipse. The Cholesky parameterisation is the rule that the chalk must trace a real closed ellipse and not a figure-eight. The NLL loss is the foreman’s performance review: get caught with the true corner outside your ellipse and you’re penalised, but draw an ellipse the size of the car park and you’re penalised too. The kNN smoothness rule is the crew policy that surveyors standing shoulder to shoulder on the same wall must report similarly shaped ellipses — someone claiming radically different fog from their neighbour is probably wrong about the fog, not observing real fog. Uncertainty-weighted seed scoring is the foreman choosing the tight-ellipse surveyors as the reference stakes to build the layout from. And the Mahalanobis test is the foreman finally saying “I’ll no longer judge you in centimetres; I’ll judge you in your own claimed ellipses” — being 50 cm off in the direction you admitted you were unsure about is fine, being 5 cm off in the direction you swore you were certain about is a firing offence. ECE is the audit at the end of the season: of all the stakes where surveyors drew 90% ellipses, did the true corner really land inside 90% of the time?

Key Concepts

• Anisotropic covariance (a full 3x3 vs a single number): Suppose you ask where a friend is standing and they say “somewhere within 10 metres”. That’s isotropic — a sphere of ignorance, one number. Now suppose they say “I’m definitely on this street, within half a metre of the kerb, but I could be anywhere along a 30 metre stretch of it”. That’s anisotropic — a long thin cigar of ignorance, and it needs a matrix to describe: three axis lengths plus the orientation of those axes, six numbers total. This distinction is not academic for LiDAR. A voxel on a big flat wall genuinely *is cigar-shaped in its uncertainty, because sliding it along the wall leaves the geometry looking identical. Collapse that to one number and you either become overconfident along the wall or needlessly timid across it.

• Negative log-likelihood as a self-calibrating loss: A normal regression loss asks “how far off were you?” NLL asks “how surprised should you be by how far off you were?” Concretely, with a predicted spread of 1 cm, a 10 cm error is a catastrophe; with a predicted spread of 1 m, the same 10 cm error is unremarkable. Because the loss also charges you for the volume of your claimed spread (log det Sigma), you cannot escape by claiming total ignorance everywhere. The only stable strategy is to report your actual error distribution. The useful side effect — and also the confound, see below — is that hard examples get automatically down-weighted, since the network can pay the log det fee rather than fight an unwinnable regression.

• Expected Calibration Error for a regression problem: ECE is normally a classification idea: of all the times the model said “70% sure”, was it right about 70% of the time? Porting this to 3D coordinates works via ellipsoids. For each prediction, compute the Mahalanobis distance of the true point; under a correct Gaussian this quantity follows a known chi-squared distribution with 3 degrees of freedom, so you know exactly what fraction of points *should fall inside the 50%, 70%, 90% ellipsoids. Compare the predicted fraction to the observed fraction across confidence levels and average the gaps. Low ECE means “the model’s error bars are honest”, which is a completely different claim from “the model is accurate” — a model can be accurate and wildly overconfident, and that combination is exactly what breaks downstream filters.

Framework Shift

Before (mainstream LiDAR SCR):        After (UQ-Loc):

 point cloud                           point cloud
     |                                     |
  backbone                              backbone
     |                                     |     \
   mu (3)                                mu (3)   Sigma (3x3)
     |                                     |     /
     v                                     v
 +-----------------+                  +-----------------------+
 | solver          |                  | solver                |
 |  threshold in   |                  |  threshold in         |
 |  METRES, same   |                  |  SIGMAS, per-voxel    |
 |  for all voxels |                  |  and per-direction    |
 +-----------------+                  +-----------------------+
     |                                     |
   pose                                  pose  +  covariance
                                                    |
 (downstream gets                          (downstream can
  a number, no trust)                       fuse and refuse)

 error model: implicit,             error model: explicit,
 global, spherical                  local, ellipsoidal, learned

From a global metric threshold to a per-voxel learned metric, the core shift is that the distance function used to judge geometric agreement is itself predicted by the network rather than hand-set by a hyperparameter.

Expert Assessment

Problem choice: Real gap, and a slightly overdue one. Visual SCR started grappling with uncertainty and confidence-weighted pose solving years ago; LiDAR SCR is young enough that it inherited deterministic heads without much argument. Meanwhile the actual consumers of LiDAR localisation — pose graphs, EKFs, autonomy stacks with fallback logic — have always wanted covariances. So this sits in a genuine hole. It is not a *deep hole, though. This is the kind of paper the field was always going to produce; the question is whether it produces it well, not whether the idea is surprising. The anisotropy argument is the strongest part of the motivation, because LiDAR-on-planar-surfaces is a textbook case where a scalar confidence is provably the wrong object.

Method maturity: Every component is borrowed and none of them is new in isolation. Cholesky-parameterised heteroscedastic Gaussian heads trace to Kendall & Gal (2017) and are standard practice. Mahalanobis inlier gating is standard in probabilistic registration (GICP, NDT, and the whole distribution-to-distribution lineage). ECE is off the shelf. The contribution is integration and the observation that you should thread the covariance through the *solver, not just report it. That is a legitimate but modest contribution. The kNN smoothness regulariser is the one place with a bit of taste in it — it is the right prior for the domain, since uncertainty derives from local surface geometry — but it also smells like a stability crutch for NLL, which is notoriously prone to collapse and to the “give up on hard voxels” pathology. Simpler things I would want ruled out: a per-voxel isotropic scalar (does the full 3x3 actually beat one number?), a fixed analytic covariance derived from local surface normals and point density with no learning at all, and plain Huber loss with a tuned threshold. The analytic-normal baseline in particular could plausibly capture most of the anisotropy for free.

Experimental integrity: I can only judge what the abstract commits to, and it commits to remarkably little: “consistent improvement in 6-DoF localization accuracy” with no magnitude, no dataset named, no numbers. In my experience “consistent improvement” in an abstract usually means small but reproducible, and the choice not to lead with a headline figure is informative. The bigger methodological worry is a confound the paper must address to be credible: NLL is a *loss reweighting scheme. Switching from L1 to NLL down-weights hard voxels, and that alone often improves pose accuracy regardless of whether the covariances mean anything. So the essential ablation is NLL-trained model + old isotropic solver versus NLL-trained model + Mahalanobis solver. If the gain lives entirely in the first step, the paper’s actual finding is “learned loss attenuation helps LiDAR SCR”, which is a fine but different result. Two more things I would look for: whether ECE is reported per confidence level or only aggregated (aggregate ECE hides systematic overconfidence at the tails, which is exactly the regime downstream fusion cares about), and whether calibration holds under distribution shift across sessions, since aleatoric-only models typically fall apart there and this one explicitly models only aleatoric uncertainty. Ignoring epistemic uncertainty is defensible for a first paper but should be stated as a limitation, not omitted.

Writing quality: The abstract is efficient and honest in structure — it names the parent architecture, lists the four changes, and says what is measured. The corner cut is the vagueness of the results claim; an abstract that will not name a number is asking the reader to grant trust it hasn’t earned. The section that would most elevate the paper is the ablation and calibration analysis. If the author rewrote it to isolate loss-reweighting from covariance-aware solving, show reliability diagrams rather than a single ECE scalar, and add the analytic-normal-based covariance baseline, this goes from “reasonable engineering extension” to “the reference paper on uncertainty in LiDAR SCR”. Adding one qualitative figure — covariance ellipsoids drawn on a real scan, showing them flatten against walls and stretch along corridors — would do more persuasive work than another table.

Verdict: weak accept — the right object (full anisotropic covariance) plumbed into the right place (the solver’s inlier metric), assembled entirely from known parts, and standing or falling on ablations the abstract does not preview.

Takeaways

Things worth stealing regardless of whether you care about LiDAR:

• Push uncertainty into the decision rule, not just the report. The reusable move here is replacing a hand-tuned global threshold with a per-sample learned metric. Anywhere you currently write if error < tau, ask whether tau should be predicted per-sample and per-direction. This applies to matching, tracking, outlier rejection, active learning triggers, and anomaly detection.
• Cholesky-parameterise anything that must stay positive-definite. Predict L, output L L^T + eps**I. No projections, no clamping, no invalid states, differentiable throughout. Standard among people who do this daily, still under-used by everyone else.
• A smoothness prior over uncertainty, not just over predictions. Spatial or temporal regularisation is usually applied to the mean. Applying it to the *covariance is the transferable idea: if your uncertainty is caused by a physical property that varies smoothly (surface geometry, illumination, sensor coverage), then neighbouring predictions should agree about their uncertainty too. This is a cheap stabiliser for NLL training in general, which is otherwise notorious for collapse.
• Separate “accurate” from “honest” in your metrics. Adopt a calibration metric alongside your error metric, and use the chi-squared trick to make ECE work for multivariate regression. A model that is accurate and overconfident is more dangerous downstream than one that is slightly worse and honest, and you will never notice the difference if you only track RMSE.
• Watch the confound when you adopt NLL. If accuracy improves after switching to NLL, do not assume your uncertainty estimates are doing the work. Run the ablation that keeps the loss and reverts the inference-time use of the covariance. This confound has misled plenty of papers beyond this subfield.