I think I've found the fix here. The problem for the SolveTOI function is that the first fixture's body position X and Y values are NaN. I can imagine how this prevent SolveTOI from converging
Now the question became: what is setting that body's position to NaN? So I looked into all the instances where b2Body:m_xf.position was changing, and found that the position is changed within the step() as well. Here's the callstack:
Turns out m_sweep.c values were set to NaN.
I started to look everywhere for b2Vec2 classes being set to NaN, and I realized I was multiplying the position of a mouseJoint attached to that body by a NaN value (1 divided by 0). So! My understanding is that by setting the b2MouseJoint:SetTarget to a NaN vector, I caused this hang.
I'm not sure where, but it sounds like we would want to add an assert somewhere
to make sure no one hits that hang again. The SolveTOI part is the part that hangs, but the cause of the problem was really earlier. It was a stupid bug and really my fault, but it would help if Box2D could detect that problem before hanging.
Thanks for your help on this, I can now sleep properly