Signed angle between two 3D vectors with same origin within the same plane

What I need is a signed angle of rotation between two vectors Va and Vb lying within the same 3D plane and having the same origin knowing that:

1. The plane contatining both vectors is an arbitrary and is not parallel to XY or any other of cardinal planes
2. Vn - is a plane normal
3. Both vectors along with the normal have the same origin O = { 0, 0, 0 }
4. Va - is a reference for measuring the left handed rotation at Vn

The angle should be measured in such a way so if the plane would be XY plane the Va would stand for X axis unit vector of it.

I guess I should perform a kind of coordinate space transformation by using the Va as the X-axis and the cross product of Vb and Vn as the Y-axis and then just using some 2d method like with atan2() or something. Any ideas? Formulas?

The solution I'm currently using seems to be missing here. Assuming that the plane normal is normalized (|Vn| == 1), the signed angle is simply:

For the right-handed rotation from Va to Vb:

atan2((Va x Vb) . Vn, Va . Vb)

For the left-handed rotation from Va to Vb:

atan2((Vb x Va) . Vn, Va . Vb)

which returns an angle in the range [-PI, +PI] (or whatever the available atan2 implementation returns).

. and x are the dot and cross product respectively.

No explicit branching and no division/vector length calculation is necessary.

Explanation for why this works: let alpha be the direct angle between the vectors (0° to 180°) and beta the angle we are looking for (0° to 360°) with beta == alpha or beta == 360° - alpha

Va . Vb == |Va| * |Vb| * cos(alpha)    (by definition) 
        == |Va| * |Vb| * cos(beta)     (cos(alpha) == cos(-alpha) == cos(360° - alpha)


Va x Vb == |Va| * |Vb| * sin(alpha) * n1  
    (by definition; n1 is a unit vector perpendicular to Va and Vb with 
     orientation matching the right-hand rule)

Therefore (again assuming Vn is normalized):
   n1 . Vn == 1 when beta < 180
   n1 . Vn == -1 when beta > 180

==>  (Va x Vb) . Vn == |Va| * |Vb| * sin(beta)

Finally

tan(beta) = sin(beta) / cos(beta) == ((Va x Vb) . Vn) / (Va . Vb)

Use cross product of the two vectors to get the normal of the plane formed by the two vectors. Then check the dotproduct between that and the original plane normal to see if they are facing the same direction.

angle = acos(dotProduct(Va.normalize(), Vb.normalize()));
cross = crossProduct(Va, Vb);
if (dotProduct(Vn, cross) < 0) { // Or > 0
  angle = -angle;
}

You can do this in two steps:

1. Determine the angle between the two vectors

theta = acos(dot product of Va, Vb). Assuming Va, Vb are normalized. This will give the minimum angle between the two vectors

2. Determine the sign of the angle

Find vector V3 = cross product of Va, Vb. (the order is important)

If (dot product of V3, Vn) is negative, theta is negative. Otherwise, theta is positive.

You can get the angle up to sign using the dot product. To get the sign of the angle, take the sign of Vn * (Va x Vb). In the special case of the XY plane, this reduces to just Va_x*Vb_y - Va_y*Vb_x.

Advanced Customer provided the following solution (originally an edit to the question):

SOLUTION:

sina = |Va x Vb| / ( |Va| * |Vb| )
cosa = (Va . Vb) / ( |Va| * |Vb| )

angle = atan2( sina, cosa )

sign = Vn . ( Va x Vb )
if(sign<0)
{
    angle=-angle
}

Cross one vector into the other and normalize to get the unit vector.

The sine of the angle between the two vectors equals the magnitude of the cross product divided by the magnitudes of the two vectors:

http://mathworld.wolfram.com/CrossProduct.html

Let theta be the angle between the vectors. Let C = Va cross product Vb. Then

> sin theta = length(C) / (length(Va) * > length(Vb))

To determine if theta is positive or negative, remember that C is perpendicular to Va and Vb pointing in the direction determined by the right-hand rule. So in particular, C is parallel to Vn. In your case, if C points in the same direction as Vn, then theta is negative, since you want left-handed rotation. Probably the easiest computational way to quickly check if Vn and C point in the same direction is to just take their dot product; if it is positive they point in the same direction.

All this follows from elementary properties of the cross product.

Suppose Vx is the x-axis, given the normal Vn, you can get the y-axis by cross product, you can project the vector Vb to Vx and Vy (by the dot product you can get the length of the projection of Vb onto Vx and Vy), given the (x, y) coordinate on the plane, you can use atan2(y, x) to get the angle in the range [-pi, +pi]

This is the Matlab code to compute the signed angle between two vectors u,v either in 2D or in 3D. The code is self explanatory. The sign convention is such that a positive +90° is output between ix and iy ([1,0,0],[0,1,0]) or iy and iz ([0,1,0],[0,0,1])

function thetaDEG = angDist2Vecs(u,v)

if length(u)==3
    %3D, can use cross to resolve sign
    uMod = sqrt(sum(u.^2));
    vMod = sqrt(sum(v.^2));
    uvPr = sum(u.*v);
    costheta = min(uvPr/uMod/vMod,1);

    thetaDEG = acos(costheta)*180/pi;

    %resolve sign
    cp=(cross(u,v));
    idxM=find(abs(cp)==max(abs(cp)));
    s=sign(cp(idxM(1)));
    if s < 0
        thetaDEG = -thetaDEG;
    end
elseif length(u)==2
    %2D use atan2
    thetaDEG = (atan2(v(2),v(1))-atan2(u(2),u(1)))*180/pi;
else
    error('u,v must be 2D or 3D vectors');
end