in development

roll that camera, zombie: rotation and conversion from YV12 to YUV 420 Planar

In our last episode, I was converting the NV21 image format from my Android device’s camera into the more display-friendly RGB. However, now that I’m trying to push the images over a network, I’m having to run them through a video codec, and the list of color formats supported by the codec is…hilariously small. Literally, there are two available formats and neither one will work directly with the camera. The closet match is between the YV12 image format (supported by all Android cameras) and the YUV 420 Planar image format (supported by all Android codecs).

The catch? The two formats have their U and V chroma planes swapped, making everyone look as if they just stepped out of a zombie flick.

Image rotation is another problem. When I rotate the device, the camera rotates and the display rotates, but the image data has no idea about any of this. Turn the device upside-down, and the feed from the camera shows an upside-down room. This is not what I want.

I’m in a generous mood. Let’s solve both issues.

I can get the current orientation state of the device from the window manager, then use the base orientation of the camera to derive a rotation for the image. I’ve adapted some code from the Android docs to do just that. (I can’t use Camera.setDisplayOrientation() to solve the rotation problem because it doesn’t affect the camera output, just the preview display.)

private int getRotation() {
	Display display = getWindowManager().getDefaultDisplay();
	int rotation = display.getRotation();
	int degrees = 0;
	switch (rotation) {
	case Surface.ROTATION_0: degrees = 0; break;
	case Surface.ROTATION_90: degrees = 90; break;
	case Surface.ROTATION_180: degrees = 180; break;
	case Surface.ROTATION_270: degrees = 270; break;
	}
	int result = 0;
	if (cameraInfo.facing == CameraInfo.CAMERA_FACING_FRONT) {
		result = (cameraInfo.orientation + degrees) % 360;
		result = (360 - result) % 360;	// compensate the mirror
	} else {
		result = (cameraInfo.orientation - degrees + 360) % 360;
	}
	return result;
}

Once I’ve got the rotation state I apply to it the image. We assume that the input and output buffers are the same size.

public static void rotateY12toYUV420(byte[] input, byte[] output, int width, int height, int rotation) {
	boolean swap = (rotation == 90 || rotation == 270);
	boolean flip = (rotation == 90 || rotation == 180);
	for (int x = 0; x < width; x++) {
		for (int y = 0; y < height; y++) {

I’m breaking the rotation into two components: swap and flip. Swap means swapping the x & y coordinates, which provides a 90-degree rotation, and flip means mirroring the image for a 180-degree rotation. We do the following for each pixel in the image.

			int xo = x, yo = y;
			int w = width, h = height;
			int xi = xo, yi = yo;
			if (swap) {
				xi = w * yo / h;
				yi = h * xo / w;
			}
			if (flip) {
				xi = w - xi - 1;
				yi = h - yi - 1;
			}
			output[w * yo + xo] = input[w * yi + xi];

I make copies of the x/y and width/height because I’m going to manipulate those values. If we’re swapping x and y, I have to stretch them as I can’t guarantee that the width and height of the image are the same. Flipping the image simply reverses it in both x and y. Note that I’m iterating over the output coordinates to derive the input coordinates. I tried it the other way originally and found that it left gaps in the final image where two sets of input coordinates mapped to the same output coordinates. Doing it this way ensures that all output pixels are covered. This takes care of transforming the Y (or intensity) plane of the YUV format, but there’s two more planes to deal with.

			int fs = w * h;
			int qs = (fs >> 2);
			xi = (xi >> 1);
			yi = (yi >> 1);
			xo = (xo >> 1);
			yo = (yo >> 1);
			w = (w >> 1);
			int ui = fs + w * yi + xi;
			int uo = fs + w * yo + xo;
			int vi = qs + ui;
			int vo = qs + uo;
			output[uo] = input[vi]; 
			output[vo] = input[ui]; 
		}
	}
}

In the second half, I transform and swap the UV planes. Both YV12 and YUV420  feature sub-sampled color planes. Each plane is half the width/height of the Y plane, and therefore a quarter of the size (if size = width * height, then width/2 * height/2 = size / 4). To locate a pixel in these planes, we divide all our x and y coordinates by two. We’re using width as our stride, so that also gets divided by two. Planar offsets into the image can be found by using the size of the Y plane (width * height) and the size of a chroma plane (width * height / 4). Don’t forget to swap the UV planes between input and output!

Update: I’ve had requests for a version of this routine that works for NV21 format. Here it is:

public static void rotateNV21toYUV420(byte[] input, byte[] output, int width, int height, int rotation) {
	boolean swap = (rotation == 90 || rotation == 270);
	boolean flip = (rotation == 90 || rotation == 180);
	for (int x = 0; x > 2);
			xi = (xi >> 1);
			yi = (yi >> 1);
			xo = (xo >> 1);
			yo = (yo >> 1);
			w = (w >> 1);
			h = (h >> 1);
			// adjust for interleave here
			int ui = fs + (w * yi + xi) * 2;
			int uo = fs + w * yo + xo;
			// and here
			int vi = ui + 1;
			int vo = qs + uo;
			output[uo] = input[vi]; 
			output[vo] = input[ui]; 
		}
	}
}	

    • Hi Kirtan,

      NV21 should be very close to YV12, except the U and V color planes are swapped. You should be able to get it working just by changing these lines:

      output[uo] = input[vi];
      output[vo] = input[ui];

      to read:

      output[uo] = input[ui];
      output[vo] = input[vi];

      Give that a try.

      Regards,
      Chris

      • Hi chris.

        In the NV21,

        output[uo] = input[vi];
        output[vo] = input[ui];

        to read:

        output[uo] = input[ui];
        output[vo] = input[vi]

        not work well.How can I do?

        • Hi TwoForest,

          Sorry, my mistake! The NV21 format has the UV planes interleaved, so my original statement was wrong. I’ve just written and tested the following code on an NV21 stream, and it worked fine.

          public static void rotateNV21toYUV420(byte[] input, byte[] output, int width, int height, int rotation) {
          	boolean swap = (rotation == 90 || rotation == 270);
          	boolean flip = (rotation == 90 || rotation == 180);
          	for (int x = 0; x > 2);
          			xi = (xi >> 1);
          			yi = (yi >> 1);
          			xo = (xo >> 1);
          			yo = (yo >> 1);
          			w = (w >> 1);
          			h = (h >> 1);
          			// adjust for interleave here
          			int ui = fs + (w * yi + xi) * 2;
          			int uo = fs + w * yo + xo;
          			// and here
          			int vi = ui + 1;
          			int vo = qs + uo;
          			output[uo] = input[vi]; 
          			output[vo] = input[ui]; 
          		}
          	}
          }	
          

          Hope that works for you!

          Regards,
          Chris

          • Hi Chris,

            I used Your function to rotate bytes in ‘onPreviewFrame’. And the thing is that the image gets rotated for 90 degrees but is also mirrored and flipped. Do You have any suggestions on how to just get image rotated.

            Regards,
            Domagoj

  1. Hi Domagoj,

    You could try playing with the swap/flip flags. Invert one (flip = !flip) then the other (swap = !swap) and see what you get.

    Chris

  2. Hi, is YUV420 in your post I420? I use output of rotateNV21toYUV420 as I420 but it is incorrect.

    • Hi Hieu,

      The output format is IYUV, which is supposed to be compatible with I420. However, see http://www.fourcc.org/yuv.php#IYUV for a discussion of this topic…it isn’t clear if they really are the same. Wish I could tell you more.

      Chris

  3. Hi Chris,

    Thanks for the wonderful blog, very informative and helpful.
    The rotateNV21toYUV420 you provided works great but the dimensions of the byte[] are kept intact, I mean byte[] obtained from onPreviewFrame() in portrait mode is 176*144 and after applying the method you provided the dimensions are still 176*144, could you please point to me what changes would be needed to make it 144*176 dimensions.

    • Thanks Zenith! I haven’t worked with this code in ages, but I think you’d have to do something like

      int w = width, h = height;
      int wo = height, h0 = width; // add this
      

      Then, for every transformation to output, use the (wo, ho) in place of (w, h). Example:

      output[wo * yo + xo] = input[wo * yi + xi];
      

      Sorry, I know it’s a little vague. Play around and see what you get. Good luck!

Comments are closed.