Official specs Nokia N9 say that the phone is able to play the following video formats, including HD files at 720p: WMV 9, H.264/AVC, MPEG-4, XVID, Matroska MKV, 3GPP formats (H.263), AVI, VC-1, Video Flash, ASF.
The list looks impressive, but the reality is not as colorful. Unfortunately, it is hard enough to find an HD 720p file encoded in a way that Nokia N9 could deal with it without sweat. More on this topic can be read in the "epic" thread in the forum Talk Maemo.org. The thread is rather lengthy, so I'll try to summarize what is most important.
Nokia N9 plays high-definition video only in the so-called H.264 Baseline Profile. Videos encoded in H.264 Main and High profiles will either not play at all, or playback will stop after a few seconds and the message "Can not play these files in the device" will appear.
This is bad news for all mobile cinema lovers - many movies currently available on the network are encoded in High Profile (which is equally good quality, and movies take up much less space).
Oddly enough, the older Nokia N8 much better coping with playing numerous films, including 720p HD at Main and High Profile. This follows from the fact that N8 is more versatile than PowerVR SGX530 used in N9. Find more info here.
How to convert HD videos to N9
To convert movies for N9 use a great program WinAVI All-in-one Converter. It is not free, but offers the most options of all I had tried. I encode to N9 friendly format, usually on the MKV or MP4 or H.264 Baseline Profile XVID. To do this:
• In WinAVI menu on the left to select HD video file.
• You must add the files to be converted and select the folder where you want the ready movies. The whole process is usually very time consuming so it is best to leave your computer on overnight.
• Before you start, click Advanced and adjust the details: Change to Baseline profile. You can leave a 720p resolution, but the truth is that anything over 854 x 480 will just be scaled, so reducing the video resolution is your phone screen size, we can save valuable space.
• In addition, you can select the "Blend subtitles on video," which will allow us to watch on your movies with subtitles. Subtitles will be "embedded" in the film during the conversion. The condition is that the conversion of subtitle file must have the same name as the movie file.
• Frame rate does not matter, because the Nokia N9 can easily handle 30 frames per second.
In WinAVI I do recommend creating a "template" of conversion, which will change the profile of H.264 video, and scales each for Nokia N9 to make things faster in the future.
There is a video I made that shows the steps: http://www.youtube.com/watch?v=6zkOGQGTenU