EEVblog #726 - Dual Xeon Video Editing Machine Build

[Fungus]

I just noticed that PowerDirector is listed on Intel's "Quick Sync Enabled" page but neither Sony nor Adobe are:

http://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-general.html

Maybe that's why PD renders so fast.


PS: And I guess it also destroys George's "professional grade" software theory. Surely "professional grade" software would use any special hardware resources that are available and our cheapo "consumer grade" software wouldn't...right?

[Armxnian]

3) Direct uncompressed output using the Sony YUV Video For Windows codec actually works (it had issues on my old machine)
1:46 for the 1min test video which is pretty good and the fastest yet. That was to the SSD. File size was 12.163GB for the 1 min, so that means a 1 hour video would need 730GB.
Such an output would need about 114MB/s write speed, so that makes a 7200 rpm drive a WD Black suitable. 1GB+ of SSD would of course be very expensive and of no improvement.
So this seems like the best solution for now.
I might try some of the other VFW codecs again, but I've been advised by top people close to the metal that the Sony CODEC is pretty darn good at this.

You could try the Lagarith Lossless Video Codec. It probably compresses a bit better and is quite fast.
http://lags.leetcode.net/codec.html

could try going to huffyuv codec (multithreaded version) in Sony and then huffyuv to h264 in handbrake

Huffyuv is compressed but lossless. Files are large but realistic for a intermediate step.

Im not sure how the multithreaded performance of huffyuv compares to other codecs.

These codecs are outdated, not even developed anymore. They also require ridiculous CPU power which would increase render and encode times compared to uncompressed. Compressed lossless also has to be decoded to be reencoded using x264, which will take time. Also lagarith has problems with playback so you can't watch the video until you encode it again. These codecs were meant for systems with good CPU power but low drive speed. Capturing uncompressed 4:4:4 at resolutions of 1080 or higher and at high frame rate was impossible unless you had 10 drives in raid. Not much of a problem now as 2 ssds can give you over 1GBps writes.

Non ecc and faster memory can improve performance, not dramatically, but it can.

 
Seems not:
http://www.techspot.com/article/845-ddr3-ram-vs-ecc-memory/

And someone pointed out the E5-2630v2 does not support faster than 1333MHz memory, but that seems to be incorrect:
http://ark.intel.com/products/75790/Intel-Xeon-Processor-E5-2630-v2-15M-Cache-2_60-GHz
It can support up to 1600Mhz it seems.

This article is meh. Why test system ram performance using a GPU benchmark instead of video rendering or encoding, where lots of data is written and accessed from ram. Also they don't mention the fact that non ecc sticks have been available in much higher bandwidth at lower cost for quite a while now. Everyone in this thread is comparing 1333 to 1600, well that isn't that big of a jump, and 1600mhz is often CHEAPER than 1333mhz. You can pick up some Kingston ddr4 16gb quad channel 3000mhz and 12 cas latency ram for $250. Compare that to 1333 instead of 1600. Ddr3 prices for higher bandwidths are even cheaper.

Also, the figure on the Intel site is just what the memory controller is rated for. You can use speeds as high as your motherboard supports if your controller is up to it. 16gb doesn't put much strain on the integrated memory controller, so you can get much higher bandwidth sticks. Ivy bridge i7s were rated for 1600mhz memory, I ran 2400 with no problem.

For your case, there is no point in upgrading. As mentioned before, upgrading one part is almost always going to be bottlenecked by another part. Upgrading a system based on parts you already have is a waste of money. I've learned the hard way, and just ended up selling the parts and building from scratch, essentially losing money.

I said think about it for a future investment. No one "needs" 4k, you provide high resolution images of teardowns.

Right now the 4k cameras are still in the 'early adopter' price range, ie. too expensive.

Those who say there is no noticeable difference in image quality in 1080 vs 4k are the same who said there is no difference in 720 vs 1080 and 480 vs 720 a few years ago. If there is no difference, an entire industry wouldn't make the jump as the technology becomes available.

The 'industry' will jump as a marketing move - to sell new TVs. They'll advertise and pay for TV presenters/celebrities to casually mention how amazing their new TV is, even though they probably got theirs for free, it was the top-of-the-line model, and they probably don't even know what to look for in a TV image.

Plus: As soon as manufacturing costs go down to the same price level as the old ones, all the stores will stop selling the old stuff. All you'll be able to buy is 4k (or whatever) whether it's noticeably better or not.

In short: The industry 'jumping' doesn't tell you anything about the quality of a product. The difference to the consumer might be very small.

If you watch movies from 10 feet away on a 100" TV then of course 4k is going to look much better than 1080p. On a 24" panel? Not so much...

TV manufacturers are a business, they make money by milking early adopters of 4k tvs were no 4k content exists. Monitors are another story, you create content, view images, videos, games, whatever, but you get the benefit as 4k content exists. There is no 4k on cable/sattelite, streaming services, or bluray, so buying a 4k tv is a waste of money. You can't compare that to monitors.

For the rest of us, it is a positive jump. More 4k screens means more 4k content. Wouldn't you want to watch your movies in 4k? I would.

1080 on a 24inch screen is fairly low PPI, you can get a noticeable increase in sharpness by upgrading. You don't sit far from your monitor, you sit close.

[Fungus]

Just an idea: Give us the RAW Sony Vegas project and leave us fight to win the contest.

the best raw clip is TV static noise. it will max out encoding capability.

Download this blog video and use that. Let us know how you get on.

I already posted my results here: https://www.eevblog.com/forum/blog/eevblog-726-dual-xeon-video-editing-machine-build/msg635434/#msg635434

[Fungus]

buying a 4k tv is a waste of money. You can't compare that to monitors.

Yep, that's why I didn't...

[Armxnian]

buying a 4k tv is a waste of money. You can't compare that to monitors.

Yep, that's why I didn't...

No, you didn't buy one because they're overpriced, not because you think there is no benefit . 1080 on a 4k screen is theoretically better than on a native 1080 screen.

Monitors on the other hand have come down in price. You can get a 4k IPS panel for $500. They were $3000 a year ago. You honestly might not notice the difference vs a 1080 monitor, so there is no need to get a 4k monitor. But I will enjoy 4k content on a 1080 or 4k screen, and will persuade Dave to make 4k content in the future 

[Fungus]

buying a 4k tv is a waste of money. You can't compare that to monitors.

Yep, that's why I didn't...

No, you didn't buy one because they're overpriced, not because you think there is no benefit

I meant I didn't make that comparison. Obviously there's a benefit when you're using a computer close up, just not for watching TV.

[IanJ]

Hi all,

I haven't gone right through this thread.......so don't flame me if it's been posted already!.......but have you seen this:-

http://www.sonycreativesoftware.com/Forums/ShowMessage.asp?Forum=12&MessageID=716066

This hidden menu is available in Vegas Movie Studio HD V11 & Movie Studio Platinum 13 (I have both).
Interesting!....albeit a pity I only have a single i7(4770).....but I wonder what Dave's machine would make of these tweaks!

Ian.

QUOTE:-
Vegas 9 can make use of all the processor threads your system provides, but it has a built-in limitation of 4. This can be changed to 8 however if you know the following trick:

Hold down the Shift key when you open Options/Preferences. This will cause the Preferences window to open with a new tab called Internal. This is where you can change the limit of 4 threads to 8. Do this as follows:

1. Enter the text "threads" (no quotes) in the text box at the bottom. This will display all the parameters containing the word "threads."

2. Find the line that says Maximum video render threads (or something like that - I no longer have VMS 9 to look at.)

3. In that line change both the Default and Value numbers to 8 (or whatever the number of processing threads your system has.)

4. Hit OK, then close Vegas and re-open it.

You are now running with all threads active and usable.

To clarify some terminology: Intel now makes CPU chips that are multi-threaded. That means a single chip can process more than one instruction stream at a time, thus enabling one physical chip to do the work of multiple single thread ships.

"Thread" means a stream of instructions, which is the work processed by the CPU or "core". Core is really an obsolete/incorrect term that came into being yeas ago when main memory was referred to as "core memory" to differentiate it from cache memory built into processor chips.

In the case of the i7 line of processors, each i7 has 4 CPUs inside it. And each of these is dual channel. This means an i7 based computer can process 8 instruction streams ("threads") simultaneously, assuming there is enough memory in the system to support that.

Since raw CPU power is typically the limiting factor on render speeds, having more threads running speeds things up considerably. When rendering with Vegas you want all the CPUs you can get (and all the memory too,)

[FHR]

So this is the limiting factor! I said it has to be software fault. People here were saying all kinds of crazy and hyper-sophisticated things, including highly advanced NUMA architecture. And it all comes down to a software, respectively it's config.

[EEVblog]

Here you go dave check this video out.i even did that while capturing my screen in HD res lol
i didn't show entire rendering process just time estimated time left by SOny movie studio and i completed the rendering it is almost the same

Without the screen cap running the time is:
8:13 CPU
1:20 GPU

don't tell me your video are different and this stuff, it is clear man that my CPU had a rough time dealing with this 2 minute clip while with GPU it was a breeze although it is an old beast. so Opencl is the way to go still wondering why it didn't work for you
http://youtu.be/1zbfnwaV9ds

edit: the whole clip is 2 minutes long realtime

Nope, doesn't work for me, sorry.
Yep, I did it exactly the same as you showed, just like I've done it time and time again before when it didn't work.

[EEVblog]

Hi all,
I haven't gone right through this thread.......so don't flame me if it's been posted already!.......but have you seen this:-
http://www.sonycreativesoftware.com/Forums/ShowMessage.asp?Forum=12&MessageID=716066
This hidden menu is available in Vegas Movie Studio HD V11 & Movie Studio Platinum 13 (I have both).
Hold down the Shift key when you open Options/Preferences. This will cause the Preferences window to open with a new tab called Internal. This is where you can change the limit of 4 threads to 8.

Well I'll be damned, didn't know about this.
But no, sorry, doesn't work. Increased to 12 threads and it's now slower. Over 3 minutes.

[IanJ]

Hi all,
I haven't gone right through this thread.......so don't flame me if it's been posted already!.......but have you seen this:-
http://www.sonycreativesoftware.com/Forums/ShowMessage.asp?Forum=12&MessageID=716066
This hidden menu is available in Vegas Movie Studio HD V11 & Movie Studio Platinum 13 (I have both).
Hold down the Shift key when you open Options/Preferences. This will cause the Preferences window to open with a new tab called Internal. This is where you can change the limit of 4 threads to 8.

Well I'll be damned, didn't know about this.
But no, sorry, doesn't work. Increased to 12 threads and it's now slower. Over 3 minutes.

Presume you changed BOTH max settings in there.

Ian.

[EEVblog]

Presume you changed BOTH max settings in there.

Yes. And BTW, I'm pretty sure it doesn't matter because the "64bit" version already has it's own max threads setting and that is already set to 16 on both by default.

[rs20]

But no, sorry, doesn't work. Increased to 12 threads and it's now slower. Over 3 minutes.

Well, if increasing it makes it slower, decreasing it should make it faster, right?    But more seriously, that is interesting. The only (no particularly productive) theory I can offer is that the CPU's cache can't keep all those extra threads in memory, so stuff is continually getting bounced off to the RAM. Just like using too much RAM causes swapping.

[EEVblog]

Well, if increasing it makes it slower, decreasing it should make it faster, right?    But more seriously, that is interesting. The only (no particularly productive) theory I can offer is that the CPU's cache can't keep all those extra threads in memory, so stuff is continually getting bounced off to the RAM. Just like using too much RAM causes swapping.

No, it has nothing to do with the setting, I'm using the 64bit version. I just verified that it's still slower after changing back, using the the Sony AVC codec, so somethign else is at play here.
Sony YUV high bitrate output time remains consistent.

[Smokey]

I'm joining this party 10 pages late.  I didn't read everything, and I'm sure someone already said this but....

You can either have the warm fuzzy feeling of something you are used to and comfortable with, or you can build a system for speed to get some job done.  It's a render box for your job, not a daily driver facebook browser.  Why limit the machine in any way?  If it runs faster on Linux, install Linux.  If your Sony Vegas software isn't optimized and can't take advantage of the hardware, get new software.  It's like driving your car with the parking brake on and complaining about not going fast enough.  You proved that handbrake is significantly faster (even without hyperthreading) so the machine has more potential than you are able to take advantage of with the setup.  To put it in EE terms, if you needed to professionally debug a USB3 bus are you going to buy an optioned up modern scope but refuse to enable the USB3 protocol analyzer?

[EEVblog]

If your Sony Vegas software isn't optimized and can't take advantage of the hardware, get new software.

I've said this a dozen times before, but it seems I have to keep repeating myself.
I have tried many different software packages, even the "professional ones", practically every package on the market in fact.  They all have little issues for my particular workflow that keep making me come back to Sony (which has issues too, but less, and yes, it's familiar).
Don't underestimate the importance of familiarity. In the end, it's only rendering time, I don't have to sit there and watch it. But switching to another package that did this more convolauted than I am used to for my workflow, then that would stress me out day in and day out.
It ain't as easy as you think to just change software.

[Rotareneg]

When rendering from Movie Studio 13 Platinum I use Debugmode FrameServer and AviSynth to serve frames to ffmpeg (this works with Handbrake too.) That way I can avoid having to render to an intermediate file.

[Grecord]

So a few things you mentioned in the vid reminded me of some other vids I've seen recently, Linking them here.

https://www.youtube.com/watch?v=kc6Rbel6n1g,  https://www.youtube.com/watch?v=khDsbxa5_G0,       
Charts at4:40 and mention of vid rendering at 5:20, Well I hope there's a nugget of insight in there somewhere.

[Fungus]

Hi all,
I haven't gone right through this thread.......so don't flame me if it's been posted already!.......but have you seen this:-
http://www.sonycreativesoftware.com/Forums/ShowMessage.asp?Forum=12&MessageID=716066
This hidden menu is available in Vegas Movie Studio HD V11 & Movie Studio Platinum 13 (I have both).
Hold down the Shift key when you open Options/Preferences. This will cause the Preferences window to open with a new tab called Internal. This is where you can change the limit of 4 threads to 8.

Well I'll be damned, didn't know about this.
But no, sorry, doesn't work. Increased to 12 threads and it's now slower. Over 3 minutes.

Menus like this are there to let people figure out the optimum settings. It makes sense that the one the final end-user sees is the optimum for the codecs. Rendering speed is a big selling point, and also one that's easy to measure/compare on different machines. They wouldn't knowingly choose a bad setting.

I think the problem lies in the codecs themselves, there's only so much parallelism you can do - compressed video depends heavily on the results of the previous frame. You can't process every frame in the video in a separate thread, you have to wait for the previous frame to complete.

You can split up the workload of each frame but there still needs to be a master thread making the decisions about which data goes through to the next frame and that job can't really be done in parallel. Once it becomes the bottleneck then using more threads won't help.

[rs20]

I think the problem lies in the codecs themselves, there's only so much parallelism you can do - compressed video depends heavily on the results of the previous frame. You can't process every frame in the video in a separate thread, you have to wait for the previous frame to complete.

You can split up the workload of each frame but there still needs to be a master thread making the decisions about which data goes through to the next frame and that job can't really be done in parallel. Once it becomes the bottleneck then using more threads won't help.

What I'm saying here isn't connected to the reality of how codecs are implemented, but: You're describing the hard way of parallelizing a video codec. Every nth frame in a compressed video is a complete, independent frame, so you can break up a video into a bunch of separate, independently compressible blocks, boundaried by these complete frames. The easy way to parallelize is to hand out these blocks to cores as they become idle. Downside is that there are a lot more frames waiting for consideration in memory at a time, but it wouldn't be GB. A hybrid of these two techniques would make even more sense.

[Fungus]

What I'm saying here isn't connected to the reality of how codecs are implemented, but: You're describing the hard way of parallelizing a video codec. Every nth frame in a compressed video is a complete, independent frame, so you can break up a video into a bunch of separate, independently compressible blocks, boundaried by these complete frames.

If it's embarrassingly parallel then why don't they do it?

[rs20]

If it's embarrassingly parallel then why don't they do it?

It is embarrassingly parallel, if you're willing to set aside a gig of RAM just for keeping all these blocks of frames stored. The hard way of parallelizing requires less RAM, so if they think they can implement it correctly the hard way, that'd be the optimal way to do it -- leaving more RAM for the renderer. If they choose that approach, but forget to design with 12-core machines in mind (as you correctly suggested several messages ago)... then we end up having this discussion.

[coppice]

What I'm saying here isn't connected to the reality of how codecs are implemented, but: You're describing the hard way of parallelizing a video codec. Every nth frame in a compressed video is a complete, independent frame, so you can break up a video into a bunch of separate, independently compressible blocks, boundaried by these complete frames.

If it's embarrassingly parallel then why don't they do it?

The term embarrassingly parallel is usually applied to something which can be split into a huge number of threads, so it would keep something like a GPU busy. He didn't say that. He said video could be split into enough threads to keep a 24 core machine busy, which is feasible with the amount of RAM buffering you can do in a modern machine.

[sync]

Well, if increasing it makes it slower, decreasing it should make it faster, right?    But more seriously, that is interesting.

Almost all applications will go slower with too many cores. Amdahl's law give you the theoretical maximum speed up. But in reality there are things which will slow down with more cores.
Also is often preferable not too use all available cores with your application. Leave one or two free for the operating system.

[Fungus]

If it's embarrassingly parallel then why don't they do it?

It is embarrassingly parallel, if you're willing to set aside a gig of RAM just for keeping all these blocks of frames stored.

A whole gig of RAM? Yeah, that's such a massive problem these days. If that's the only reason it's not doing it, then...

The term embarrassingly parallel is usually applied to something which can be split into a huge number of threads, so it would keep something like a GPU busy. He didn't say that. He said video could be split into enough threads to keep a 24 core machine busy, which is feasible with the amount of RAM buffering you can do in a modern machine.

From the Wikipedia page I linked to: "In parallel computing, an embarrassingly parallel workload, or embarrassingly parallel problem, is one for which little or no effort is required to separate the problem into a number of parallel tasks."

He claimed that all they need to do is to split the video up in to chunks of 'N' frames then compress each chunk separately in its own thread. That seems like "very little effort" to me so why aren't they doing it?

RAM? Finger-in-the-air calculations would suggest that even 24 threads wouldn't need more than a couple of gigs for the codec.

eg. A 1920x1080 buffer with RG+B stored in single precision floats is under 24Mb, a couple of those per thread, multiply by 24, it's about 1Gb. They'll also need 'N' frames of source video each and some workspace. Source is 6Mb per frame, the compression routines don't need much because they work on individual 8x8 blocks....call it another 1Gb.

Even if it was twice that it would be no big deal - a machine with 24 CPUs can be expected to have that much.

Or ... thinking radically .... the codec could look at how much RAM is in the machine and adjust the number of threads to use 50% of that (or whatever).

Bottom line: It's easy to code so I really can't see an excuse for not maxing out all the CPUs.

PS: Render speed is probably more important to a codec developer than the end users...the developer has to sit there for hours watching video encode while he twiddles his thumbs. Every single optimization counts!