Use ffdshow tryouts
During install make sure to check the boxes 'VFW Interface' and 'VirtualDub' under 'Application plugins'.
If you already have other codecs, and you don’t need ffdshow to decode anything except H264 in VirtualDub, uncheck everything else except “H.264/AVC” at the two steps where it lets you choose what codecs ffdshow should decode.
When you’re done, go to your start menu (all programs on XP), ffdshow, VFW configuration. Select the decoder tab, and click on Codecs on the upper left. Click on “H.264/AVC” on the right, and in the middle column make sure it’s NOT disabled (aka libavcodec is selected). Then click apply and your H264 video should be openable in VirtualDub.
I know there’s other H264 decoders out there, but I’m not really up-to-date on which ones are VfW.
I’m the same as Jac100. I followed all the steps, but when it came to clicking “Apply”, the button was greyed out because there was nothing TO apply. The H.264/AVC codec was already active, but when I ran Vdub again, it showed the same error as before and couldn’t open my file.
What am I missing?