The original post was posted on my devblog in august  but I figured people might be curious so I'll post it here. 

original post: https://silveredgegames.com/2018/08/01/captains-log-august-1-cooking-up-some-lip-syncing-and-facial-animations/

Here is how to make facial animation and lip syncing in your game using Melanie level up 1 script.

1. Get a program call Papagayo. I think the pros (Bioware) uses FaceFX and Huniepop has a one from the unity asset store ( might be wrong but there is one on available in the store). Papagayo is free.

2. Drop the audio and text into the editor. Make sure you set the frame rate to 60. (assuming your game update loop is every 60 seconds) 

side note: trim the fat (sides, long pauses) from the audio file. it just makes your game bigger for no reason.

3. Make sure the last word is on the very end of the timeline if not you going to need to stretch to the green bar to fit which will reset all  words on the timeline; make sure you do this first. Now go through each word and stretch, shrink and move them to the correct location. This will make the voice actor sound like they have speech impediments as they say the word “the” 10 times before it sounds right.

4. Once you have it sounding right and looking right, export it.

5. Cleanup the file by removing the “mohoswitch1” make sure the first line index is 0 and that double rest are removed so that everything should be in numerical order.

Side note: The MBP, E, L, AI, Etc, E, you see beside the number are the lip shapes that need to be made on that tick.

before	after

6.  Time to add animation. I create a tool to make this easier for me (take the day to create a tool… editing the exported file by hand sucks). Under the hooded what it does is it create a list in game that keeps track of the mouth index and the mouth shapes (export file). Then on the update, it moves a counter and checks to see if it match the next mouth index. If it does it changes the mouth. This principle can be used for eyes and eyebrows as well as special effect. To make life easier I create a timeline with each mouth position place on it as a red square and a way of exporting the file again with the correct face animation added.

In the end, you should have something that looks like this.

Now that you have the steps go out there and make a better game with lip syncing and animation!