MeiShe Face Sticker Design Plugin
AR Scene Editor Instruction
Version 1.4.0
Beijing Meishe Network Technology Co., Ltd.
Contents
1、Overview.. 4
2、Environment. 4
3、Description. 4
4、Model Creation Specification. 5
4.1、 Model Creation Specification. 5
4.2、Sticker Creation Specification. 7
4.3、Export FBX. 8
5、Installation and Project Creation. 9
5.1、Installation of plugin. 9
5.2、AR Scene Generation. 10
6、Unity 3D Sticker Creation Specification. 12
6.1、Unity Model Creation Specification. 12
6.2、Unity Material Creation Specification. 14
6.3、Unity Lighting Creation Specification. 17
6.4、Unity Animation Creation Specification. 18
7、Unity Physical System Creation Specification. 21
7.1、Ensure the Hierarchy. 21
7.2、Load Bone Physics. 22
7.3、Preview.. 23
7.4、Plugin Tune. 24
8、Unity 2D Sticker Creation Specification. 25
8.1、Static 2D Sticker Generation. 25
8.2、2D Image Series Sticker Generation. 26
9、Event Trigger. 28
9.1、Trigger Condition. 29
9.2、Trigger object switch. 30
9.3、Action trigger example. 31
9.4、Time trigger example. 33
10、Fake Face function. 34
10.1、Fake Faceparameter. 34
10.2、Example. 35
11、AR Scene Generation. 36
11.1、AR Scene Package Generation. 36
11.2、Test in Mobile Device. 38
Version
Description
Authors
1.1.0-1
Doc Creation
Liu Chong, Fan Chenchen
1.3.0-1
Doc Upgrading, Features envent and fakeface are added.
Zang Junqing, Zhang Siming
1.4.0-1
Doc Upgradin, and the problem that the packaged .Arscene file cannot be uploaded has been modified
Zang Junqing
1.4.0-1
The document is updated, adding a combination with 2D foreground and 2D trigger.
Fan Chenchen
1、Overview
MeiShe SDK provides face sticker rendering engine, which helps users to add various face stikcers during capture. For users to design more personalized stickers, we have developed a helper plugin called AR Scene Editor. This document is to guide users to the correct use of the plugin and how to complete the design work.
2、Environment
Hardware：Windows PC / MAC
Software：Unity 5.6.3 or above，the following instruction is based on Unity 5.6.3
Notes
MeiShe SDK only provides the design plugin. For Unity, we suggest users purchase the legitimate software for use. Meishe Network Technology Co., Ltd. will not be responsible for any problems or losses arising from the use of pirated Unity software.
3、Description
AR Scene Editor can only produce the face sticker packages applicable to the SDK. This plugin provides a whole set of AR Scene file making scheme for the SDK, realizing the visualization of model, material, lighting, animation and physical system effects. The tool is currently in the process of being perfected, and the types of stickers that can be made include
A. 2D face sticker
B. 3D face sticker
4、Model Creation Specification
Users can use MAYA,3Ds MAX, etc. as the 3D model design tool. We provide the standard face model (meishe_reference_head.fbx), All designs should be based on the standard face model, so that the designed stickers could accurately match the real human face and that the occlusion relation is right.
Notes
All specification referenced in this doc should be combination of letter, number and underline. Chinese characters are not allowed.
All path referenced in this doc should be combination of letter, number and underline. Chinese characters are not allowed.
A.Users use the standard face model (meishe_reference_head.fbx) to create face sticker.
Notes:
1
Number of facets should be within 2W.
2
Model history should be removed and transform should be zero.
3
In Meishe SDK, the opposite side of normal direction could not be displayed, models should be sealed. If necessary, AR Scene supports double side display, but this function has not been integrated into the AR Scene plugin in version 1.1.0. And double side display could lower the efficiency of the SDK.
B.The models in the outline will be rendered from top to bottom, which needs to be structured according to the spatial context. Otherwise it will not display correctly when using transparent sticker, etc.
C. Use the common method to bind the model，functions like IK, restriction can`t be use on bone.
D. All models and bones can be put into one main group, if the main group does not exist, unity will automatically generate one which has the same name as the FBX.
Notes
4.2、Sticker Creation Specification
A. stickers in JPG, PNG, JPG/PNG series（caf）are supported，transparent stickers are created in PNG or caf.
B. MeiShe SDK supports stickers in any aspect ratio, however ratio in 2ⁿ is suggested. And it should not exceed 2048x2048.
Notes
1
After sticker is applied to the model in 3D software, attribute Diffuse Color will not work at all, it won`t be read by the MeiShe SDK.
2
caf is the SDK`s internal format for image series, which can`t be read by other 3D software. After converting image series to caf, a companion image is generated automatically, which is used as the delegate of the caf in unity. When generating the sticker package, the delegate is then replaced by the real caf. For more information of caf creation, please refer to 8.2 2D Image Series Generation.
4.3、Export FBX
Notes
For the imported fbx file, pay attention to clear unnecessary history. The animation must be based on the Polygon model itself. The model should not be bound to an empty object for group animation. The SDK will not recognize similar locator nodes in Maya.
5、Installation and Project Creation
5.1、Installation of plugin
A. unity project creation.
Notes: Unity of version 5.6.3 and above is required and 5.6.3 is suggested. This doc is based on 5.6.3.
Notes
File unitypackage need not be copied to unity project, double click on the AR Scene plugin is just OK.
5.2、AR Scene Generation
Notes：Scenes created in other ways can`t generate AR Scene package.
Notes
AR Scene folder`s name is the name of the AR Scene package generated afterward. The 1.4.0 version of the Basic menu bar adds the Translation property, which can write the Chinese name of the prop.
Notes:
1
The camera is only used to observe the front 2D view. When the front 2D view needs to be observed, switch to the camera view for observation
2
The Layer Group is created to observe face movements and physical system effects in real time. Do not copy, change outline levels, or delete it. It is only used when the entire head section is required to be manipulated in a whole
Notes
All types of stickers should undergo this operation. Although 2D stickers could be imported from other folders expect for the unity project folder, this is not suggested.
6、Unity 3D Sticker Creation Specification
6.1、Unity Model Creation Specification
A. Drag the FBX to the Hierarchy
Notes:
1
Don`t change the order of the FBX`s sub layers. This should be completed in the 3D software. Though Unity will reorder sub layers when import, the rendering order set in the 3D software is not affected.
2
If the 3D model has certain occlusion relationship with the real human face in motion. A plain head model with opacity 0 should be created to test the occlusion. And the occluding layer should be placed above the occluded layer.
6.2、Unity Material Creation Specification
A. AR Scene plugin supports “no_shading” and “standard_lighting”. Users can switch the whole FBX`s material in “Shadings Mode” in FBX group. 
Notes
Users can only switch the whole FBX`s material in “Shadings Mode” in FBX group. Material adjustment on sub layer of model cann`t be recognized by AR Scene.
B. In version 1.3.0 and after, Settings of Blending Mode an Animation are added, Animation setting is discussed in “6.4 Unity Animation Creation Specification”.
Shading Mode
No Shading
Standard Shading
Blending Mode
Normal
Multiply
Subtract
Screen
Add
Exclusion
Animation
C. Attributes of the two types of material are listed in the figure below.
Notes:
1
Tilling and Offset is invalid
2
Unity can`t recognize Reflection Map. Users need to select it manually.
3
In version 1.1.0, Opacity of “no shading” is invalid, if opacity is needed, the material should be set to “standard lighting”.
4
For image series sticker creation, please refer to “8.2 Image Series Sticker Generation”.
6.3、Unity Lighting Creation Specification
A. AR Scene has a light by default，users can create other lights manually.
B. AR Scene only recognize light attributes “Transform”, “Type”, “Color”, “Intensity”.
Notes
For now, MeiShe SDK only supports two lighting mode, “Directional” and “Point”.
6.4、Unity Animation Creation Specification
Select the original FBX in AR Scene folder, Set attribute “Animation Type” to “Legacy” under “Rig” in the “Inspector”. Set “Warp Mode” under “Animations” to “Loop”. Then click on “Apply”. In this way, Unity will play the animation effect previously created in the 3D software in loop.
Notes:
1
Animation is created in the 3D software. To create animation in Unity, please refer to “7.3 Preview”.
2
Do not alter the previously created FBX in Unity. Animations created in Unity can only be previewed, and can be adopted by MeiShe SDK.
3
It happens that the preview in “Inspector” is not very accurate. The effect played in “game” is considered to be always accurate.
In version 1.3.0 and after, After selecting model in “Hierarchy”, users can set the “Animation” in the “Inspector” as following image.
If the imported FBX contains animation, the drop-down list of “Animation” will display the attached animation clips. It’s pretty straightforward for models with multiple animation clips.
In the image above, animation clip “a1” is chosen to be the active clip for the model. In “Show More Animation Fields”, users can set the loop modes and loop regions for all clips in the model.
Wrap Mode
Repeat: repeat
Mirror: mirror repeat
Constant: keep repeating the last frame
Repeat Pattern
When wrap mode is “Repeat” or “Mirror”, this param takes effects, the default value is null, meaning lopping for the whole segment
When wrap mode is “Repeat” or “Mirror”, users can set the repeat region and repeat times.
For instance, the lifetime of a1 is 3000 milliseconds, the input is 1000;2000;2， among which “1000” is the start time to loop, “2000” is the end time of loop, “2” is the loop times. The model will render the animation in 1-1000 milliseconds of a1, then render the animation in 1000-2000 twice, and finally render animation in 2000-3000. If the repeat times is 0, it means after rendering the animation in 0-1000, the engine will render the animation in 1000-2000 until it stops.
7、Unity Physical System Creation Specification
7.1、Ensure the Hierarchy
Make sure all FBX layers are dragged to “Layer Group”
Notes:
1
Do not alter “Layer Group” and “meishe_reference_head”, Do not try to unlock the locked attributes.
2
If there is a need to scale or translate all the FBX or standard face model as a whole, users can create a new “GameObject” sub layer, and then put all relevant objects to it. Then scale or translation on “GameObject” will also affect objects belonging to it.
7.2、Load Bone Physics
Select the root bone, which should be the origin of all the motion units in the physics system. Click on “Add Bone Physics” to add “Bone Physics” script to bones.
Notes:
1
Load separate “Bone Physics” script to all units in the physics system.
2
The root motion bone is decided by the “Root”, default of which is the bone equipped with the script. “Root” can be changed to other bones in the same FBX, however the translation of it is calculated according to object equipped with the “Bone Physics” script. Therefor we suggest unifying the “Root” and the bone equipped with the “Bone Physics” script.
7.3、Preview
There is an embedded head animation in “Resources” folder. In order to preview, users need to drag to “Controller” named “Layer Group” to “Layer Group” and click “Play”.Select “Face Animation” to alter the animation in “Animation”,
Notes:
1
When playing in “Animation”, “Bone Physics” script does take effect.
2
Attributes changed during playback will not be saved after stopping the playback.
3
If there already exists an animation in the FBX, both the two animations will be played. Animation created in Unity can only be previewed, and AR Scene does not recognize it.
7.4、Plugin Tune
Notes
In current version of AR Scene and SDK, same gravity value does perform the same. It is suggested that gravity is added after all other effects to avoid mis-observation.
8、Unity 2D Sticker Creation Specification
8.1、Static 2D Sticker Generation
A. Copy image to AR Scene folder.
B. Click on “Create Sticker With Image”, select the image in AR Scene folder to generate an image in the scene. 
C. Tune the “Transform” of the image.
8.2、2D Image Series Sticker Generation
A. Select a sequence of frames, set resolution, frame rate, “Pose” to generate the caf file.
Output Image Format：image format, PNG, JPG is supported.
Output Image Width：image width
Output Image Height：image height
Frame Rate：frame rate，usually set to 15/1, namely 15 frames per second.
Default Loop Mode：animation loop mode
Poster Frame Index：the index of the PNG image which is chosen as the delegate of the series.
B. Copy the caf file to AR Scene folder to automatically generate the company PNG file of the caf, of which “Poster Frame Index” is already set. Click on “Create Sticker With Image” to preview the company PNG file. Tune the “Transform” as needed.
Notes
Users can also preview the image by straightly selecting the caf file. However, “Poster Frame Index” must be set, otherwise the first frame will be regarded as the poster. And if the first frame is transparent, no image will be displayed.
9、Event Trigger【1.3.0 new】
In version 1.3.0 and after, event trigger mechanism is added. Users an set trigger conditions to control model switching, short/long audio playback. 2d sticker can’t be controlled yet.
In window “Event”, users can click “Add Event” to add triggers, detailed tunable params are listed below the button. In order to set the trigger, users need to set the trigger condition (in the red block) and the trigger object (int eh blue block). 
9.1、Trigger Condition
Trigger condition contains the following params.
Event Type
Action Increasing
Action Decreasing
Face Emerging
Face Vanishing
Timer
Action Type
It takes effects only if the “Event Type” is “Action Increasing” or “Action Decreasing”.
Blink
Lips Part
Head Yaw
Head Pitch
Brow Jump
Lip Corner Puller
Threshold: Amplitude of motion
It takes effects only if the “Event Type” is “Action Increasing” or “Action Decreasing”. Its valid region is 0 – 1.
Min Interval: The minimum interval between two triggers.
Time: The time elapse triggered.
It takes effects only if the “Event Type” is “Timer”.
Period: Period length in periodic switch.
It takes effects only if the “Event Type” is “Timer”. 0 means infinite loop.
Count: Loop count
It takes effects only if the “Event Type” is “Timer”. 0 means infinite loop.
Notes
The unit of Min Interval, Time and Period is milliseconds. 1 second equals to 1000 milliseconds.
Time set by the “Timer Trigger” is counted on the basis of the time when the AR Scene package is loaded.
9.2、Trigger object switch
Type: Trigger type
Change Animation
Play Sound
Play Music
Stop Music
Layer ID: Model name
It takes effects only if the “Type” is “Change Animation”.
Animation Name: Animation name in the model
It takes effects only if the “Type” is “Change Animation”.
Start Index: Sequence number in the cycle
Default value is -1. Meaning this trigger doesn’t repeat.
Index Period The total number of triggers in the cycle
Timeout Single trigger delay trigger time length
The starting point of Timeout is the moment when the "trigger condition" is reached most recently
Loops Cycles
Only valid when Type is Play Sound and Play Music, 0 means infinite loop
Notes
When using the Timeout parameter, if the trigger action is made again within the specified time (the trigger condition is reached), the state of the timeout will be refreshed. If the trigger action is frequently made, the effect corresponding to the timeout may not be seen.
The difference between Sound and Music. Sound is used to control short-duration sound effects, while Music controls background music. Music is generally the only one in the same effect package.
9.3、Action trigger example
An animation can have multiple trigger events. The following example uses Action Increasing and Face Emerging. Here we make a helmet wearing effect, recognize the face and wear the helmet, and trigger the lifting and falling of the helmet mask by opening the mouth. Make the helmet model in the 3D software, and make the required animation clips a1, a2, a3 in the same model.
a1 represents the cartoon clip with the helmet on the head;
a2 represents the animation clip of the helmet lifted up;
a3 represents the animation clip with the helmet visor dropped.
Import the helmet model and adjust the initial parameters of the model as shown below:
1：First of all, we need to automatically put on the helmet when the face appears, that is, apply the animation clip a1. Then add the first trigger Event1, set the Event Type to Face Emerging (face appears), and Add Trigger to add a trigger object. The trigger object settings are as follows:
2：Later, when you open your mouth for the first time, apply animation clip a2 to raise the mask; when you open your mouth for the second time, apply animation clip a3 to let the mask down. After that, opening your mouth continuously will make the animation loop between a2 and a3. Then add a second trigger Event2.
Add Trigger twice to set a2 and a3 as trigger objects respectively, and set the Index Period cycle period of both trigger objects to 2. Set the Start Index of a2 to 0, which means that a2 will be executed first in a single loop. Set the Start Index of a3 to 1, which means that a3 will be executed second in a single loop.
The settings are as follows:
9.4、Time trigger example
This example uses the helmet model in the previous example, selects the helmet props (regardless of whether there is a recognized face), and starts to perform the animation of the mask lifting and falling after 5 seconds.
1: Set the Time value to 5000, the helmet will start to perform the animation a2 when the helmet is lifted after 5 seconds,
2: Set the Period value to 2000, which means that the trigger interval of a single Trigger is 2 seconds. After 2 seconds, the helmet starts to execute the animation a3 of the mask falling, and then loops the animation effects of a2 and a3. As shown below:\
10、Fake Face function
Fake Face can fit a single image closely to the face, and enable the texture to change with changes in facial expressions. Usually used to make face painting and beauty makeup.
10.1、Fake Faceparameter
To make the Fake Face effect, first draw the texture with other software (for drawing, please refer to the model UV map fakeface_UV.png attached to the tool), and then use the plug-in to fit the texture to the face model. In addition to allowing the import of textures, the plug-in also provides adjustments to the lighting and overlay methods to achieve a more realistic effect. The specific parameters are as follows:
Shading Mode
No Shading
Standard Shading
Fake Face Map
Blending Mode
Normal
Multiply
Subtract
Screen
Add
Exclusion
10.2、Example
1. Check Fake Face in the Fake Face panel of the plugin。There will be a fakeface model without textures（White area of the screen）。
2. Click on the dot behind Fake Face Map, and add a map drawn according to the UV grid map
3. Select the appropriate overlay mode, the default is Normal11、AR Scene Generation
11.1、AR Scene Package Generation
A. Set values in “AR Scene Editor”,
AR Scene Package Version： Each update of the package without altering the ID will add one to the version.
Max Face： Max number of faces supported, 0 as default, no upper limits.
Use Z-Buffer：Enable Z buffer, which is enabled by default, If the rendering order doesn`t matter, users can disable it to enhance the efficiency of SDK.
Anti-Aliasing：Anti-aliasing。
Notes
“AR Scene Package Id” can be altered manually, but not all characters are supported, no space in the end.
12、Make a face sticker with 2D foreground
1. Use Meishe special effects tools to make a 9v16 ratio foreground filter, and select the album mode to export.
2. Decompress the resulting album compression package.
3. Copy the foreground material and "fx9v16.xml" in the album folder to the 3D model folder exported by unity, and then use the 3D model folder to operate, the album folder is no longer used
4. Open the "fx9v16.xml" file and delete the "uuid=" field
5. Open the "arscene9v16.xml" file and add the "" field
6. Compress the unity folder into a zip, and add the suffix ".1" to the compressed package name
注意
给压缩包而不是文件夹添加后缀，给文件夹添加后缀会导致生成的ARScene文件读取失败
Notes
Add a suffix to the compressed package instead of the folder. Adding a suffix to the folder will cause the generated ARScene file to fail to read
7、If you want to modify the effect of the ARScene, you need to open the "info.json" file, and each time you modify the value of version +1, the number of suffixes of the compressed package also needs to be correspondingly +1
8. If you need to change the ID, use "UUIDFactory.exe", select Without braces, click Generate, generate a random ID, click Copy to copy this ID, and replace the uuid in the info file with this ID
9. Put the compressed package into the "arscene" folder in the "SDK conversion tool PackageTools" folder, double-click the "arscene.bat" file or "arscene.bat-shortcut" to generate the arscene file.
13. Make the trigger effect of 2D foreground/2D face sticker
2D elements cannot be directly triggered, only 2D foreground or 2D face stickers can be written into the script to trigger the script.
13.1 2D foreground trigger
A. Copy the material and "fx9v16.xml" file in the album generated by the Meishe special effects tool to the folder exported by unity and modify the "fx9v16.xml" file(For specific operations, please refer to Steps 1, 2, 3, 4, and 5 of 12. Make a 2D foreground face sticker)
B. Add the "faceActiOnDetection="true"" field to the "arscene9v16.xml" file. This field indicates that the SDK will read the movement of the face. If this field is not added, the expression trigger will not be recognized. If the exported 3D model has a trigger function, this field will be included in the script.
C. Write the "disable="1"" field to the added 2D layer. This field indicates that this layer is hidden at the beginning, and will not appear until triggered
D. Write the "name=" field to the added 2D layer. This field is used to name the script that needs to be triggered in the trigger. The name can be any name you want. The source needs to correspond to the corresponding xml name. If the corresponding xml needs to be renamed, for example, when multiple scripts need to be triggered, the name needs to be renamed here.
E. Add the trigger event layer. For the writing and meaning of event specific parameters, please refer to the 9.Event trigger chapter and the attachment "arscene-desc-sample-event.xml".
Among them, "fxOn" means to trigger the script, and "fxOff" means to stop the script. The effect of the example in the figure is that the 2D foreground layer will be triggered when the mouth is opened to 0.3 degree, and it will stop after running for 2200ms.
F. Add the fields shown in the figure to the info, and the corresponding text prompt will appear when the face sticker is used. Pay attention to the field format, the comma cannot be omitted.
G. Use the folder to generate the arscene file. (For specific operations, refer to Steps 6, 7, 8, and 9 of 12. Make a 2D foreground face sticker)
13.2 Triggering of 2D face stickers
A. Use unity to generate 2D face stickers (for specific operations refer to 8. Unity 2D stickers production specifications)
B. Add the "faceActiOnDetection="true"" field to the "arscene9v16.xml" file. This field indicates that the SDK will read the movement of the face. If this field is not added, the expression trigger will not be recognized.
C. Write the "disable="1"" field to the 3D layer. This field indicates that this layer is hidden at the beginning and will not appear until triggered. The name defaults to main3d, and you can modify its name when you need to trigger multiple 3D scripts. The source needs to correspond to the corresponding xml name. If the corresponding xml needs to be renamed, the name also needs to be renamed here.
。
D. Add the trigger event layer. For the writing and meaning of event specific parameters, please refer to the 9.Event trigger chapter and the attachment "arscene-desc-sample-event.xml".
Among them, "fxOn" means to trigger the script, and "fxOff" means to stop the script. The effect of the example in the figure is that the 2D face sticker will be triggered when the mouth is opened to 0.3 degree, and it will stop after running for 2200ms.
E. Add the fields shown in the figure to the info, and the corresponding text prompt will appear when the face sticker is used. Pay attention to the field format, the comma cannot be missing.
F. Use the folder to generate the arscene file. (For specific operations, refer to Steps 6, 7, 8, and 9 of 12. Make a 2D foreground face sticker)
14、Test in Mobile Device
Import the generated AR Scene package to the MeiShe SDK Demo. Test the effect and tune attributes in AR Scene plugin.
Android Demo
IOS Demo