So I set out with what I thought was a simple idea:
“Start with an image of someone’s face and turn that into a spooky Halloween character, with costume, makeup and full Facepaint with a spooky background.”
BUT it had to look enough like them at the end - that they would be pleased with the result…
The starting point was easy - I wanted to train a Halloween LoRA on lots of images of people wearing Halloween Facepaint - so I did that…
A couple of the 48 images i used to train with:
So I had a Flux LoRA - now I tested that in Tensor.Art with simple “Man in Halloween Facepaint”, “Woman in Halloween Facepaint”
So far so good, I thought ok, this is going to be easy peasy!
At this point (End of September 2024) there were limited options in TA for Flux Face swap… (No Pulid available then) so I started trying with Facedetailer…
I built out the workflow - made a separate flow for the background - and was all excited…
But no matter what i tried (and I tried a lot!) the facedetailer would wipe out the Facepaint from the Lora - restoring the face back to the original person, nice and clean, or with a half hearted smear of greasepaint.
Or it would look nothing at all like the person and the makeup would look like it was a badly stuck on mask…
So i went back to my Discord buddies and we talked about the options - and decided to try Reactor nodes with insightface…
It would generate a Florence description of the original reference face (cropped) - build a dummy Halloween Image with a lookielikie from the description and with Facepaint - and then reactor the ref face back over the top (or so i thought)
But the Reactor’d one cleaned up the face and removed 90% of the makeup and it didn’t want to do the costume or background at all the way I had envisaged… as soon as I gave it enough freedom to be creative, the reference person was lost completely…
I think by now people in all my discord groups were sick of me asking for ideas on how to do this - I tried every setting and balance on reactor nodes.
Could I use an llm to rewrite the visual description of the face to include the Halloween description first, and so on.
I looked at IPAdapter and using Depth maps - but although they captured the shape of the face - they couldn’t preserve the familiar features through costume stylemakeup.
At this point - I pretty much gave up in disgust… I put out a final round of help requests on various discord’s and went onto another project
A few days later my good friend told me “ hey - finally they released Pulid for Flux on TA!” - and I already had built Flux Pulid workflows for face swapping the previous week on my MimicPC Cloud version of Comfyui (where you can load any kind of node and model you want and really design and play with freedom) so I started to regain my enthusiasm…
I managed to merge some of the earlier ideas for generating the Halloween style with LLM’s and a Joycaption of the cropped reference face - and the Flux Pulid face swaps - and experimented with the positioning of the LoRA to get maximum effect - and was finally able to release a workflow and AI Tool that did what i had seen in my head those few weeks back when I started… https://tensor.art/template/785795972520313546
And the workflow - https://tensor.art/workflows/785793305345589081
And the LoRA - https://tensor.art/models/785804669831296337
If you have enjoyed my article - please like and use my AI Tools and Models…
I welcome comments and constructive feedback.
