The solution was much simpler than what I expected, I should just accept that some problems is just brute force...
It might be fun.