Autocad - problem with fillet two objects

Generally this problem appears if the fillet radius is to big. Try a smaller fillet radius and check if this problem still appears

Here is the example file

I tried your file and found the reason for your problem.
You tried to fillet polylines and arcs.
Use the command "explode" and apply this command to the 2 Polylines edging your arcs in the file. This will change your polylines to ordinary lines and you will be able to fillet them with the arcs.

Here you go: handmade fillet with a circle and trim. Handy for when the program says no. Like filleting polylines or other invalid for some reasons.