You need to select the coinciding sliding line and plane for the geometry. Have a look at my example.

Here is a file.

After checking up your geometry, I came with idea to create prismatic joint (yet this is somewhat advanced trick). First thing to keep in mind is what kind of constrains are needed to create any particular joint and then work accordingly.

In this, I have created a line using point on brown part as it doesn't have any tooth structure as in above example (two lines are ready for prismatic joint)

After this, there should be two same plane such that they slide on the plane. For this I created one plane on green and other on brown part (make sure to have same plane orientation).

Now all the conditions for prismatic joint are over, then create a joint. This time you will get results.