Topology optimization is just one big math problem. First of all you will need to understand how FEA works, because thats where an optimized model is derived from. When a part is put through FEA (with the correct boundary conditions) we can see high levels and low levels of stress in the part. This is where topology optimization comes in. The goal is to remove the most amount of material from a part and still keep structural properties. We then set up an equation that says:
Remove areas of stress lower than "X"
The software (Ansys) then removes that material and leaves a rather rough looking model.