I was just wondering about how I can delete all stuff of a port MacPorts fetched from the internet and built from the source files.

It’s quite simple. You just need to use the option --all .

Example: sudo port clean --all cppunit