Portfolios

I’m a bit new to this so forgive me if this is a stupid question. If you are working as a contractor, is it generally okay to take work you produce and use it in your portfolio or do you always need permission?