Is frugal and ethical/organic grocery shopping possible?

I'm generally a pretty frugal person, but I struggle when it comes to grocery shopping. Organic animal products and vegetables are significantly more expensive than the normal alternatives, which makes it hard to justify, but at the same time I worry about long term side effects (and potential costs) of non organic food as well as the ethical dilemma around animal treatment.

Is there a common ground here? Is it possible to be frugal at the grocery store while still getting quality food? Any advice would be very appreciated!