Xcode is a soft dependency of Homebrew, but if you don't intend on using Xcode then it seems a waste to install the 4 GB program.

Instead, you can just install the command line tools for Xcode, which comes up to about 8K. Much better!