Check your Town code. It's more than common courtesey. Most Towns require that the "nice" side of the fence faces out, or on your neighbor's side.

People do all kinds of things that don't follow code, but generally, the side of the fence that has the cross beams, and where you can see the posts should be on the owner's side.