Many WordPress themes have simple control panels that let you change the copyright area or insert Google Analytics into your design. Add To All is a free WordPress plugin from Ajay Dsouza that extends this functionality by letting you add custom code to your header, footer, sidebar, content or feed.
At the moment the plugin lets you enter your account details for Google Analytics, Statcounter and Kontera though more statistic and affiliate services are planned for future services.
In the header you can add custom CSS and custom HTML to wp_head. HTML can be added to your footer to wp_footer and content can be added before and after your feed content. A copyright notice can also be added directly to your feed. Unfortunately, PHP can not be inserted into any of the content areas.
There are plugins available that let you insert Google Analytics into your website and there are plugins that let you customise the output of your feed. Very few let you insert code into all areas of your site which is why I believe Add To All is such a useful plugin. I recommend checking it out if you are looking for a simple way to insert code into your website.