There are several ways to change key in a song. You could go from the E to Am rather than A. There are then three keys you could use, C, G or F. To get back to the original key of A just play an E or E7. Listen to the chord progression to make sure it sounds good to your ear.
Alternatively you could follow the circle of fifths. From E you go to C, then G, then D then back to A.
It gets to be a bit like maths and can become really complicated.
Finally you could just pick a key and move straight there, when you've finished in that key just jump back. It can sound weird but if the lyrics are weird they could both work together.
Just experiment and see what sounds good and what doesn't.
Hope this helps. If you have any questions on what I've written, just ask.
Keith