You're free to ignore the artist's process, but if you do you're likely to make a program that's great for programmers but terrible for artists and/or game designers (the two tend to be one in the same.) And more often than not, the programmer is spending more time in the code of the engine than in the engine itself. If you want it to actually be usable by anyone other than a programmer, remember that the end-product is something that helps a team make a game, not something for you to program. Design for how the end-product is going to be used, not for whatever interests you at the moment.
Then again, you're free to take the immediate programmer's approach and just program as you feel like it. Since this is your first time working on anything like this, acknowledge that this is a learning experience rather than a practical one. Try to combine the two and, I promise, you will fail miserably. Trying to design when you don't know what you're designing for/against tends to go badly unless you have guidance. Study other engines as much as you like, but you're only looking at the API, not the architecture. An engine where you can get a feel for the architecture through the API is not an engine that should be studied.
In terms of mathematics, I highly recommend 3D Math Primer for Graphics and Game Development by Fletcher Dunn/Ian Parberry. It covers a LOT of required mathematics in linear algebra for multiple coordinate systems and also provides wonderful explanations and diagrams for an explanation.
(Note that I have the second edition, which is linked below. I can't comment on the first edition.)
Another suggested read would be Game Engine Architecture, by Jason Gregory. Note that this is more of a book of guidelines than a comprehensive walkthrough, but despite that it still packs a lot of content. Definitely indicative of how much a full engine can have. It'll let you know what you might need and how you might go about handling it. It does not
, however, explicitly go over systems interaction and the architecture as a whole. Ironic, but this seems to be the case. Take that note with a grain of salt, though, since I don't have the book right in front of me to reference. (Currently being borrowed by a friend.)