Any system which allows you to locally decrypt information, for the purpose of doing anything for the user, should be assumed to allow an attacker who roots the box to locally decrypt information. That's the unfortunate harsh technical reality.
If you have compliance reasons motivating this need for encryption, you'll find that e.g. HIPAA and PCI-DSS ignore technical reality, in favor of requiring that you encrypt information stored at rest and imposing substantial penalties on you if it leaks. There are a variety of ways to do this. One fairly common one for HIPAA-compliant applications is putting the e.g. MySQL data files on a partition which is block-level encrypted. You then issue decryption keys to folks who need them, such as e.g. the application.
If your host is totally compromised, the host holds both the decryption key and the ciphertext, which means "Sucks to be you." However, this does provide non-zero increase in security (e.g. if an old copy of the DB drive ends up floating off to eBay because of poor physical control on your part, and you can document that it doesn't include the encryption key, you just avoided a reportable information breach), and it does check the appropriate boxes on e.g. HIPAA.
To expand on this a bit, absolute security for encryption just doesn't exist. If you wanted your data 100% secure, put it in a database, disconnect the DB from your network, put it in a locked room, guarded by biometric locks and security guards. Even in that scenario, the data is vulnerable, but why even bother discussing that point, as the data is worthless if you can't access it.
With that reality in mind, I was responsible for PCI for a large part of the infrastructure at a Level 1 Merchant, meaning a yearly audit had to be passed. Ultimately, our solutions boiled down to restricting access to an external (read different machine/network segment), firewalled host that did the decryption. In some cases this was an appliance that was purchased (this helps with compliance, but they're expensive, and they're a nightmare if they become a performance bottleneck as they're a black box you know little about). In other cases we used a web service we built that worked similarly (auditors will pick this apart because it isn't a "standard" solution).
In all cases here is a high level of how they work: encrypted data is passed to the service, which pulls the encryption key out of memory, decrypts the data, and sends it back to the requesting host. The encryption key is stored in (at least) two pieces, each piece is encrypted with a key encrypting key, key encrypting keys are know to very few employees, no single employee holds both key encrypting keys. The encryption keys is only assembled in its entirety while in memory.
Again, there are problems to this, as patio11 intimates, compliance includes much theater a times, but this is reality, and it does provide benefit over other solution, in this case, at least three layers of security must be compromised before you could decrypt everything.
I'm in ecommerce as well and I've seen PCI/DSS auditors require vendors/hosts to rearchitect using an encryption/key management appliance. You wrote you built your own solution - are there no known, trusted open source alternatives? As you mention, the appliances are almost astronomically priced, so it seems like an area OSS (or a disruptive startup) would help.
Also, no mention was made of architecture. If this is a web app then putting a network and application firewall between the web application and database will allow you to observe traffic and then block or refer requests that attempt to access volumes or types of data that don't match typical usage patterns. Even better would be a dedicated DB API secured as required, and not allow the web application direct access to the database. At an absolute minimum, grant the web application rights to access only required stored procedures, and deny direct access to tables and views.
If there's a client (desktop, phone, tablet or another server) then keys should be generated there, limiting the impact of disclosure should the DB be lifted.
[Edit] Forgot to add that it's always worth creating a threat model. Draw a box around each process. Enumerate the inputs and outputs that cross process boundaries. Consider how each might be attacked, and how each might be secured. There are 6 types of threat to worry about - spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege.
If you have compliance reasons motivating this need for encryption, you'll find that e.g. HIPAA and PCI-DSS ignore technical reality, in favor of requiring that you encrypt information stored at rest and imposing substantial penalties on you if it leaks. There are a variety of ways to do this. One fairly common one for HIPAA-compliant applications is putting the e.g. MySQL data files on a partition which is block-level encrypted. You then issue decryption keys to folks who need them, such as e.g. the application.
If your host is totally compromised, the host holds both the decryption key and the ciphertext, which means "Sucks to be you." However, this does provide non-zero increase in security (e.g. if an old copy of the DB drive ends up floating off to eBay because of poor physical control on your part, and you can document that it doesn't include the encryption key, you just avoided a reportable information breach), and it does check the appropriate boxes on e.g. HIPAA.