How Does a Database Use IO?

We have a shiny new database server. We ran some tests on it, and found it was about the same speed as the old one. Worryingly the IO system seemed to be struggling to keep up with the load.

In case you don’t know, IO is short for Input/Output, and is used to describe the use of a hard disc, or similar storage. We all know that hard discs are slow compared to memory, but for a database to work well, the IO system needs to be able to keep up with the work the database is doing. Otherwise the database will eventually appear to lock up.

We use a Virtual Private Network - a VPN to access all our work servers. But we use MS teams for meetings, and our VPN admin asks us not to to push loads of video over the VPN. So what to do? Disconnecting from the VPN every time a call came in got old pretty fast. And what do you do if you want to show a colleague something which required VPN access?

A colleague had created a new service operation in peoplesoft integration broker, but it wasn’t working. After a lot of investigation, and a call to Oracle, we found the cause was SNI: Service Name Indication. This is a facility which allows several different hostnames to be on the same IP address, but still listen in HTTPS.

It seems the Integration Broker converts the hostname to an IP address before trying to contact the remote site. The problem is then that the remote site doesn’t know which website it wishes to talk to, and this causes a handshake failure. The messages in the logs are not helpful to say the least.

Silent install

A new feature of 8.58 is the silent installation. The documentation says you can run a command such as:

1
2
3
4

./psft_dpk_setup.sh \
  --silent \
  --response_file=response.txt \
  --customization_file=psft_customizations.yaml

The customizations file is optional, but I decided this time around to use it as much as possible to create a system as close as possible to what I wanted to end up with. So I copied the file from <psft_base_dir>/dpk/puppet/production/data/psft_customizations.yaml and edited it to my satisfaction. <psft_base_dir> above is whatever was specified when the installer asked interactively, or in the response file. This means I had to do a test install before I could do a proper one, but that is fine.

We recently purchased a new database server. We ran a performance test which our brilliant testing team have put together and found out the system with the new server was slightly slower than the system with the old one. The testing team was happy that the test was a pass - the users wouldn’t notice the difference. However, new servers are supposed to be quicker than old ones, so I was disappointed. I investigated a little further, and found that the AWR report showed there were more waits on IO on the new hardware than on the old.

Redo Investigation

I was investigating a performance issue on new hardware and installed SLOB to do so. I discovered in the AWR report, that the redo generated per transaction was around four times that in development.

I checked everything I could think of, including copying the parameter file from development to the new hardware and using that, but the redo stayed stubbornly high. I asked the oracle-l mailing list for ideas. Jonathan Lewis responded suggesting the high redo could be caused by the following:

Django on RHEL7

Here is how I chose to set up Django on RHEL7. This is somewhat painful because the default Python in RHEL7 is still version 2. Using this to create any new programs seems foolish as it has already been de-supported by the Python project, so I really have to use Python 3.

Set up Linux

We need Python 3, Django, Apache, and Postgresql. See the instructions linked in the next section if the Postgresql packages aren’t found. The version that comes with RHEL7 is too old for the version of Django we want to install.

As discussed previously, we use regpg to manage Ansible secrets. This has been really useful. One annoyance though is that some tasks can take up to 6 hours to run, but the gpg agent only caches the gpg passphrase for 10 minutes or so. I end up having to type the passphrase in several times during a run. I occasionally kick off a run before I leave for the day. It would be a shame if it was stalled overnight due to waiting for a passphrase.