Posts on Curious DBA

Patching using Auto Upgrade

Sat, 16 May 2026 09:15:10 +0000

Oracle are recommending we use out of place patching, and AutoUpgrade for patching. Until now we have used a custom patch download script and an ansible patching process. Let’s see how AutoUpgrade works for patching.

Documentation is in the Oracle Database Upgrade guide under AutoUpgrade Patching. As with many Oracle things, it takes some effort to understand how it is supposed to work.

Downloading Auto Upgrade

We download auto upgrade from Oracle Support document KB123450, or from the Oracle website So we can do:

PeopleSoft Change Assistant

Tue, 06 Jan 2026 11:59:47 +0000

I am looking into running Change Assistant automatically, but also with the option of manually running it if necessary. The problem with this is permissions. We are supposed to log in as our own users and not as admins if we can help it.

Inspiration

The clever people at psadmin.io came up with a method which is pretty good start. There are a couple of refinements to this I felt needed to be made. These mean that the solution is a bit more complicated than the above.

PeopleSoft in Containers

Fri, 28 Nov 2025 09:13:40 +0000

warning

TL;DR: If you need Cobol, there is no supported way to get this working.

Oracle mentioned that PeopleSoft is supported in Containers. So, with great excitement I decided to try to get it working.

The general approach is as follows:

Install the build environment

This uses the DPK. We need to ensure Podman is installed, and the associated build tools. Then we run the DPK install to create the build environment:

Setting up VSCode for Ansible on Windows using Podman

Wed, 21 May 2025 15:10:49 +0000

My work laptop runs Windows. I would prefer to run Linux, but some programs I need only work on Windows. Another program I need, Ansible, does not support Windows. I get round this by not running it on Windows, but it would be nice to be able to develop on my laptop to avoid frustration with network slow downs etc.

VSCode supports dev containers. This is great, because development happens in a container. But the officially supported way of doing this is with Docker. Docker Desktop for Windows is not free. However, Podman desktop is, so we can use that.

Auto Restart Oracle Databases with Systemd

Wed, 02 Apr 2025 14:19:07 +0100

The Problem

Our databases didn’t stop and start automatically with the OS. The oracle supplied scripts don’t cater for standby databases. Lastly we have one listener per database. Whether this is necessary or not is open to debate, but I would like to make our databases stop and start with the operating system. I’d also like the following features:

Stop all the databases whether or not they are in the oratab on shutdown.
Start all database that were stopped on startup.
Deal with standby and logical standby databases.
Deal with our unusual listener setup which has one listener per database instance.

Oracle provides scripts called dbstart and dbshut in $ORACLE_HOME/bin which are documented in the administrator guide, but they only start and stop databases in the oratab.

Port and TLS scanning with nmap

Tue, 14 Jan 2025 10:02:47 +0000

I have had a couple of instances where I have needed to look at which ports are open. On one occasion a firewall change meant I needed to check in a hurry whether ports I needed were open. On another instance, another team raised concerns with some of the TLS ciphers listening on some of the ports in our system.

I do not recommend port scanning across the internet. All these scans were completed within my employers infrastructure as part of my job. If you would like to try these commands and don’t have a similar job to me, I would suggest using devices on your home network such as a raspberry pi, or scanning a VM running on your laptop. The nmap security scanning book has a chapter on legal issues.

Getting Shorter Garbage Collection Pauses

Fri, 20 Dec 2024 16:59:47 +0000

We noticed that our test PeopleSoft system was very slow on occasion, such that the load balancer decided it was broken. Sessions were redirected to the webpage we have for the system being in maintenance.

Since the webserver is Weblogic, it runs in a Java Virtual Machine (JVM). The first thing to check is how long the garbage collection pauses are. Fortunately I had garbage collection logging switched on, so I could see that they were over 100 seconds on occasion, which is far too long. This is what my garbage collection log parameters were set to (Java 11).

The Problem with Ansible on RedHat

Wed, 11 Dec 2024 09:59:47 +0000

Normally newer versions of Operating systems have newer packages. But not RedHat when it comes to Ansible. On my workstation:

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.10 (Ootpa)
$ ansible --version
ansible [core 2.16.3]
 config file = /etc/ansible/ansible.cfg
 ...
 python version = 3.12.6
 jinja version = 3.1.2
 libyaml = True

But on the management server:

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.4 (Plow)
$ ansible --version
ansible [core 2.14.17]
 config file = /etc/ansible/ansible.cfg
 ...
 python version = 3.9.18
 jinja version = 3.1.2
 libyaml = True

Wait, what? The older operating system has a newer version of Ansible, and Python?

Refreshing a Test PeopleSoft Campus Environment

Thu, 28 Nov 2024 09:57:49 +0000

We have a number of PeopleSoft test environments. I have written about my automated build process before, but I have not yet mentioned what we do to the database when we refresh.

My approach here is that I want as much as possible to build the environment from scratch. This means that we have a consistent build. There are also database fields that need to be changed. Recently a colleague and I reviewed the tables that needed changing and came up with the following.

Peoplesoft Log Parsing with Regular Expressions

Wed, 25 Sep 2024 09:00:49 +0000

As mentioned in my previous post on this topic, we need to configure BindPlane to read our files. I chose to use the file source as there was no prepared parser for PeopleSoft logs.

Configuration again

Application server and process Scheduler

On the application server there are three types of files. These are:

Application Logs

I set this up as a file, and added the following regex to split it into fields. This is wrapped for readability - in reality it is all on one line. The spaces are part of the regex, the newlines are not.

Sending Logs to Google Observability Logging

Fri, 02 Aug 2024 15:00:49 +0000

At present all our logs are in random files in random places in the operating system. I would like to see whether we can improve this. As an example of the problems caused by the current situation, if a user reports an issue, they normally don’t give a timestamp, so we have to assume the issue occurs say up to 20 minutes before the call was raised. Then we have to search the logs on the operating system for their user ID. As mentioned these logs are in various places. It isn’t easy to limit to a time span using standard operating system tools. Also we have a redundant architecture, meaning the users session could have been on any one of four web servers, and four application servers. The error could have happened on any of these 8 VMs.

Notes on Caching

Fri, 14 Jun 2024 08:00:49 +0000

Just a quick one this time. PeopleSoft has three different types of cache at the application server level, and here are some notes on it.

Unshared Cache

What it is

So called because each application server process creates it’s own cache under %PS_SERVDIR% which is by default $PS_CFG_HOME/domainname/CACHE/ Under here are directories for each process, e.g. PSAPPSRV_1 etc.

How to use it

This is the default type of cache. It is used if you configure

Oracle Backup Restore Failures

Fri, 10 May 2024 14:15:00 +0000

How I Test My Backups

I like to test my backups. It helps me sleep to know I could get my data back if the worst happened and it was scrambled by ransomware, or a bug in our code.

My sleep was rendered less peaceful when the restores suddenly started failing for no reason that I could understand. We use RMAN to backup and restore the data, and the script is fairly simple - it effectively says to restore the database as it was at noon yesterday. Something like this:

Process Scheduler Auto Update

Fri, 26 Apr 2024 16:16:00 +0000

I have been setting up process monitor auto update, and have managed to get it to work - I can see the process status updates in the process monitor screen. Here is what I did.

Oracle support document id 2772617.1 explains how to set this up manually. I wanted this to work as part of the automated build, which means supplying the parameters as part of psft_customisations.yml

A Gotcha!

We have multiple application servers running on the same port (but on different VMs). This means we need different domain IDs for each process scheduler because domain ID and port are used as a key. I would have thought that the hostname should be included to make this a unique identifier, but Oracle have chosen not to do that. Note that domain ID is different from domain name. Oracle documentation suggests using the database name in lower case. The DPK default is APPDOM (which is also the default domain name). If either of these is used, when you set up the inter domain event credentials on the process scheduler and configure the domain (for example by running):

Installing Change Assistant

Fri, 22 Mar 2024 16:57:49 +0000

I am not sure what best practice would be as to a location to install and run change assistant. To get the GUI it has to be on Windows. Since we use a VPN we can’t really connect from a laptop as if the VPN drops out we will interrupt the process which can run for several hours. So at present we use a Windows server which has a full PeopleSoft installation to run change assistant.

Django on RedHat 9

Tue, 12 Mar 2024 17:03:50 +0000

Django on RHEL9

The reason I am revisiting my Django website is because I am updating the operating system to RHEL9. At the same time I decided to take the opportunity to correct a couple of things I was unhappy with in the previous setup.

I prefer to take a clean VM and build from that. That way we know the exact configuration and can document it.

Set up Linux

We need to install the following (Using YUM)

More Django and SAML

Mon, 03 Jul 2023 15:19:07 +0100

Some time ago I set up a website using Django which I protected using SAML and Python Social Auth. I drafted this post, as a follow up to my original one but never published it until now.

The way to integrate SAML into Django, indeed to integrate most Single Sign On/Identity provider solutions into most python based websites (Sorry, Service Providers) is to use Python Social Auth.

Installation

We need to ensure the following are installed in the environment

Controlling Memory and Swap Usage in Linux with Systemd

Wed, 07 Jun 2023 09:15:07 +0100

The Problem

We have an application that uses a lot of memory when a user takes particular actions. I feel the application should take steps to protect itself - it shouldn’t allow an action from a user which will cause issues. I wasn’t able to influence the application so we lived with occasional out of memory issues, and times where processes could not be forked due to lack of memory.

PeopleSoft Critical patches on Windows

Thu, 30 Jun 2022 11:12:07 +0100

Back to my series about applying patches to PeopleTools 8.59. I am now visiting the Windows process scheduler. Like the Unix servers, Oracle installs all the software, so it all needs to be patched. But there are always additional complications with Windows.

Java Issues

Java Patch Download

If the same procedure as Unix is followed to download Java, we end up with an executable installer. The only silent install option is /s - there is no way to specify the folder it ends up in. Also this does a Windows install updating the registry etc. It’s likely not to want to have two copies of the same Java in different locations.

Encrypting PeopleSoft Internal Communication - Elastic Search and Kibana

Fri, 24 Jun 2022 08:24:07 +0100

Now we have the core PeopleSoft stack communicating encrypting traffic, we need to address the add-ons. Elastic search is becoming more important, and it is difficult to use PeopleSoft without it. Here is how to add encryption to Elastic Search and Kibana.

There is two way communication between Elastic search and PeopleSoft, Both of these directions need to be configured. Fortunately we have already done some of this work.

This procedure is documented in PeopleBooks under Products -> Development Tools -> Search Technology -> Working with PeopleSoft Search Framework Security Features. Here are links for Elasticsearch and Kibana

Character Sets and Field Sizes

Tue, 14 Jun 2022 16:20:07 +0100

Here is an interesting problem we had recently. A field which should be length 1 was being reported in some cases to be length 4. How is this possible?

The Problem

I use dbms_sql.describe_columns to discover the size of the field in a remote database. I use this value to recreate a copy of the view in the remote database as a table locally. This is used for testing. We can mimic this in the local database as follows:

PeopleSoft 8.59 Infra DPK and Critical Patch Updates - Tuxedo and DB

Thu, 26 May 2022 09:16:07 +0100

Further to my last article about applying critical patches to the web server, lets take a look at the application and database servers.

Which Servers?

In my mind WebLogic runs on the web tier, and Tuxedo runs on the Application and process scheduler tier. The Oracle Client is also on the Application and process scheduler tier. However, since these and Java are installed on all tiers, we should apply the patches on all tiers.

PeopleSoft 8.59 Infra DPK and Critical Patch Updates

Thu, 19 May 2022 09:11:07 +0100

As I write this it is a month after Oracle released the critical patch updates, but there is still no sign of the Infra DPK, which contains Java and WebLogic updates. If Oracle are not going to supply this patch reliably, we will have to work out how to do it ourselves.

WebLogic (On the Web Server)

What WebLogic Patches does Oracle Install?

In a default install we can do the following as user psadm1:

Encrypting PeopleSoft Internal Communication - Application Server

Thu, 12 May 2022 10:45:07 +0100

In the last article we looked at encrypting communication between WebLogic and the load balancer. Now it is time to investigate the traffic between WebLogic and the Application server. Without this configuration the logs get filled with messages like this:

WARNING: LLE Configuration discovered!
 Note that LLE has been deprecated.
 You should upgrade to SSL to secure network links.

Let’s upgrade to SSL then!

Application Server

We already discussed TLS with regards to the Integration Broker and the Web Server. The application server conceptually works in the same way, but it (mostly) isn’t written in Java, so in practice the procedure is slightly different.

Encrypting PeopleSoft Internal Communication - WebLogic Server

Thu, 05 May 2022 16:50:07 +0100

In the past we used to assume communication on our network was protected by physical security. Now that seems not to be a reasonable assumption. So we should probably encrypt communication within PeopleSoft. Here is how I do that.

We control both ends of communication between tiers of PeopleSoft. So we can create our own certificate authority, and instruct our software to trust it.

Creating the Certificate Authority (CA)

We need a root certificate for our Certificate Authority (CA). I created it as follows:

TLS and PeopleSoft Integration Gateway

Thu, 28 Apr 2022 16:11:07 +0100

PeopleSoft in general leaves it to the administrator to ensure that digital certificates are set up properly. Given digital certificates don’t tend to change very often, and the provider changes even less frequently, it can be difficult to understand and remember how this works, and prevent issues.

What is a digital certificate?

The two parts

There are two parts to a digital certificate. One part is the private key which is used to encrypt data, and is installed, in our case, on the load balancer. It is important to keep this safe so nobody can impersonate our system. The public key is what we are mostly dealing with here. This is available to anyone and can be used to decrypt the data, and check it came from someone we trust.

Auto Restart Weblogic with Systemd

Thu, 03 Mar 2022 12:19:07 +0100

The Problem

Our WebLogic server recently crashed. There were a number of issues identified in the post mortem, which were addressed to prevent the same thing happening again. It also occurred to us that WebLogic should restart itself automatically if it fell over. This is easily achieved using systemd, but for whatever reason Oracle chose not to configure it to do this.

Oracles Default Setup

This is a PeopleSoft system which might be configured slightly differently to other WebLogic installations. PeopleSoft by default provides a systemd unit file which is mostly copyright warnings, so I won’t include it here. Oracle have created a legacy init.d file which is called by systemd as a oneshot. This in turn calls the startPIA.sh and stopPIA.sh scripts as appropriate. So if the web server crashes, it stays stopped.

SAML SSO for Django

Mon, 15 Nov 2021 08:10:07 +0100

The University has a Single Sign On (SSO) system. There are a number of ways that it can be used. In this case I am investigating the use of Security Assertion Markup Language (SAML). There is also Shibboleth which is related, and our SSO can also use, but I will leave that till another time.

I am creating a test application running in django on my desktop. Django by default only listens on the loopback interface which means it can provide friendly information to developers safe in the knowledge that anyone who can view it is logged on to my desktop. Sites are identified using their URL, so I need to add a unique hostname. I edited /etc/hosts and added myhost.local as a hostname to the end of the line that starts 127.0.0.1. Now I can visit http://myhost.local:8000 in my web browser and get to my test website.

Parsing XML with Ansible

Wed, 20 Oct 2021 16:19:07 +0100

I am trying to gather some information about an environment once it has been created and save it in a small Django app. This is about my adventures trying to discover the Weblogic version from the it’s registry which is an XML file.

The XML registry is in the Oracle inventory, and starts like this:

<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?>
<registry home="/opt/oracle/psft/pt/bea" platform="226" sessions="7" xmlns:ns2="http://xmlns.oracle.com/cie/gdr/dei" xmlns:ns3="http://xmlns.oracle.com/cie/gdr/nfo" xmlns="http://xmlns.oracle.com/cie/gdr/rgy">
 <distributions>
 <distribution status="installed" name="WebLogic Server" version="12.2.1.4.0">

I’d like to extract the WebLogic Server version which is on line 4. As I don’t know XML it’s tempting to use grep and sed to find the information I want, but I notice there is an XML module in Ansible. It is community maintained, and it’s not stable, so it’s behaviour might change. This is valid from versions between 2.4 and 4, maybe later.

Kubernetes, Terraform and Secrets

Fri, 23 Jul 2021 17:28:30 +0100

Getting started with Kubernetes and Terraform

I’ve been looking into how to learn terraform. I have also discovered for my project I need to use Kubernetes. It turns out that it is really easy to create a kubernetes cluster on the local desktop to have a play with. Here goes:

I got started using the following tutorial I used Kubernetes in Docker (kind) to test. This turned out to be really easy to install. I already had Docker installed, so I didn’t need to worry about that. I downloaded kind from it’s website - there is a compiled executable which is easiest. The instructions say on Linux:

Troubleshooting Etherpad In Google Cloud

Fri, 21 May 2021 10:20:49 +0100

There is quite a lot that could go wrong with the work in the previous post, and I feel I experienced most of the issues!

We need to remember there are two projects in play here. One to create the container inside the infrastructure, and the other to create the infrastructure the conatiner lives in. A problem could be caused by either of these, and so the correct action needs to be taken to correct the issue. Most of the issues I encountered were caused by the infrastructure, so after I corrected each problem I ran:

Connecting To The Database

Fri, 30 Apr 2021 09:19:07 +0100

Now that we have a database it is time to connect Etherpad to it.

We see in the Hands on Guide to Google Cloud that the way to achieve this is by setting some environment variables to be read by Etherpad when it starts. Lets have a look how to do that.

Adding Environment Variables

Once again this is set up by the cloud run app module. which as I noted before is the basis of my webapp.

Creating The Database

Fri, 23 Apr 2021 09:19:07 +0100

The next step in the Hands on Guide to Google Cloud says we should connect the Etherpad instance to a database. Let’s see how we create the database using the automated processes we are developing.

Creating the database using Terraform

Cloud SQL module

A SQL instance was created by the terraform code that was generated from the boilerplate. University members can see the sql.tf template. It contains the following:

resource "random_id" "sql_instance_name" {
 byte_length = 4
 prefix = "sql-"
}

# We make use of the opinionated Cloud SQL module provided by Google at
# https://registry.terraform.io/modules/GoogleCloudPlatform/sql-db/.
#
# The double-"/" is required. No, I don't know why.
module "sql_instance" {
 source = "GoogleCloudPlatform/sql-db/google//modules/postgresql"
 version = "4.4.0"

 name = random_id.sql_instance_name.hex

 # ... Snip ...
}

The University has decided to standardise on Postgres for new database instances. The comment helpfully links to the documentation for the Google SQL module.

Creating Infrastructure With Terraform

Fri, 16 Apr 2021 11:45:06 +0000

This is an enormous and very complicated area. The hands on guide says to change the memory limit for the application to make it run faster. So, how do we do this in terraform?

In the project generated from cookiecutter, I can simply edit webapp.tf and change the webapp module to add memory_limit="1024M". Simple, but how do I know I can do that?

I need to read up on the Terraform language to understand what is going on. It seems there are two types of module. A directory of terraform code is called a Root Module, but within that code are module definitions. These are called Module blocks. So my webapp module defined with the module keyword is a module block.

SettingUpTerraform

Fri, 05 Mar 2021 10:38:34 +0000

Before starting to change the configuration with Terraform, there is some set up work that needs to be done.

While the getting started guides are fine, in practice this leaves a problem of how to work with colleagues, and how to manage secrets.

My colleagues have created a tool called Logan which they use to run terraform. It is installed using pip, but it is a docker container, so will require a working docker to run properly. I have RedHat 7 installed on my desktop, so I had to yum install python3. I found I had to upgrade pip. Since there are other requirements to install I created a virtual environment to install and run it in:

Deploying a Container to Google Cloud

Fri, 26 Feb 2021 10:09:23 +0000

Having created the container, we now need to deploy it to the Google cloud. We create a deploy project. This was created for me, I believe from our Boilerplate Google Cloud Deployment project This project can currently only be viewed from within the University. The gitlab-ci.yml contained the following at the time of writing:

# Use UIS deployment workflow, adding jobs based on the templates to deploy
# synchronisation images

include:
 - project: 'uis/devops/continuous-delivery/ci-templates'
 file: '/auto-devops/deploy.yml'
 ref: v1.2.0

 # Include template that lints local Terraform files
 - project: 'uis/devops/continuous-delivery/ci-templates'
 file: '/auto-devops/terraform-lint.yml'
 ref: v1.2.0

# Triggered by manually running the pipeline with DEPLOY_ENABLED="development"
# and WEBAPP_DOCKER_IMAGE set to the image to deploy
deploy_webapp_development:
 extends: .deploy_webapp_template
 environment:
 name: development/$DEPLOY_COMPONENT
 url: $WEBAPP_URL_DEVELOPMENT
 variables:
 DEPLOY_ENV: "DEVELOPMENT"
 rules:
 - if: $WEBAPP_DOCKER_IMAGE == null || $WEBAPP_DOCKER_IMAGE == ""
 when: never
 - if: $DEPLOY_ENABLED == "development"
 when: on_success
 - when: never
	
.deploy_webapp_template:
 extends: .cloud-run-deploy
 variables:
 # Informative name for image. This is used to name the image which we push
 # to GCP. It is *not* the name of the image we pull from the GitLab
 # container registry. The fully-qualified container name to *pull* should be
 # set via the WEBAPP_DOCKER_IMAGE variable.
 IMAGE_NAME: webapp

 # Prefix for service-specific variable names
 SERVICE_PREFIX: WEBAPP

 # Variables set by upstream deploy job
 RUN_SOURCE_IMAGE: "$WEBAPP_DOCKER_IMAGE"

 # The name of the deploy component - will be prefixed with the environment
 # name to create a gitlab deploy environment name
 DEPLOY_COMPONENT: webapp

What it does is to take the deploy template from the devops CI templates in the University GitLab (This is publicly viewable as I write this) and use them to deploy using some environment variables to control what is being deployed. We can see the in the deploy.yml the CI is instructed to do a docker pull on the container that we built, tag it with the Google container name, then push it to the Google cloud. There is another entry for staging, but first let’s understand development.

Building A Container with AutoDevops in GitLab

Fri, 19 Feb 2021 13:46:22 +0000

In the last post I set up a runner. Now lets see if we can use it to create a container.

I am looking into how my colleagues at Cambridge do things. They have a handy guide to the Google cloud using what colleagues call click ops. When I ran through it I was given a project which already had an ID, so had to make sure I used the project ID I was given rather than the one in the document. This also applied to the project registry.

Gitlab Runner for Auto Devops

Fri, 12 Feb 2021 12:28:00 +0000

The Auto DevOps pipeline in GitLab is supposed to be magic! It just reads your mind and does whatever you wanted! At least that seems to be what the sales blurb says!

Over the next few posts I will investigate how we use it. It is pretty clever, but we do need to set a couple of things first.

Requirements

This assumes docker is installed and working on the machine that will be used for the runner. I used my Linux desktop for this exercise which has Docker already set up.

Oracle 19c Critical Patching

Wed, 20 Jan 2021 15:20:49 +0100

It is the time of year to apply critical patches. This time round we have some databases at version 19c. We tend to install a new oracle home for each patch, as we find this helps us manage the migration. It also reduces downtime, especially if you have not yet taken advantage of pluggable databases as we haven’t.

Oracle Installer

I notice from the documentation that there is an applyRU parameter that can be passed to the installer to apply the patch. The documentation is quite poor though because it doesn’t specify what should be passed after that. Even more annoying is that if the patch attempt fails, it will leave the home in an intermediate state and it will have to be deleted and recreated.

Change Assistant on Linux

Fri, 23 Oct 2020 17:20:49 +0100

Automating Tools Patches

I have already automated the VM builds, so now I want to automate the application of the patch into the database. Oracle have done some work here with their Change Assistant tool, which once the database is set up can be used to apply the patch. Hopefully I can find a way to run change assistant from the Linux command line and apply the patch using the automated procedure.

Cloning a Pluggable Database using Unix commands

Thu, 15 Oct 2020 10:22:49 +0100

I read a lot about the flexibility of Oracle commands for pluggable databases. I haven’t seen as much about the old fashioned way to copy data files around and manually creating a control file. So lets see if that still works. I have a campus solutions demo instance, lets see if I can copy it and rename it.

Back Up and Edit the Control File

Running a familiar command is promising:

Java in the database

Fri, 02 Oct 2020 14:20:49 +0100

Why start using Java in the database?

I have had a couple of situations recently where using Java in the database might come in handy.

One is replacing a self-hosted database with one on the cloud. The provider gives access to the database, which means remote procedure calls don’t work - the provider doesn’t give access to the underlying OS, they provide the database as a service. This means the remote procedure call will have to be replaced in some way.

More Lessons About IO

Wed, 19 Aug 2020 15:20:49 +0100

I expect the question after the last I/O article is: How did we make our server go faster?

Things That Did Work

Looking at the AWR report, I could see the main problem was that async I/O was slow. We set

`1`	`filesystemio_options='SETALL'`

in the spfile, and this helped a lot. As I understand it, this enables asynchronous I/O, which means that rather than waiting for confirmation after writing, the database keeps on sending data to the SAN. This sped up the I/O by a factor of about 3.

Lessons About IO

Mon, 13 Jul 2020 15:20:49 +0100

How Does a Database Use IO?

We have a shiny new database server. We ran some tests on it, and found it was about the same speed as the old one. Worryingly the IO system seemed to be struggling to keep up with the load.

In case you don’t know, IO is short for Input/Output, and is used to describe the use of a hard disc, or similar storage. We all know that hard discs are slow compared to memory, but for a database to work well, the IO system needs to be able to keep up with the work the database is doing. Otherwise the database will eventually appear to lock up.

Split Tunnelling in Windows 10

Fri, 12 Jun 2020 15:20:49 +0100

We use a Virtual Private Network - a VPN to access all our work servers. But we use MS teams for meetings, and our VPN admin asks us not to to push loads of video over the VPN. So what to do? Disconnecting from the VPN every time a call came in got old pretty fast. And what do you do if you want to show a colleague something which required VPN access?

SNI and Integration Broker

Wed, 10 Jun 2020 17:06:10 +0100

A colleague had created a new service operation in peoplesoft integration broker, but it wasn’t working. After a lot of investigation, and a call to Oracle, we found the cause was SNI: Service Name Indication. This is a facility which allows several different hostnames to be on the same IP address, but still listen in HTTPS.

It seems the Integration Broker converts the hostname to an IP address before trying to contact the remote site. The problem is then that the remote site doesn’t know which website it wishes to talk to, and this causes a handshake failure. The messages in the logs are not helpful to say the least.

Silent Installation of Tools 8.58

Thu, 04 Jun 2020 10:00:00 +0000

Silent install

A new feature of 8.58 is the silent installation. The documentation says you can run a command such as:

./psft_dpk_setup.sh \
 --silent \
 --response_file=response.txt \
 --customization_file=psft_customizations.yaml

The customizations file is optional, but I decided this time around to use it as much as possible to create a system as close as possible to what I wanted to end up with. So I copied the file from <psft_base_dir>/dpk/puppet/production/data/psft_customizations.yaml and edited it to my satisfaction. <psft_base_dir> above is whatever was specified when the installer asked interactively, or in the response file. This means I had to do a test install before I could do a proper one, but that is fine.

Testing IO with SLOB

Tue, 19 May 2020 14:49:50 +0000

We recently purchased a new database server. We ran a performance test which our brilliant testing team have put together and found out the system with the new server was slightly slower than the system with the old one. The testing team was happy that the test was a pass - the users wouldn’t notice the difference. However, new servers are supposed to be quicker than old ones, so I was disappointed. I investigated a little further, and found that the AWR report showed there were more waits on IO on the new hardware than on the old.

Redo Inconsistency

Tue, 28 Apr 2020 11:03:50 +0000

Redo Investigation

I was investigating a performance issue on new hardware and installed SLOB to do so. I discovered in the AWR report, that the redo generated per transaction was around four times that in development.

I checked everything I could think of, including copying the parameter file from development to the new hardware and using that, but the redo stayed stubbornly high. I asked the oracle-l mailing list for ideas. Jonathan Lewis responded suggesting the high redo could be caused by the following:

Django on RedHat 7

Fri, 06 Mar 2020 00:03:50 +0000

Django on RHEL7

Here is how I chose to set up Django on RHEL7. This is somewhat painful because the default Python in RHEL7 is still version 2. Using this to create any new programs seems foolish as it has already been de-supported by the Python project, so I really have to use Python 3.

Set up Linux

We need Python 3, Django, Apache, and Postgresql. See the instructions linked in the next section if the Postgresql packages aren’t found. The version that comes with RHEL7 is too old for the version of Django we want to install.

Gnu PG Cache Time To Live

Thu, 27 Feb 2020 10:03:50 +0000

As discussed previously, we use regpg to manage Ansible secrets. This has been really useful. One annoyance though is that some tasks can take up to 6 hours to run, but the gpg agent only caches the gpg passphrase for 10 minutes or so. I end up having to type the passphrase in several times during a run. I occasionally kick off a run before I leave for the day. It would be a shame if it was stalled overnight due to waiting for a passphrase.

Ansible trick - Retry an intermittent error

Wed, 26 Feb 2020 10:03:50 +0000

We have recently been having issues with mounting windows shares. It occasionally doesn’t work. We don’t have access to fix it. The playbook fails at this step. This is really annoying, because if it tried again it would probably work fine, and the playbook could complete. It turns out that Ansible does have a way to retry. I got the idea from this Stack Overflow question

The solution is the Ansible retries keyword. Here is a test of a command that intermittently fails:

Auditing Options in the Database

Tue, 04 Feb 2020 11:35:00 +0000

Oracle Auditing

Oracle have a number of different types of auditing, and in recently created databases they all coexist. I looked at this recently and thought I had better make some notes before I forget.

There are three types of auditing:

Traditional auditing. This is the way things worked before 12.1
Unified auditing. This is the new rewrite of auditing, but needs some effort to get working.
Fine grained auditing. This stays the same between traditional and unified auditing.
Mixed mode auditing. In the Database Security Guide it mentions that newly created databases can use mixed mode auditing. This allows us to use the functionality of both types of audit.

Auditing is one tool that can be used to help secure the system. A particular risk is that of an attacker updating or removing the audit trail. So auditing should be considered as one tool to keep the system safe. Unfortunately it is one of those things that doesn’t look shiny or help users to do their work, so doesn’t tend to get enough resources to make it work properly.

Fixing Logical Standby Go Slow

Mon, 06 Jan 2020 14:03:50 +0000

Logical Standby Databases

We use a logical standby database. The logical standby database is inherently fragile, because it mines the primary logs, and attempts to rebuild the SQL to apply on the logical.

This rebuilt SQL is fine most of the time, but if an application release has altered a table, or updated most of the rows in a large table, this generated SQL often performs very poorly.

Logical Standby Failure Modes

Typically there are two failure modes for the logical standby database. Either it will fall over because a SQL statement it has generate doesn’t work, or the generated SQL statement will perform really badly and take forever. The first is easy to deal with. Find the error message, fix the error, or ignore the SQL and continue. The second is more difficult, as there is no error message to investigate. Typically we will notice that the logical standby is getting a backlog of redo to apply. We need to investigate what is taking a long time.

SQL Performance Issues

Mon, 23 Dec 2019 11:03:50 +0000

The initial investigation of this issue is written in the Production Emergency post. You might want to read that first if you haven’t already.

Addressing the underlying problem

We had identified the source of the problem. The SQL text can be extracted from the database as follows:

set linesize 300
set pagesize 0
set long 30000
set longchunksize 300

select sql_text from v$sql where sql_id = '67bqun92ngrsj';

This displays the SQL text. If the application populates the module, program and client_id of v$session using dbms_application_info, then there is enough information to find out what code is causing the problem and even who was running the program that caused it.

What to do when production locks up

Fri, 20 Dec 2019 14:03:50 +0000

Recently our PeopleSoft system locked up. Nobody could do anything, they just got a blank page in the browser.

The System Model

The approach to use in this situation is to consider how the application works. In our case a user’s web browser will connect to the load balancer, which will connect to a web server. The web server will pass the query to an available application server out of the pool, which will then pass the query to the database. Then the results go back up the chain.

Triggering an Action in Another Repository

Tue, 10 Dec 2019 09:34:09 +0000

GitLab Organization

It seemed reasonable when specifying the repositories, to have three (at first):

One to hold the Ansible roles to build the VMs
One to hold the custom code that the above repository will deploy
One to define what the environments look like, things like
- Memory
- Number of VMs at each tier
- The names of the VMs
- Passwords
- And so on.

Problem

The problem is that when the developer commits code to his repository, they would like it to be deployed to an environment so they can test it. It makes sense to do this automatically, we have all the information about the environment in the environment repository. This is different to the repository where the code is being checked in. It would be nice if we could call across to that and trigger the automated deploy.

Copying files using Ansible

Mon, 02 Dec 2019 09:34:09 +0000

Introduction

Copying large numbers of files around - you would have thought this would be easy using Ansible. Most of what we do with Ansible is copying files and editing them. It turns out to be rather difficult to get right.

Copy module

The copy module seems like the correct thing to use at first glance. It copies files from the build host to remote locations. So I can clone the repository of files to deploy and send them off to where they need to go? Brilliant! There is a note at the bottom

ORA-13831: SQL profile or patch name specified is invalid

Mon, 25 Nov 2019 10:06:09 +0000

We had a process error with the above error message. I believe this is related to Oracle bug 27496360 which is a duplicate of 29942554, which is apparently in QA, but is targeted for database 20.1. It being 2019, Oracle haven’t released Oracle 20 yet, so we have no fix. We note it seems to be triggered by applying critical patches. We have an SR open for this which is attached to the bug.

PeopleCode Debugger - Setup and Diagnosis

Fri, 22 Nov 2019 10:06:09 +0000

The debugger requires application designer to be run in three tier mode, i.e. it should connect to the application server workstation listener rather than to the database itself.

We found that to achieve this we had to do the following:

On the application server, run psadmin, and administer the domain where the debugger is to be switched on. As of tools 8.57 this is done by choosing the following options from the menu. Note that the domain will be shut down once the configure option is chosen.

Gitlab Inventories, Private Keys and Secrets

Mon, 18 Nov 2019 10:03:50 +0000

Secrets and the Problem with Build Servers

I am not totally sure I understand this, so let’s see if I do when I write it down.

I decided that gitlab needs to stop accessing hosts using keys from the gitlab runner, because this means:

Everything on gitlab runner can access every VM
If the gitlab runner moves, nothing can access any VM

So 1 is too open, and 2 just doesn’t work. While I could configure a gitlab runner per environment, it also strikes me that the secrets are in the ansible build directory and shouldn’t be. We need a better approach.

What is automation?

Fri, 15 Nov 2019 11:23:50 +0000

What is automation

A colleague asked me if I could give them an overview of automation. It is something we do sometimes without thinking too much about it.

Think about Lean Enterprise. It is all about making things more efficient.

A concrete example

Technical staff like me can apply this principle to what we do daily. Here is a simple example of what might have to be done to deploy an imaginary application.

Oracle Sales: Cloud Analytics, Blockchain, IoT, AI - HEUG EMEA 2019

Thu, 14 Nov 2019 14:33:50 +0000

Oracle puts effort into selling at HEUG, which makes sense because we are all users of Oracle software. It is interesting to know what is out there, but the likelihood is we will never get to use most of it. Also, Oracle pushes cloud which I find quite depressing because I don’t know where I fit in to the new cloud paradigm.

I made notes from a couple of these sessions, which are quite short, so I will combine them here.

Journey to the Cloud - HEUG EMEA 2019

Fri, 08 Nov 2019 09:53:50 +0000

Is the cloud my friend?

I am concerned about the cloud. As a DBA I believe the idea is to replace most of what I do with a cloud provider. So if I want to keep working, should I work for a cloud provider, or change what I do? In the mean time I am also trying to ensure I can deliver a better service than the cloud providers using similar technologies in a way that is more responsive to the needs of my customers. So it is interesting to learn the experiences of Jo Ellen Dinucci, Associate VP, Finance and Administration - Boise State University during their journey to the cloud.

Celebrating 10 years of failure - HEUG EMEA 2019

Thu, 07 Nov 2019 09:44:01 +0000

Session Celebrating 10 years of failure, because you won’t succeed from the start.

Sarah Kobus, University of Amsterdam

I thought this looked interesting, not least because I have recently realised that getting everything perfect all the time is not a realistic or desirable goal. As if to demonstrate the truth of what she was saying, her co-presenter was ill, so Sarah had to present the session alone!

How good are we at learning from failures?

Everyone encounters failure. As an illustration Sarah spoke about replacing their Student Information System. The core issue encountered was not investing enough in changing the business processes to work well with the new system.

HEUG 2019

Fri, 01 Nov 2019 09:44:01 +0000

I went to HEUG EMEA 2019 last week at the University of Amsterdam. I enjoyed it. It was good to meet some other people who work in education and use Oracle technologies. I haven’t been to HEUG before. The HEUG was smaller and I feel more friendly than the UKOUG. There were fewer exhibitors, and they seemed less desperate to sell, and happier to just have a chat about the conference and travel, which actually makes them more approachable, but I am not sure they sold any more!

Missing File

Fri, 18 Oct 2019 11:09:47 +0000

This is a post which was sitting in my drafts since the start of last year, but it still seems useful to me.

It’s been a while since I have had a file that was deleted. What course to take depends on context - what do you want to achieve? In this case I wanted to remove the tablespace. I offlined all the files in the tablespace and deleted it. It is pretty easy really. The other thing I could have done is recovered the datafiles from the redo logs. Maybe I should try that another time.

SQL Tuning

Fri, 11 Oct 2019 12:00:11 +0100

SQL Tuning

As a DBA I find that people think I know about how to tune SQL. They present me with a query that looks simple, but on inspection has views on views on complex views, and has an explain plan of over 100 lines! It is difficult to know where to start, particularly when you don’t know what the SQL is supposed to be doing, or understand the structure of the data within the database.

Ansible Tricks

Fri, 04 Oct 2019 11:11:36 +0100

Some Useful Ansible tricks

Including a playbook

I am working on extending my automation to do some new things. We refresh test environments from production. Previously we used to copy all the code from production back to development. Now we build the VMs from scratch, so we don’t need to do this any more, but it would be convenient to call the VM build in the middle of the refresh.

Hiding Passwords In Ansible

Fri, 27 Sep 2019 15:00:36 +0100

Often configuring a system involves running scripts and passing passwords as a parameter. If all goes well, this is fine, Ansible just reports the task has changed. If not, the password is logged, which is bad if the output is being sent to people who shouldn’t know it. Here are some thoughts on how to get round that.

Setup

Start with the following playbook for demonstration purposes:

---
 - hosts: 127.0.0.1
 connection: local
 tasks:
 - name: This works
 command: "echo This is my secret password"

 - name: This fails
 shell: "echo This is my secret password && false"

And run it:

Moving

Tue, 17 Sep 2019 16:24:21 +0100

Moving

My hosting at the University of Cambridge: people.ds.cam.ac.uk/psh35 is being discontinued. I am therefore moving this blog to Netlify:

Please update your bookmarks or feed reader to: https://curiousdba.netlify.app/

Netlify

This wasn’t as hard as it might have been. I took the opportunity to upgrade the theme, because GitHub complained about the security of some of the components. This caused the site not to build. The new version of the theme requires Hugo pipeline, which means it needs a recent version of Hugo extended. This won’t run (easily) on RHEL7 due to glibcxx being too old. However, Netlify does supply Hugo extended, and allows the user to pick a version. I needed to upgrade because of a conflict with the theme and the default Hugo version. I picked the latest and my blog built!

Git On Windows

Mon, 01 Jul 2019 12:54:48 +0100

Here is a possible way to use git on Windows to work on a git repository in GitLab. The nice thing about GitLab is it uses git, so any Windows git client can be used. I prefer command line myself, but there are GUI options including the git supplied one which I will be using.

Installing software

I suggest using chocolatey to manage software. Follow the installation instructions on the chocolatey website to do this.

Enroling A New Admin

Thu, 13 Jun 2019 10:18:20 +0100

The Idea

Sometimes you see a private shared lane which has a gate to stop people using it, but the people who are allowed have padlocks in a chain. Anyone who has a padlock in the chain can open it with the key in their keyring.

Image from pixabay.

This is very much like how the secure keys work. If someone else wants to be able to use the gate, they have to get one of the three key holders to go to the gate with them, open their padlock, and insert their padlock into the chain.

Secrets, Ansible and Regpg

Fri, 07 Jun 2019 10:13:36 +0100

I like Ansible, but I find one omission in the way it works is the lack of a way to manage secrets, i.e. things like private keys, passwords, and access tokens.

I stored passwords in the inventory file. This means the inventory file is large, and can’t be checked into version control, which makes it difficult to manage.

My first test is to create another git repository to check out onto the VM. This contains some application code which needs to be installed. To check this out to any one of 100 or so VMs I am using gitlabs deploy token functionality, which creates a URL like this:

Installing Open SSH on Windows (Automatically)

Tue, 04 Jun 2019 10:44:34 +0100

Our PeopleSoft system has a couple of maintenance tasks which are kicked off from the database server. I am converting it to use Ansible and a management server, but in the meantime I need this to work.

We had been using Bitvise SSH server on Windows, but experienced problems with it locking up occasionally. Also we needed to create some new Windows VMs and wondered if there was a way to do the work without paying for more licenses. Also as we are upgrading to Windows server 2016, I am seeing if there is a way to automate this as part of my Ansible build.

Patching Java for PeopleSoft

Tue, 28 May 2019 13:44:07 +0100

Security Updates Not Included!

Oracle’s Deployment Package DPK is supposed to deliver all the software required and at the correct versions. However, it appears this isn’t true. PeopleSoft itself is at the current version in the DPK, but all the software it depends on only has the previous quarters security updates applied. This often leaves us vulnerable to some quite serious security flaws.

The solution is to apply the patches after applying the DPK. It is annoying that we have to do this, because the sales documentation at the time suggested everything would be at the correct version with the latest patches.

Creating Test Data

Mon, 13 May 2019 10:45:54 +0100

Here are some notes as to how to create test data from real data with SQL.

I created a package to do this. It generates SQL statements to mask columns which have personal data, so we don’t end up testing with real peoples data. Here are some things I made it do.

The procedure loops through a list of tables which is in a local table. In this case we are selecting across a database link. I use a SQL statement to discover the format of the remote table using dbms_sql.parse.

GitLab Runner

Wed, 08 May 2019 11:52:56 +0100

The Problem

Our release process means that we have to refresh test environments on a particular schedule around the date of the release. These dates were put into a confluence page, but we discovered that the problem with this is that if a release is moved we have to change all the refresh dates. Also sometimes projects mean particular environments aren’t refreshed or the schedule is altered slightly.

So it ends up that we have to:

Scheduling Tasks in Task Juggler

Fri, 12 Apr 2019 12:10:22 +0100

Last time I showed how I set up a project in TaskJuggler. Now it is time to add in the tasks.

Thinking about what I want to achieve, the project goes something like this:

The Project

Update the operating system on the database server

Update the development database server during the patching window.
Run a test to make sure it works. This includes

People using it during the day.
An automated test running overnight.

Assuming it all works, update the test database server during the patching window.
Run the same tests as above
Update the UAT database server during the patching window
Run the tests
Update production during the patching window.

Patch the databases

This looks very similar to above

Manging Projects

Fri, 05 Apr 2019 15:54:02 +0100

One thing I have found terribly dull is project management. This is something we all have to do at times. My example is the quarterly patching from Oracle. We also take the opportunity to do operating system updates at the same time.

Thinking about how to do the work, we have a patch window where systems are allowed to be shut down (Given the appropriate change control of course) for maintenance to be done. These are twice a week. We have a numer of systems, dev, test, UAT and production like everyone else, and the patches are moved through so any issues are discovered before we hit production.

Gitlab and Sql Developer

Tue, 12 Mar 2019 17:06:31 +0000

We recently got access to Gitlab. It is rather more user friendly than our previous Gitolite install, and has lots more features.

One thing I thought might be useful is to connect SQL developer to it and use the IDE features.

The first thing I discovered is that running SQL Developer off a Windows share is a painful business. Also it is likely to be out of date. Installing it locally seems like a good move.

Automated Testing Users

Mon, 05 Nov 2018 16:42:08 +0000

Imagine a situation where a production database is copied to test, and the data scrambled. Our testers want to access the database because they need to find some data to use for testing.

The testing is automated.
The refresh is automated
The scrambling is automated.

How do we get the testers the password in a secure fashion? Also, how do we ensure testers don’t just use the test user normally? They should log in with their own user name and passwords. Also, the test user shouldn’t have the same password all the time, it should change periodically. This will ensure that if a tester has found out the password because they need it to develop the automated tests, they won’t know it after they leave, or be tempted to use it when they ought to be using their own user name.

Reducing Downtime With Tmadmin

Fri, 21 Sep 2018 14:23:00 +0100

Reducing Downtime

I want to reduce down time. Is it possible to clear a cache, or to restart a server without downtime? Tmadmin is provided by Oracle to perform low level tasks and includes this type of functionality.

Lets have a look.

Purging the cache

This can’t be done if cache sharing is enabled, but if it isn’t, it clears the cache for each process one at a time without downtime.

Un-Delete Open file

Fri, 07 Sep 2018 11:56:26 +0100

There is a way to un-delete a file if a program has it open in Linux.

How directory entries work in Unix

When you remove a file on Linux, what actually happens is that the file is unlinked. Lets have a look. I start with a clean directory:

$ ls -la
total 4
drwxrwxr-x 2 me me 6 Sep 4 15:58 .
drwxrwxr-x 14 me me 4096 Sep 4 15:28 ..

The directory has two entries, the current directory and the parent directory. The second entry, the number (2 and 14 above), is the number of links to the file. The current directory has 2 links, one for it’s entry in itself ‘.’ and one for its entry in the parent directory. If we create a file, it has one link:

Delete Online Logs

Wed, 05 Sep 2018 14:56:26 +0100

In the Oracle training courses they always say:

Your junior DBA deleted the online redo logs.

Of course, it is never your junior DBA that does that. They are standing behind you laughing.

Anyway, what to do if your junior DBA deletes the online redo logs. Or even if someone else does it, is first of all, not to panic. According to Ask TOM the worst thing to do is a shut down abort because if you do that, the database will need to be recovered from the online redo logs, and they have been removed.

More Windows Woes

Fri, 17 Aug 2018 11:50:12 +0100

Introduction

Having tested my Windows Ansible build all seemed to be going well. A colleague took the script and ran it, and all seemed to be well, except that Microsoft word didn’t work - It just hung.

The Problem

On the Windows process scheduler Microsoft Office can be invoked to run a macro to populate a template to form a Word Document.

The problem was that despite the Ansible install having run, and supposedly being repeatable, it didn’t work. The process never ended. Looking at task manager we can see that Microsoft Word is running, but it never ended.

Vagrant Multi VM provisioning

Fri, 10 Aug 2018 15:02:45 +0100

Multiple VMs required

I am looking into Postgresql, replication and fail over. One way to set this up is to have a primary, a standby and a witness server. Two run databases, and one - the witness to decide whether a fail over is needed. To test this it would be nice to create three VMs on my desktop to help test ansible scripts. I have downloaded Virtualbox from Oracle, and my desktop has loads of memory. It would be nice to be able to script the creation of the VMs and then be able to run the same ansible playbook that has been developed for the “real” VMs. Then I can learn postgres and help develop the playbook and associated scripts.

How We Use Ansible

Tue, 31 Jul 2018 10:53:39 +0100

Introduction

I realise I have written a few posts on Ansible, but no overview on how we manage it. I haven’t done any training on Ansible, and this is my first effort at automation (Apart from some bash and python scripts) so there is probably much that could be improved.

Our Systems

We have about 15 - 25 environments to manage, depending on how you count them. Around 2-5 of these are vanilla in one form or another, so don’t need to be managed by Ansible. The rest are all production like to some extent, so need to be managed in a similar manner.

Ansible Strings And Numbers

Tue, 10 Jul 2018 11:06:14 +0100

I had a playbook fail because of the error dict object has no element

It took me ages to debug it, but the problem is that I had defined the index as a string, but was trying to match it to a number. To fix it, I only had to quote the version in the inventory, and everything worked.

I was only caught by this because I was using an old inventory file to test with. I think it used to automatically cast numbers to string, but that stopped a while ago.

Managing Windows with Ansible

Tue, 26 Jun 2018 18:05:23 +0100

After a lot of trial and error, I finally found a way to make it work. Here is what I did.

As noted before, rather than trying to run an installer from a share, I simply copied it on to the local disc of the VM.

Before running Ansible on windows, the operating system has to be configured using (for example) the ConfigureRemotingForAnsible.ps1 script. I took the defaults. I didn’t bother setting up any of the advanced options like CredSSP, as in practice there didn’t seem to be any benefit.

Getting Windows and Ansible to play nicely

Tue, 26 Jun 2018 15:09:57 +0100

The challenge is to run an installer from a share, then run some extra configuration. We do this in Unix (Well, GNU/Linux) all the time, and it is really, really easy. We even have the software on a Windows share, so we just need to mount that and run it!

It turns out that Windows is different from Unix. A mounted share doesn’t belong to the system like in Linux, it belongs to a session. A windows session is what is created when a user logs in to Windows, it contains the desktop, the windows, and the various attributes of the connected user, their permissions for example.

Weird PeopleTools Install Issue On Windows

Wed, 23 May 2018 15:34:25 +0100

I am not sure how this situation arises, but I discovered that sometimes the windows installer (Puppet based) on Windows gets into a state where it isn’t installed, but thinks it is, which means that the install won’t work, but it generates messages as if it had worked.

Under c:\psft\pt\ps_home8.55.23 is only the jre directory. Running the install again appears to work, but doesn’t do anything.

The solution is to run the uninstall, then the install will work.

Powershell Parameters

Tue, 03 Apr 2018 16:17:46 +0100

I got an error from powershell:

PS C:\> Start-Process -NoNewWindow -Verb RunAs
Start-Process : Parameter set cannot be resolved using the specified named
parameters.
At line:1 char:1
+ start-process -NoNewWindow -verb runas
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : InvalidArgument: (:) [Start-Process], ParameterB
 indingException
 + FullyQualifiedErrorId : AmbiguousParameterSet,Microsoft.PowerShell.Comma
 nds.StartProcessCommand

This is because the Start-Process command has two parameter sets. Looking at the page linked, the parameters are listed in two groups. These groups can’t be mixed and matched. You have to pick one or the other. So if you want to do runas, you can’t use NoNewWindow.

Redundancy

Wed, 28 Mar 2018 17:25:35 +0100

I have been thinking about our DR process, and how to improve it. More on that in a later post. However we recently had a couple of planned server room outages, where my previous planning has been beneficial.

Production is redundant and resilient across our server rooms. However, the other environments are not because redundancy is expensive and only really needed for production systems.

So what we do is to have half of the non-production systems at one site. and the other half at the other. This means that we can shut a machine room down, and half the development environments will continue to run.

Rman Fail

Fri, 09 Feb 2018 15:37:15 +0000

Here is another issue we had with RMAN. This one has been bugging me for years.

We were doing a duplicate using rman. For some reason the recovery catalogue didn’t contain the archive log we needed to do the recovery, so the restore completed, and finished with the familiar error:

...
executing command: SET until clause

Starting recover at 2018-02-02 16:03:13

starting media recovery

unable to find archived log
archived log thread=1 sequence=307
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: 'system.dbf'

The normal thing to do here is to correct the error, and redo the backup. But we already had the redo logs on the disc, and wanted to be able to apply them.

More Crash Consistent Recovery

Tue, 23 Jan 2018 10:45:57 +0000

Different Scenarios

In my previous post on this topic, I noted that you could use the snapshot time on the recover database command to recover the database from a SAN snapshot.

I realise that there are different possible scenarios and my write up wasn’t clear on which approach is applicable when. Also, the test I did was unrealistic as I used the logs after the snapshot was taken, and the whole point of using SAN snapshots is that they contain everything required for a crash consistent recovery.

Recovery Manager Problems

Fri, 19 Jan 2018 11:41:13 +0000

Lots of people seem to like Oracles Recovery manager. I am not one of them. I think this is because of a lack of understanding on my part of how it works. It is a complex beast, and at the same time has some annoying limitations.

I like to automate things. I have a number of scripts to call RMAN to do backups and restores in common situations. These fail far too often for my liking. I feel I should look into why. Maybe I can learn to love RMAN? We shall see.

High Water Mark

Tue, 09 Jan 2018 15:23:14 +0000

We are running a data conversion and got a wait event I don’t normally see: The brown is identified as Configuration. It I drill down, I can see more detail. Here light purple is HW Contention (i.e. High Watermark Contention). Darker purple is Write Complete waits, and yellow is buffer busy waits.

We have deferred segment creation switched on for the database. This means that the segment needs to be created before data can be written. The high watermark can only be moved by one process at a time.

Exporting Statistics

Wed, 20 Dec 2017 11:28:20 +0000

This was surprisingly more difficult than I expected. We know that we can export stats from the dictionary to a table, and from the table to a file, and that file can be copied and imported to another database for the stats to be imported. Easy right?

$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Tue Dec 19 08:55:39 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

SQL> exec DBMS_STATS.CREATE_STAT_TABLE('SCHEMA','MYSTATS','USERS');

PL/SQL procedure successfully completed.

SQL> @export_stats
BEGIN
DBMS_STATS.EXPORT_TABLE_STATS ( 'SCHEMA', 'MYTABLE', NULL, stattab => 'MYSTATS');
END;

*
ERROR at line 1:
ORA-20002: Version of statistics table "SCHEMA"."MYSTATS" is too old. Please
try upgrading it with dbms_stats.upgrade_stat_table
ORA-06512: at "SYS.DBMS_STATS", line 18000
ORA-06512: at line 1

Hang on, I just created it! How can it be too old? Still, I will upgrade it if it insists.

Parsing

Tue, 19 Dec 2017 10:54:05 +0000

We are running a data conversion. The powers that be decided to use APIs to convert the data as they contain error checking. The problem is that they are generally designed for interactive use updating one row at a time, so they are very slow to update large batches of data.

This was tuned and is getting much faster, however we noticed that there are a lot of waits on cursor: pin: S wait on X. From experience I know that this is caused by excessive hard parsing. However, I couldn’t find any statements with a lot of hard parses. The most was about 40, so nothing like the thousands which were a cause of this problem early in the release cycle of 12c, where cursors weren’t being shared properly (Bug 20476175 fixed by the patch of the same number).

Crash Consistent Recovery

Mon, 11 Dec 2017 16:17:41 +0000

Problem

Since Oracle 12c, you can recover a crash consistent snapshot. Oracle support note 604683.1 says how to do this.

We had an issue where the recovery wanted to effectively run to the end of time, and wouldn’t ever finish. No matter how many logs were applied, it said:

1
2
3

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/CS_SR/system/system01.dbf'

Solution

The solution is the snapshot time of the recovery clause. To demonstrate I have a database being recovered from a snapshot, and here are some archived redo logs from the source database.

Tech 17 - Oracle 18c The Next Update

Thu, 07 Dec 2017 17:40:54 +0000

Dominic Giles drew our attention to the safe harbour statement. Anything can change between now and when the new version is released.

There has been some confusion about the announcements regarding autonomous database. This is a service on top of oracle 18c. Oracle 18c is a product.

Release Schedule

Oracle will change its release process to release annually. Hopefully this will lead to more stable releases, as there won’t be a rush to get the new features into the current version of the database. Changes will be incremental, but will happen every year. There will be quarterly release updates. These will go back to using a number rather than a date format, These will include all fixes, not just security fixes, but performance changes will be switched off by default.

Tech 17 - Oracle Database 12c Release 2 - What is new in the Oracle Optimiser

Thu, 07 Dec 2017 14:55:12 +0000

The first session after lunch was Oracle Database 12c Release 2 - What is new in the Oracle Optimiser. It was taken by Nigel Baylis who is the product manager for the oracle optimiser.

I was impressed by Nigel’s honesty when he talked about what worked, what didn’t and how things are being changed to correct the mistakes of the past. This is the main reason to come to user groups of course. It made me feel like Oracle are willing to accept where things haven’t worked out as they hoped, and take action to correct it.

Tech 17 - New Indexing Features

Thu, 07 Dec 2017 14:52:38 +0000

For the last session before lunch I listened to Richard Foote talking about new index features in 12.2. Richard is well known as the Oracle Indexing expert, and so I was eager to hear what he had to say.

Object name length

Indexes can now have 128 character names, where before they could only be 30 characters. I expect this will make life easier where naming standards dictate long names for whatever reason.

Tech17 - Auditing the Oracle Database

Thu, 07 Dec 2017 14:45:16 +0000

The presenter was Pete Finnigan.

Pete talked through a solution (PFCATK) he had developed for auditing the oracle database. This appears to be currently in development and not available for general use. He seemed inclined to release the core tool and sell a pretty front end or consultancy, but it seemed he hadn’t yet decided. The lack of anything to play with (Unless you ask for a copy of the tool and agree to install it and give feedback) made this session less interesting than it might otherwise be.

Tech 17 -The Answer to the Ultimate Question of SQL, Performance Tuning and Everything

Thu, 07 Dec 2017 14:41:38 +0000

This is second session I attended in Tech 17. It was presented by Martin Bach and David Kurtz. The answer is eDB360. The session was in two parts with a presentation by Martin, then a demo with David.

Martin discussed the problems of doing a database health check, particularly for a third party company who may not actually have access to the database. It needs a standard approach, which is consistent across databases, and repeatable. Ideally performance data in AWR should be persisted for just over a year so that the performance of annual business cycles can be compared e.g., year end for financial systems.

Tech17 - Cost Based Optimizer - The Panel session

Thu, 07 Dec 2017 14:31:18 +0000

On Wednesday 5th December, I attended Tech 17. While it is a pain to organise getting there and back, I found in previous years that it was useful for me to go and find out what is happening in the industry.

Cost Based Optimisation - The Panel Session

Panelists:

Slow Start

This took questions from the internet and the audience, so it is a bit variable. The question has to match something the panel can talk about. The first questions about the Autonomous database was disappointing in this regard as very little information has been released about it, and the Oracle employees had to stick to the company line, so couldn’t speculate.

Broken PeopleTools Deployment Packages

Wed, 08 Nov 2017 12:13:40 +0000

The October Critical patch saw Oracle release

Patch 26743107: PT 8.55.19 PRODUCT PATCH LINUX DPK.

This is important to install because there was a serious security vulnerability in the performance monitor component which was easily exploitable over the network without authorisation.

However, once the DPK was installed, looking at the Weblogic home, we can see that the October critical patch

Patch 26519417: WLS PATCH SET UPDATE 12.1.3.0.171017

hasn’t been applied. It would be nice if it was delivered fully patched, but presumably Oracle’s internal time scales don’t allow this, so I will have to patch it myself.

SessionHistory

Mon, 18 Sep 2017 10:14:21 +0100

Here is another note to myself about how to look at what sessions were doing over time.

The situation was that we saw a lot of blocking sessions during a performance test. The developer wants information about the blocking sessions. To do this we want historical information from v$session, which helpfully tells us the blocking session, and also the SQL being run. This can be sampled every so often using a simple script, but Oracle have already thought about this and created v$active_session_history which you can use if you have licensed the Diagnostics pack.

PeoplesoftPerformance

Wed, 16 Aug 2017 17:37:19 +0100

PeopleSoft Performance

Occasionally I have to diagnose a performance issue in PeopleSoft. This is a reminder to myself of the tools available and what I can check.

PeopleSoft Ping

This is a really useful tool because it gives real performance indications as to how the system as a whole is performing from the desktop to the database. The user can see the time taken at the browser, web server, application server and the database server for a simple query. When a slow response time is seen it is important to drill down to see what cause it. It could just be Windows downloading updates for example. PeopleSoft ping data is stored in the database used by the application in the table sysadm.ps_ptp_tst_cases. I used the following columns (See Dave Kurtz’ website for the rest).

ApacheUcamWebauth

Fri, 11 Aug 2017 17:18:34 +0100

Installing Apache and Ucam Webauth on Centos/RHEL7

Apache is pretty easy:

 yum install httpd

Done.

Ucam Webauth

It is hosted on github, but won’t be much use to anyone who doesn’t have a need to authenticate users with the University of Cambridge’s single sign on which is called raven.

I downloaded the SRPM from the raven page as suggested, and compiled it. The following packages are prerequisites. I used the follwing ansible task:

DBLinks

Fri, 28 Jul 2017 10:01:33 +0100

Database Links

Our PeopleSoft system suddenly started showing spikes in network usage. Users were complaining of slow performance. Enterprise manager shows big spikes in network waits. Strangely these don’t seem to be reflected in the SQL. The table on the left doesn’t have as much beige/grey as the one on the right.

Lets click on network and see what the waits really are. SQL*Net message from dblink.

Criticalpatch

Fri, 21 Jul 2017 11:02:09 +0100

Oracle Critical Patches

It was critical patch Tuesday this week. Because of the time differences between here and the USA we actually only get to find out about the critical patches on Wednesday, though there is a pre release announcement.

With software as complex as Oracles you can pretty much guarantee that there will be serious bugs somewhere in there which need patching every quarter.

Considering the types of organisations that run Oracle - the ones who can afford to pay the license fees, i.e. rich and powerful ones, you can be sure that someone is looking into what changed for each release, and how they can benefit from stealing or changing information.

Jenkins On Centos

Fri, 14 Jul 2017 16:40:55 +0100

Install Jenkins. Automatically?

The documentation for Jenkins is pretty sparse, but fortunately a colleague had set up Jenkins already, and was able to give me some pointers.

The idea is to use the Apache web server as a reverse proxy to access Jenkins. This can also take care of user authentication.

Prerequisites

So first of all download Java from Oracle. I download it manually and keep it in case the link changes. I use the RPM and install it with yum. Note that there isn’t an easy way to automate this. There is no repository for Java which means you have to check every quarter, when Oracle release their Critical Patches then download and update it.

Automation

Tue, 11 Jul 2017 16:33:32 +0100

Automating The Build of New Environments.

The Problem

I look after a Peoplesoft Campus Solutions system, which we call CamSIS. This is the student records system for the University of Cambridge. It is a pretty important system, as it is core to the mission of the University. This means that there is always some kind of work going on to improve it, upgrade it, or add new functionality.

Having a lot of changes, and work going on means there are a lot of test, development and training environments. Normally there are about 20, during large projects there can be even more than this.