Oracle Private Database Cloud: Understanding the Resource Model

Tags

dbaas, em12c, emcli

In the public cloud parlance, consumers never provision to a specific server, instead they provision to a pool of infrastructure within a geographical region or data center. You would find this pattern in AWS, where you select Availability Zones or in Oracle Cloud, where you select the desired Data Center.

EM12c’s private database cloud follows a similar paradigm. It offers a two tier hierarchy in PaaS Infrastructure Zones, and Software Pools.

PaaS Infrastructure Zone (or Zone)

Zone is a logical grouping of cloud infrastructure resources (like servers, network, storage, etc) based on QOS, functional, departmental or geographic boundaries. For example, Finance Zone, East Coast Zone, etc. Cloud users or consumers provision into a Zone. A Zone is also used to enforce access control and chargeback/showback.

Database Software Pool (or Pool)

A  group of homogeneous clustered or non-clustered database resources exhibiting common characteristics. For example,
– Pool of 11.2 Database Oracle Homes (for dedicated databases)
– Pool of 12c Container Databases (for PDBs)

Okay now that we have covered some theory, lets take an example and walk through it. Our goal to offer a database service that is highly available and redundant across multiple data centers. This image below captures the situation well.

So lets get modeling.

A. Modeling Zones

1. We have two data centers – say one on the East Coast and the other on the West Coast. With this information, we could model two Zones based on the location dimension – East Coast Zone and West Coast Zone. As easy as this sounds, i don’t think putting all your hardware resources in a single grouping makes much sense.

2. Its likely that we have servers from different vendors, with different architectures, etc. Assume we own a few Exadata, some commodity servers, some SPARC servers, some VMs, etc. To accommodate this, we can update our model with the hardware dimension – East Coast-Exadata Zone, East Coast-Commodity Zone, East Coast-Sparc Zone, etc.

3. Now typically, hardware is rolled out to host applications, databases, etc. Applications have a lifecycle, they start in the development environment, then move to test, stage, performance, and finally production. Separate hardware is allocated for each of these environments, each with different characteristics – performance, cost, redundancy, etc Again, we can update our model with this new lifecycle dimension – East Coast-Exadata-Development Zone, East Coast-Exadata-Production Zone, etc

It is important to note, that all of the above dimensions are derived based on my experience with a bunch of customers. You are not required to use all of them, and it always helps to keep things simple. Lets look at pools next.

B. Modeling Pools

Pools are more software and platform centric and thus can be modeled based on various dimensions. Common dimensions are:

• Service Type: EM supports 3 service types – database, schema, and pluggable databases
• Version: This is the database software / Oracle home version
• Platform: This corresponds to operating systems like linux x86, solaris, hp-ux, etc
• High Availability: This represents if the infrastructure is RAC or SI (single instance)
• Disaster Recovery: Indicates if the pool will be used to host standby databases or just primary

So the naming format for Pools could be like this: 
<Service_Type>-<Version>-<Platform>-<HA>-<DR> 

Some examples: 
DB-11204-linux-RAC, DB-11204-linux-SI-STANDBY, PDB-12102-RAC, etc

Again, as i said before these are mere suggestions, and it is really up to you to decide which works best for you.

Lets come back to our example from above. If i try to implement the resource model so as to deploy a highly available and redundant database service, it would look something like this:

The zone name should indicate its composition. In the pool name, i skip the platform part since i am using Exadata and hence i assume Linux automatically. If you are using VMs or commodity hardware you may want to specify the platform as well. So my pools are composed of GI clusters deployed on Exadata compute nodes and the DB Oracle home pre-provisioned. Note a pool can contain multiple clusters. At the time of provisioning, the placement algorithm ensures that the requested database is created on a cluster with least utilized nodes. This is the power of automation provided by EM 12c.

In summary, with the EM12c resource model, cloud providers have the ability to organize and manage their infrastructure the way they would like, while keeping the consumer experience simple and intuitive.

References:

Screenwatch: Build Service Catalogs with EM12c DBaaS

Advertisement

What is EM 12c DBaaS Snap Clone?

Tags

Automation, cloning, dbaas, em12c, private cloud, snap clone

[This is a repost of my original blog on Oracle blog from about a year ago.]

Lets look at a relatively new feature in EM that has generated significant interest in a very short time – EM 12c DBaaS Snap Clone.

The ‘Oracle Cloud Management Pack for Oracle Database’ a.k.a
the Database as a Service (DBaaS) feature in EM 12c has grown tremendously
since its release two years ago.  It started with basic single instance and RAC database provisioning, a technical service catalog, an out of box self service portal, metering and chargeback, etc. But since then we have added provisioning of schemas and pluggable databases, add/remove of Dataguard standby databases, full clones using RMAN backups, and Snap Clone. This video showcases the various EM12c DBaaS features.

This blog will cover one of the most exciting and popular features – Snap Clone. In one line, Snap Clone is a self service approach to creating rapid and space
efficient clones of large (~TB) databases.

Self Service – empowers the end users (developers, testers, data analysts, etc) to get access to database clones whenever they need it.
Rapid – implies the time it takes to clone the database. This is in minutes and not hours, days, or weeks.
Space Efficient – represents the significant reduction in storage (>90%) required for cloning databases

Customer Scenario

To best explain the benefits of Snap Clone, let’s look at a Banking customer scenario:

• 5 production databases total 30 TB of storage
• All 5 production databases have a standby
• Clones of the production database are required for data analysis and reporting
• 6 total clones across different teams every quarter
• For security reasons, sensitive data has to be masked prior to cloning

Based on the above scenario, the storage required, if using traditional cloning techniques, can be calculated as follows:

5 Prod DB                  = 30 TB
5 Standby DB            = 30 TB
5 Masked DB             = 30 TB (These will be used for creating clones)
6 Clones (6 * 30 TB) = 180 TB
——————
Total                           = 270 TB
Time = days to weeks

As the numbers indicate, this is quite horrible. Not only 30 TB turn into 270 TB, creating 6 clones of all production databases would take forever. In addition to this, there are other issues with data cloning like:

• Lack of automation. Scripts are good but often not a long term solution.
• Traditional cloning techniques are slow while, existing storage vendor solutions are DBA unfriendly
• Data explosion often outpaces storage capacity and hurts ITs ability to provide clones for dev and testing
• Archaic processes that require multiple users to share a single clone, or only supports fixed refresh cycles
• Different priorities between DBAs and Storage admins

Snap Clone to the Rescue

All of the above issues lead to slow turnaround times, and users have to wait for days and weeks to get access to their databases. Basically, we end up with competing priorities and requirements, where the user demands self service access, rapid cloning, and the ability to revert data changes, while IT demands standardization, better control, reduction in storage and administrative overhead, better visibility into the database stack, etc.

EM 12c DBaaS Snap Clone tries to address all these issues. It provides:

• Rapid and space efficient cloning of databases by leveraging storage copy-on-write (or similar) technology
• Supports all database versions from 10g to 12c
• Supports various storage vendors and configurations NAS and SAN
• Lineage and association tracking between clone master and its various clones and snapshots
• ‘Time Travel’ capability to restore and access past data
• Deep visibility into storage, OS, and database layer for easy triage of performance and configuration issues
• Simplified access for end user via out-of-the-box self service portal
• RESTful APIs to integrate with custom portals and third party products
• Ability to meter and charge back on the clone databases

So how does Snap Clone work?

The secret sauce lies in the Storage Management Framework (SMF) plug-in. This plug-in sits between the storage system and the DBA, and provides the much needed layer of abstraction required to shield DBAs and users from the nuances of the different storage systems. At the storage level, Snap Clone makes use of storage copy-on-write (or similar) technology. There are two options in terms of using and interacting with storage:

1. Direct connection to storage: Here storage admins can register NetApp and ZFS storage appliance with EM, and then EM directly connects to the storage appliance and performs all required snapshot and clone operations. This approach requires you to license the relevant options on the storage appliance, but is the easiest and the most efficient and fault tolerant approach.

2. Connection to storage via Solaris file system (ZFS): This is a storage vendor agnostic solution and can be used by any customer. Here instead of connecting to storage, the storage admin mounts the volumes to a Solaris server and format it with ZFS file system. Now all snapshot and clone operations required on the storage are conducted via ZFS file system,. The good thing about this approach is that it does not require thin cloning options to be licensed on the storage since ZFS file system provides these capabilities.

For more details on how to setup and use Snap Clone, refer to a previous blog post.

Now, lets go back to our Banking customer scenario and see how Snap Clone helped then reduce their storage cost and time to clone.

5 Prod DB                      = 30 TB
5 Standby DB                 = 30 TB
5 Masked DB                 = 30 TB
6 Clones (6 * 30 TB)      = 180 TB
6 Clones (6 * 5 * 2 GB) = 60 GB
——————
Total                               = 270 TB 90 TB
Time = days to weeks minutes

Assuming the clone databases will have minimal writes, we allocate about 2GB of write space per clone. For 5 production databases and 6 clones, this totals to just 60GB in required storage space. This is a whopping 99.97% savings in storage. Plus, these clones are created in matter of minutes and not the usual days or weeks. The product has out-of-the-box charts that show the storage savings across all storage devices and cloned databases. See the screenshot below.

Where can you use Snap Clone databases?

As i said earlier, Snap Clone is most effective when cloning large databases  (~TBs). Common scenarios we see our customers best use Snap Clone are:

• Application upgrade testing. For example, EBusiness suite upgrade to R12.
• Functional testing. For example, testing using production datasets.
• Agile development. For example, run parallel development sprints by giving each sprint its own cloned database.
• Data Analysis and Reporting. For example, stock market analysis at the close of market everyday.

Its obvious that Snap Clone has a strong affinity to applications, since its application data that you want to clone and use. Hence it is important to add that the Snap Clone feature when combined with EM12c middleware-as-a-service (MWaaS) can provide a complete end-to-end self service application deployment experience. If you have existing portals or need to integrate Snap Clone with existing processes, then use our RESTful APIs for easy integration with third party systems.

In summary, Snap Clone is a new and exciting way of dealing with data cloning challenges. It shields DBAs from the nuances of different storage systems, while allowing end users to request and use clones in a rapid and self service fashion. All of this while saving storage costs. So try this feature out today, and your development and test teams will thank you forever.

Additional References

Cloud Management Page on OTN

Cloud Administration Guide (Documentation)

Enterprise Manager 12c Database-as-a-Service Snap Clone Overview (Presentation)

Limit User Access to Oracle Private Database Cloud Portal

Tags

database as a service, dbaas, em12c, private cloud

When implementing Database as a Service (DBaaS) and/or Snap Clone, a common request was for a way to hide the other service types like IaaS, MWaaS, etc from the self service portal for the end users. Before EM12c R4, there was no way to restrict the portal view. Essentially, any user with the EM_SSA_USER role would be directed to the self service portal and would then be able to see all service types supported by EM12c.

Of course, you could always set Database as your default self service portal from the ‘My Preferences’ pop up, but this only helps with their post-login experience. The end user still gets to see all the options as shown in screen above.

In EM12c R4, a new out of the box role called EM_SSA_USER_BASE has been introduced. This role, by default, does not give access to any portal, that is an explicit selection. Here is how you use this role:

1. Create a custom role and add the EM_SSA_USER_BASE role to it.

2. Now in the Resource Privileges step, select the Resource Type ‘Cloud Self Service Portal for Database’, and edit it

3. Check the ‘Access the Cloud Self Service Portal for Database.’ privilege. Finish the rest of the wizard.

Now, when a user with this custom role accesses the self service portal, they can only do so for databases and nothing else.

While the EM_SSA_USER role will continue to work, we recommend you start using the new EM_SSA_USER_BASE role. For more details on DBaaS or Snap Clone roles, refer to the cloud admin guide chapter on roles and users.

Oracle Private Database Cloud REST APIs

Tags

apis, Automation, database as a service, dbaas, em12c, private cloud

While we spend a lot of time developing GUI for our private cloud features – like the self service portal, setup screens, etc, a large percentage of our customers use these features via our APIs, both EMCLI and REST based.

Q: So why would anyone not use the out of the box cloud (self service) portal and use the APIs instead?

A: Well, there are many reasons:

1. You already have a custom cloud portal that you use to deploy non-Oracle products and now you would like Oracle products to be part of it
2. You want cloud related actions to be part of a larger orchestration flow. For example, prior to provisioning, you want to integrate an approval workflow, and then post-provision update your asset management system with the service details.
3. You want to integrate the cloud actions with our IT service desk or ticketing systems
4. Integrate actions like cloning of databases (Snap Clone), or provisioning of middleware and applications with your continuous integration or devops process.

I am sure there are more reasons, but this list should do for now.

Q: What kind of APIs does EM12c expose?

A: Typically, we identify two key roles in the Private Cloud space:

1. Service Providers – This role represents the various administrators like DBAs, sysadmins, storage admins, etc that help setup and manage the private cloud infrastructure. EM12c provides EMCLI verbs and RESTful APIs for all actions that can be performed by this role.
2. Service Consumers – This role represents the users who request and consume cloud based services like databases, VMs, middleware servers, etc. This role can access the cloud services from either the out-of-the-box cloud portal, or using RESTful APIs.

This blog post is focused on the the RESTful APIs provided for Service Consumers. The best way to understand and learn about these APIs is to see them in action. In the video below, i explain the various concepts (note i assume that you are already familiar with EM12c DBaaS concepts, if not check the references section), demo the cloud portal, and then perform the same actions using RESTful APIs. The demo is approx. 23 minutes long, so grab a drink before you get started.

The Python script used to demo the APIs can be found here. Note: i am posting this script as sample implementation, so there is no formal support available for it. Also, the demo was recorded on the latest version of EM – EM12c R4 Plugin Update 1.

Finally, I am often asked about customer examples. Till date, i have seen customers use our REST APIs from Chef recipes, shell scripts using the curl command, from 3rd party tools like Service Now and VMware vCAC, custom Java based applications, etc. Unfortunately, i am not an expert on any of these 3rd party tools, so i probably cannot tell you how the integration was done, but i would be more than happy to help you with the EM REST APIs.

Hopefully this was a good introduction to our EM12c Database as a Service (Private Database Cloud) REST APIs. I encourage you to play with the APIs and post code examples for the benefit of other users.

References:

Screenwatch: Oracle Enterprise Manager 12c: Build Service Catalog with DBaaS

Documentation: Cloud Administration Guide, REST API Chapter