Oracle Private Database Cloud: Defining Database Sizes in the Service Catalog

Tags

database as a service, em12c, emcli, private cloud

Latest release of cloud plug-in (part of EM12c R4 Plug-in Update 1) brings the ability to define sizes for database cloud services (Schema and PDB services already support definition of size). Prior to this ability, customers were required to define a new template for each size – small, medium, large, etc. This will help in significantly reducing the number of templates required.

So lets see how to use them.

1. The CRUD operations for database size are performed via emcli verbs. To create a new size, we run:

./emcli create_database_size -name=Tiny -description="tiny size" 
        -attributes="cpu:2;storage:20;memory:2"

The 4 attributes supported for size are – cpu, memory, processes, & storage. All attributes are optional, and minimum name and description need to be provided.

To list all sizes, we run:

./emcli list_database_sizes
 ____________________________________________
 Name:tiny
 Description:tiny size
 CPU(cores):2
 Memory(GB):2
 Processes(COUNT):Not Specified
 Storage(GB):20
 ____________________________________________

Had this size been assigned to any service templates, we could have provided the -details flag and gotten that list as well.

Editing this size is equally easy. We run:

./emcli update_database_size -name=tiny 
        -description="Tiny database size" 
        -attributes="storage:remove;processes:500"
 ____________________________________________
 Name:tiny
 Description:Tiny database size
 CPU(cores):2
 Memory(GB):2
 Processes(COUNT):500
 Storage(GB):Not Specified
 ____________________________________________

Few things to notice are:

• name is the only fixed identifier, both attributes and description can be changed
• to remove an attribute, set its value to remove
• to set new attribute values, or change existing one, simply specify the new values

Finally, to delete a size, we run:

./emcli delete_database_size -name=tiny
 Are you sure you want to to delete?(yes/no)
 yes
 Database size tiny has successfully been deleted

Since delete is a destructive operation operation, you get the “Are you sure?” prompt.

2. Once sizes are defined, we can see them while creating or editing service templates. Note that by default, no sizes are attached with the service template. The administrator needs to explicitly associate a size with the template, as shown in the screenshot below.

3. Once the association is complete, and the template is saved. Any cloud consumer accessing the template via the cloud portal will see size as an additional input.

If you are using REST APIs, then database size will have to be provided as an additional input to the POST request.

A commonly asked question is that how are these values enforced? The answer depends on the the type of attribute:

• cpu – this translates to cpu_count and is set as the DB init parameter
• memory – this translates to memory_target and set as DB init parameter (This is a good time to mention that the memory attribute is only supported for DB version 11g and above. Other attributes work for 10g and above.)
• storage – this is used only for monitoring purposes, and not enforced. So at the time of provisioning we ensure that the storage requirements for the service are within the quota limits of the consumer, but these limits are not set on the database.
• processes – this is set as DB init parameter

In summary, the ability to define size as an external entity will help drastically reduce the number of templates required to be defined by the cloud administrators.

Additional Resources:

EMCLI Documentation

Advertisement

What is EM 12c DBaaS Snap Clone?

Tags

Automation, cloning, dbaas, em12c, private cloud, snap clone

[This is a repost of my original blog on Oracle blog from about a year ago.]

Lets look at a relatively new feature in EM that has generated significant interest in a very short time – EM 12c DBaaS Snap Clone.

The ‘Oracle Cloud Management Pack for Oracle Database’ a.k.a
the Database as a Service (DBaaS) feature in EM 12c has grown tremendously
since its release two years ago.  It started with basic single instance and RAC database provisioning, a technical service catalog, an out of box self service portal, metering and chargeback, etc. But since then we have added provisioning of schemas and pluggable databases, add/remove of Dataguard standby databases, full clones using RMAN backups, and Snap Clone. This video showcases the various EM12c DBaaS features.

This blog will cover one of the most exciting and popular features – Snap Clone. In one line, Snap Clone is a self service approach to creating rapid and space
efficient clones of large (~TB) databases.

Self Service – empowers the end users (developers, testers, data analysts, etc) to get access to database clones whenever they need it.
Rapid – implies the time it takes to clone the database. This is in minutes and not hours, days, or weeks.
Space Efficient – represents the significant reduction in storage (>90%) required for cloning databases

Customer Scenario

To best explain the benefits of Snap Clone, let’s look at a Banking customer scenario:

• 5 production databases total 30 TB of storage
• All 5 production databases have a standby
• Clones of the production database are required for data analysis and reporting
• 6 total clones across different teams every quarter
• For security reasons, sensitive data has to be masked prior to cloning

Based on the above scenario, the storage required, if using traditional cloning techniques, can be calculated as follows:

5 Prod DB                  = 30 TB
5 Standby DB            = 30 TB
5 Masked DB             = 30 TB (These will be used for creating clones)
6 Clones (6 * 30 TB) = 180 TB
——————
Total                           = 270 TB
Time = days to weeks

As the numbers indicate, this is quite horrible. Not only 30 TB turn into 270 TB, creating 6 clones of all production databases would take forever. In addition to this, there are other issues with data cloning like:

• Lack of automation. Scripts are good but often not a long term solution.
• Traditional cloning techniques are slow while, existing storage vendor solutions are DBA unfriendly
• Data explosion often outpaces storage capacity and hurts ITs ability to provide clones for dev and testing
• Archaic processes that require multiple users to share a single clone, or only supports fixed refresh cycles
• Different priorities between DBAs and Storage admins

Snap Clone to the Rescue

All of the above issues lead to slow turnaround times, and users have to wait for days and weeks to get access to their databases. Basically, we end up with competing priorities and requirements, where the user demands self service access, rapid cloning, and the ability to revert data changes, while IT demands standardization, better control, reduction in storage and administrative overhead, better visibility into the database stack, etc.

EM 12c DBaaS Snap Clone tries to address all these issues. It provides:

• Rapid and space efficient cloning of databases by leveraging storage copy-on-write (or similar) technology
• Supports all database versions from 10g to 12c
• Supports various storage vendors and configurations NAS and SAN
• Lineage and association tracking between clone master and its various clones and snapshots
• ‘Time Travel’ capability to restore and access past data
• Deep visibility into storage, OS, and database layer for easy triage of performance and configuration issues
• Simplified access for end user via out-of-the-box self service portal
• RESTful APIs to integrate with custom portals and third party products
• Ability to meter and charge back on the clone databases

So how does Snap Clone work?

The secret sauce lies in the Storage Management Framework (SMF) plug-in. This plug-in sits between the storage system and the DBA, and provides the much needed layer of abstraction required to shield DBAs and users from the nuances of the different storage systems. At the storage level, Snap Clone makes use of storage copy-on-write (or similar) technology. There are two options in terms of using and interacting with storage:

1. Direct connection to storage: Here storage admins can register NetApp and ZFS storage appliance with EM, and then EM directly connects to the storage appliance and performs all required snapshot and clone operations. This approach requires you to license the relevant options on the storage appliance, but is the easiest and the most efficient and fault tolerant approach.

2. Connection to storage via Solaris file system (ZFS): This is a storage vendor agnostic solution and can be used by any customer. Here instead of connecting to storage, the storage admin mounts the volumes to a Solaris server and format it with ZFS file system. Now all snapshot and clone operations required on the storage are conducted via ZFS file system,. The good thing about this approach is that it does not require thin cloning options to be licensed on the storage since ZFS file system provides these capabilities.

For more details on how to setup and use Snap Clone, refer to a previous blog post.

Now, lets go back to our Banking customer scenario and see how Snap Clone helped then reduce their storage cost and time to clone.

5 Prod DB                      = 30 TB
5 Standby DB                 = 30 TB
5 Masked DB                 = 30 TB
6 Clones (6 * 30 TB)      = 180 TB
6 Clones (6 * 5 * 2 GB) = 60 GB
——————
Total                               = 270 TB 90 TB
Time = days to weeks minutes

Assuming the clone databases will have minimal writes, we allocate about 2GB of write space per clone. For 5 production databases and 6 clones, this totals to just 60GB in required storage space. This is a whopping 99.97% savings in storage. Plus, these clones are created in matter of minutes and not the usual days or weeks. The product has out-of-the-box charts that show the storage savings across all storage devices and cloned databases. See the screenshot below.

Where can you use Snap Clone databases?

As i said earlier, Snap Clone is most effective when cloning large databases  (~TBs). Common scenarios we see our customers best use Snap Clone are:

• Application upgrade testing. For example, EBusiness suite upgrade to R12.
• Functional testing. For example, testing using production datasets.
• Agile development. For example, run parallel development sprints by giving each sprint its own cloned database.
• Data Analysis and Reporting. For example, stock market analysis at the close of market everyday.

Its obvious that Snap Clone has a strong affinity to applications, since its application data that you want to clone and use. Hence it is important to add that the Snap Clone feature when combined with EM12c middleware-as-a-service (MWaaS) can provide a complete end-to-end self service application deployment experience. If you have existing portals or need to integrate Snap Clone with existing processes, then use our RESTful APIs for easy integration with third party systems.

In summary, Snap Clone is a new and exciting way of dealing with data cloning challenges. It shields DBAs from the nuances of different storage systems, while allowing end users to request and use clones in a rapid and self service fashion. All of this while saving storage costs. So try this feature out today, and your development and test teams will thank you forever.

Additional References

Cloud Management Page on OTN

Cloud Administration Guide (Documentation)

Enterprise Manager 12c Database-as-a-Service Snap Clone Overview (Presentation)

Limit User Access to Oracle Private Database Cloud Portal

Tags

database as a service, dbaas, em12c, private cloud

When implementing Database as a Service (DBaaS) and/or Snap Clone, a common request was for a way to hide the other service types like IaaS, MWaaS, etc from the self service portal for the end users. Before EM12c R4, there was no way to restrict the portal view. Essentially, any user with the EM_SSA_USER role would be directed to the self service portal and would then be able to see all service types supported by EM12c.

Of course, you could always set Database as your default self service portal from the ‘My Preferences’ pop up, but this only helps with their post-login experience. The end user still gets to see all the options as shown in screen above.

In EM12c R4, a new out of the box role called EM_SSA_USER_BASE has been introduced. This role, by default, does not give access to any portal, that is an explicit selection. Here is how you use this role:

1. Create a custom role and add the EM_SSA_USER_BASE role to it.

2. Now in the Resource Privileges step, select the Resource Type ‘Cloud Self Service Portal for Database’, and edit it

3. Check the ‘Access the Cloud Self Service Portal for Database.’ privilege. Finish the rest of the wizard.

Now, when a user with this custom role accesses the self service portal, they can only do so for databases and nothing else.

While the EM_SSA_USER role will continue to work, we recommend you start using the new EM_SSA_USER_BASE role. For more details on DBaaS or Snap Clone roles, refer to the cloud admin guide chapter on roles and users.

Oracle Private Database Cloud REST APIs

Tags

apis, Automation, database as a service, dbaas, em12c, private cloud

While we spend a lot of time developing GUI for our private cloud features – like the self service portal, setup screens, etc, a large percentage of our customers use these features via our APIs, both EMCLI and REST based.

Q: So why would anyone not use the out of the box cloud (self service) portal and use the APIs instead?

A: Well, there are many reasons:

1. You already have a custom cloud portal that you use to deploy non-Oracle products and now you would like Oracle products to be part of it
2. You want cloud related actions to be part of a larger orchestration flow. For example, prior to provisioning, you want to integrate an approval workflow, and then post-provision update your asset management system with the service details.
3. You want to integrate the cloud actions with our IT service desk or ticketing systems
4. Integrate actions like cloning of databases (Snap Clone), or provisioning of middleware and applications with your continuous integration or devops process.

I am sure there are more reasons, but this list should do for now.

Q: What kind of APIs does EM12c expose?

A: Typically, we identify two key roles in the Private Cloud space:

1. Service Providers – This role represents the various administrators like DBAs, sysadmins, storage admins, etc that help setup and manage the private cloud infrastructure. EM12c provides EMCLI verbs and RESTful APIs for all actions that can be performed by this role.
2. Service Consumers – This role represents the users who request and consume cloud based services like databases, VMs, middleware servers, etc. This role can access the cloud services from either the out-of-the-box cloud portal, or using RESTful APIs.

This blog post is focused on the the RESTful APIs provided for Service Consumers. The best way to understand and learn about these APIs is to see them in action. In the video below, i explain the various concepts (note i assume that you are already familiar with EM12c DBaaS concepts, if not check the references section), demo the cloud portal, and then perform the same actions using RESTful APIs. The demo is approx. 23 minutes long, so grab a drink before you get started.

The Python script used to demo the APIs can be found here. Note: i am posting this script as sample implementation, so there is no formal support available for it. Also, the demo was recorded on the latest version of EM – EM12c R4 Plugin Update 1.

Finally, I am often asked about customer examples. Till date, i have seen customers use our REST APIs from Chef recipes, shell scripts using the curl command, from 3rd party tools like Service Now and VMware vCAC, custom Java based applications, etc. Unfortunately, i am not an expert on any of these 3rd party tools, so i probably cannot tell you how the integration was done, but i would be more than happy to help you with the EM REST APIs.

Hopefully this was a good introduction to our EM12c Database as a Service (Private Database Cloud) REST APIs. I encourage you to play with the APIs and post code examples for the benefit of other users.

References:

Screenwatch: Oracle Enterprise Manager 12c: Build Service Catalog with DBaaS

Documentation: Cloud Administration Guide, REST API Chapter