Create EM Jobs via EMCLI

Found this saved but unpublished. Only 3 yrs late. 🙂

Basic flow:

Option 1: new job

1. list job types
2. describe job type and export to file
3. edit the exported file
4. create new job

Option 2: from existing job or from library job

1. list job type
2. describe job or describe job from library and export to file
3. edit the exported file
4. create new job

Other job related actions supported from EMCLI:

• Track progress
• create library jobs
• controlling job state – suspend, resume, stop, etc
• import/export

Refer to latest EMCLI documentation for more details.

Advertisement

Discover and Promote Oracle Homes as EM Targets

Tags

add target, em12c, emcli, job system, oracle home

Typically, Oracle Homes are discovered and promoted as targets automatically along with guided flows for addition of primary targets like databases, weblogic domains, etc, but there might be instances (not very often) where you need to discover the Homes standalone.

There are two ways to do this – from the GUI and using EMCLI verbs

A. From GUI

The steps are as follows:

1. Goto Enterprise->Job->Activity menu item
2. Here select the job type ‘Discover Promote Oracle Home Target’ and click Go
3. Provide the obvious inputs like name, list of hosts, etc, but the most important tab is that of ‘Parameters’. Here you are required to provide 3 inputs:
   1. Path to Oracle Home/Inventory/Composite Home/Middleware Home you want to manage
   2. The type of entity you want to manage. Your options are – Oracle Home, Inventory, Composite Home, or Middleware Home
   3. The action – discover and promote, or just discover [I almost always select the former]

That’s it. All remaining tabs are optional. Once the job is submitted, it usually takes a few seconds to complete. The output of the job clearly lists all the Oracle Homes discovered and the target names created for these Homes in EM.

B. Using EMCLI

1. First we describe the job type to get the list of required inputs

./emcli describe_job_type 
        -job_type=discoverPromoteOHTargets  > inputs.prop

If the explanation of the required fields is not sufficient, you can additionally pass the -verbose flag to get more details.

Now update the inputs.prop file with the relevant values. My file looks like this (changed values in red):

# Description: (Optional) The user specified name of the job
name=promote_OH

# Description: (Optional) The job type for this job
type=discoverPromoteOHTargets

# Description: (Optional) The user specified description of the job
description=

# Description: The job owner. The job owner is the user who creates the job.
# Default: the logged in user
# The job owner information displayed here is for documentation only 
# and user is not expected to change it.
owner=

# Description: (Optional) The kind of job
# Legal Values: active, library
kind=active


# Fill in the target list before submitting.
# For Example:
#     target_list=MyTarget:cluster
target_list=abc.example.com:host

# Description: The type of targets to use for this job
targetType=host

# Description: (Required) Enter the action you want to perform.
# To run only discovery on the target, use : disc.To run discovery 
# and promotion on target to managed status, use : promote
variable.loc_action=promote

# Description: (Required) Enter the type of search to be performed. 
# All the homes in the Inventory/Middleware Home will be managed.
# For discovering Oracle Home, use : oh.For discovering Oracle Home's
# in inventory , use : inv.For discovering Oracle Home's in 
# middleware home, use : mwh
variable.loc_type=inv

# Description: (Optional) Enter Path to Oracle Home/Inventory/
# Composite Home/Middleware Home you want to manage.
variable.location=/u01/foo/oraInventory

# Description: (Optional) Notify the job owner when a selected state occurs
# Allowed Values:  SCHEDULED, RUNNING, ACTION_REQUIRED, SUSPENDED, SUCCEEDED, PROBLEMS
notification=

Note, i submit my job against a single host target, but you can provide a long comma separated list of <target_name>:<target_type>. Now we submit the job by passing the above inputs.prop file as input.

emcli create_job -input_file=property_file:inputs.prop
Creation of job "PROMOTE_OH" was successful.

Since the EM12c job system is asynchronous, the emcli verb will submit the job and return control almost instantly. We need to run different set of emcli verbs to get job progress.

emcli get_jobs -name=PROMOTE_OH
Name        Type                      Job ID                            Execution ID
      Scheduled            Completed            TZ Offset  Status     Status ID  Owner   Target Type
  Target Name
PROMOTE_OH  discoverPromoteOHTargets  10FBD71FF8C70205E050F00A07B46726  10FBD71FF8C90205E050F00A07B46726  
2015-03-10 22:34:33  2015-03-10 22:34:37  GMT-07:00  Succeeded  5          SYSMAN  host
  abc.example.com

The output is ill formatted, but the only 2 fields we care about are – Execution ID and the Status. Since this is a fairly quick job, the status is shown as Succeeded. If i wanted to view the output, i would run another emcli command, and for this we need the execution ID.

emcli get_job_execution_detail 
      -execution=10FBD71FF8C90205E050F00A07B46726 -xml -showOutput
<?xml version = '1.0' encoding = 'UTF-8'?>
<jobExecution jobOwner="SYSMAN" status="5" startTime="2015-03-11 05:34:34.0" id="10FBD71FF8C90205E05
0F00A07B46726" jobName="PROMOTE_OH" statusBucket="-5">
   <TargetList>
      <target name="abc.example.com" type="host" hostName="abc.example.com"/>
   </TargetList>
   <steps>
      <step command="DiscoverAndPromoteOH" status="5" name="RunCustomDiscovery" startTime="2015.03.1
0 22:34:34" endTime="2015.03.10 22:34:37" timezoneRegion="-07:00" stepId="825682" stepType="1" jobTy
pe="discoverPromoteOHTargets" stepNlsId="discoverPromoteOHTargets_RunCustomDiscovery" stepDefaultNam
e="RunCustomDiscovery" target="">
         <stepOutput>
            <output>Discovered Oracle Home Target 'OraDB12Home1_11_abc.example.com' with home
 location - /u01/db12/product/12.1.0/dbhome_1 in Inventory /u01/foo/oraInventory.
Successfully created 1 new Oracle Home Targets.
Succesfully added discovered homes matching given criteria.</output>
         </stepOutput>
      </step>
   </steps>
</jobExecution>

The output clearly states the outcome of the job, and if i check my all targets page, i will find the new Oracle Home target.

In summary, while there may not be many reasons to just discover and promote standalone Oracle Homes, if you ever need to do it, this blog tells you how to do it both via the GUI and EMCLI.

Convert Crontab to Enterprise Manager Jobs

Tags

Automation, crontab, em12c, job system

Surprisingly, a popular question posted on our internal forum is about the possibility of using the Enterprise Manager (EM) Job System to replace customer’s numerous cron jobs. The answer is obviously YES! I say surprisingly because the EM Job system has been in existence for around 10 years (I believe since EM 10.2.0.1), and my hope was that, by now, customers would have moved to using more enterprise class job schedulers instead of cron. So, here is a quick post on how to get started with this conversion from cron to EM Jobs for some of our new users.

Benefits of EM Job System

Before we learn about the how, let’s look at the why. The EM job system is:

• Free – (Yes, I said free) It is included with the base EM at no cost.
• Flexible – It supports multiple options for scheduling, notification, authentication, etc
• Infinitely scalable – the job system seamlessly scales to every new Oracle Management Server (OMS). In fact, in case of OMS failures, the job steps are automatically picked up by the next available OMS without affecting the job execution.
• General purpose – General purpose since it provides numerous out-of-the-box job types like run OS command, start/stop, backup, SQL Script, patch, etc that span multiple target types. As of today, there are over 50 job types available in the product.
• Enterprise grade – It allows users to automate multiple administrative tasks like backup, patching, cloning, etc across multiple targets. Customers have not only converted their cron jobs to EM, but have also replaced other enterprise tools like Autosys and migrated 1000s of jobs to EM Job System.
• APIs – Jobs can be scheduled and managed from the UI and using EMCLI (the command line interface).

Now back to our topic.

The Conversion Process

Let’s start with a sample crontab that we want to convert.

A cron expression consists of 6 fields, where the first 5 fields represent the schedule, while the last field represents the command or script to run.

Field Name	Mandatory?	Allowed Values	Allowed special characters
Minutes	Yes	0-59	* / , –
Hours	Yes	0-23	* / , –
Day of month	Yes	1-31	* / , – ? L W
Month	Yes	1-12 or JAN-DEC	* / , –
Day of week	Yes	0-6 or SUN-SAT	* / , – ? L #

Cron jobs run on the operating system, often using the native shell or other tools installed on the operating system. The equivalent of this capability in Enterprise Manager is the ‘OS Command’ job type. Here are the steps required to convert the first entry in the crontab to an EM job:

1. Navigate to the Job Activity page

2. Select the ‘OS Command’ job and click Go

A 5-tab wizard will appear. Let’s step through this one by one.

3. Select the first tab called ‘General’. Here provide a meaningful name and description for the job. Since this job will be run on the Host target, keep the target type selection as ‘Host’. Next, select all host targets in EM that you wish to run this script against.

While cron jobs are defined on a per host bases, in EM a job definition can be run and managed across multiple hosts or groups of hosts. This avoids having to maintain the same crontab across multiple hosts.

4. Select the ‘Parameters’ tab. Here enter the command or script as specified in the last field of the crontab entry. When constructing the command, you can make use of the various target properties.

5. Next select ‘Credentials’. Here we provide the credentials required to connect to the host and execute the required commands or scripts. Three options are presented to the user:

• Preferred – default credential set for the host
• Named – Credentials are stored within Enterprise Manager as “named” entities. Named credentials can be a username/password, or a public key-private key pair. Here we choose pre-created named credentials
• New – This allows us to create and use new named credential

Note: If your OS user does not have the required privileges to execute the set command, Named Credentials also support use of sudo, powerbroker, sesu, etc.

6. Next, we set the schedule and this is where it gets interesting. As discussed before, crontab uses a textual representation for the schedule, while EM Job system has a graphical representation for the schedule.

Our sample schedule in the crontab is ‘00 0 * * Sun’. This translates to a weekly job at 12 midnight on every Sunday. To set this in EM, choose the ‘Repeating’ schedule type. The screenshot below shows all the other selections.

The key here is to select the correct ‘Frequency Type’, the rest of the selections are quite obvious. This also lets you choose the desired timezone for the schedule. Your options are to either start the job w.r.t a fixed timezone, or start it in individual target’s timezone. The latter is very popular, for example, I want to start a job at 2 AM local time in every region around the world.

Another selection of note is that for ‘Grace Period’. This is an extremely powerful feature, but often not used by many customers. Typically, we expect jobs to be started within a few seconds or minutes (based on the load on the system and number of jobs scheduled) of the start time, but a job might not start on time for many reasons. The most common reasons are the Agent being down or due to a blackout. The grace period controls the latest start time for the job in case the job is delayed, else its is marked as skipped. By default, jobs are scheduled with indefinite grace periods, but I highly recommend setting a value for it. In the sample above, I set a 3 hr limit which may seem large but given the weekly nature of the job seems reasonable. So the job system will wait until 3 am (the job start time is 12 am) to start the job, after which the iteration will be skipped. For repeating schedules, the grace period should always be less than the repeat interval. If the job starts on time, the grace period is ignored.

7. Finally, we navigate to the ‘Access’ tab. This tab has two parts:

• Privilege assignment to roles and users: this allows you to control job level access for other users
• Email notifications for the Job owner: this allows you to control the events for which you wish to receive notifications. Note, this only sets notification for the job owner, the other users can subscribe to emails by setting up notification and/or incident rules.

To prevent EM from sending deluge of emails, I recommend the following settings in the notifications region:

• Match status and severity: Both
• Select severity of status: Critical
• Select status: Problems & Action Required

You can always come back and modify these settings to suit your needs.

Not all cron jobs need to be converted to OS command. For example, if you are taking Oracle database backups using cron, then you probably want to use the out-of-the-box  job type for RMAN scripts. Just provide the RMAN script, list of databases to run this against, and the credentials required to connect to the database. Similarly, if you run sqls on numerous databases, you can leverage the SQL Script job type for this purpose. There are over 50 job types available in EM12c, all available for use from the UI and EMCLI.

Finally, the best way to learn more about the EM Job System is to actually play with it. I also recommend blogs from Maaz, Kellyn, and other users on this topic. Good luck!!

References

Maaz Anjum: http://maazanjum.com/2013/12/30/create-a-simple-job-for-a-host-target-in-em12c/
Kellyn Pot’vin: http://dbakevlar.com/

EM12c Release 4: Job System Easter Eggs – Part 2: Export/Import of Job Definitions

Tags

Automation, em12c, job system

This is part 2 of my two part blog post on
Job System Easter Eggs (hidden features). The two features being covered are:

1. New Job progress tracking UI
2. Import/Export of job definitions

In the previous post, I talked about the new UI while in this post I will cover import and export for job definitions.

2.  Import/Export of job definitions

The ability to export and import job definitions across EM environments has been a popular enhancement request for quite some time. There are 3 primary drivers:

• Test to Production: Moving active and library job definitions from test site to production
• Migration: Move job definitions across EM versions. This is often used as an alternative to upgrade.
• As Failsafe: For that extra protection, in addition to repository backups. This can be useful for selective restoration of job definition.

In the first iteration, we are exposing these functions as emcli verbs.  Since these emcli verbs are not formally supported in the release, no documentation is included in the EMCLI Reference Guide. So to learn more about them,
we will have to look at emcli help. The two verbs are called
export_jobs, and import_jobs.

>> ./emcli help | grep _jobs

Some salient points about the import & export verbs are.

Export:

• Rich filter criteria to form the list of jobs to export
• Active and Library jobs can be exported
• Preview mode, to view results before exporting them
• Job definitions are exported as zip files for ease of transfer
• Contextual information like targets, credentials, access settings, etc is exported, but not imported. In future, we may be able to support import for this information as well.
• System jobs and nested jobs are not exported

Import:

• Special Preview mode to view contents of the exported file, and to determine possible conflicts during import
• Rich filter criteria to selectively import job definitions from the exported file
• Job definitions are imported to the library ONLY. This is true even if an active job was exported
• Two failure modes – skip on error, and rollback on error
• Import only works on same or higher version of EM12c. The patch set number matters.

Export Job Definitions

Let’s walk through an example.

1. We start with export_jobs. The help for this verb will show you all the available options. Most of the filters are self explanatory, so i will skip the explanation. The most important option of all is the -preview flag. This when used in conjunction with other filters, will show results without exporting the job definitions.

>> ./emcli help export_jobs

emcli export_jobs
[-name=”job name1;job name2;…”]
[-type=”job type1;job type2;…”]
[-targets=”tname1:ttype1;tname2:ttype2;…”]
[-owner=”owner1;owner2;…”]
[-preview]
-export_file=<Zip filename that will be created>”
-libraryjobs

2.  Now lets play with this verb. If we invoke the verb with just the -preview flag, it will list all job definitions that can be exported both that are active, and from the library. Note: system jobs and nested jobs are skipped from this output.

>> ./emcli export_jobs -preview

Not all job types are exportable. To determine the list of job types supported via the import and export verbs, use the get_job_types verb.

>>  ./emcli get_job_types

Currently, there are over 50 job types supported, and this list will continue to grow with every release.

3. From the list above, I am primarily interested in jobs that are created by the user AFULAY. So I apply the -owner filter.

>> ./emcli export_jobs -owner=AFULAY -preview

In this output i see 2 jobs, ‘Library Job’ which is a simple OS command job stored in the job library, while the ‘Test Job’ is an active OS command job scheduled against a bunch of targets.

Note: if multiple options are specified, like -name and -owner, then these are ANDed together.

4. Since i am the lazy kind, i would rather export every thing and then later decide what i wish to import. So here it goes. The -export_file option takes the location and file name for the export file. Note the actual output file is an xml that contains job metadata, but the export file is always archived and stored in a zip format. At this time, I am sure most would instinctively unzip the file and start perusing through its contents, but doing so would be analogous to removing the warranty void sticker of your new TV or Blue Ray player. Basically, attempts to manually modify the contents of the exported zip file is highly discouraged.

>> ./emcli export_jobs -export_file=/tmp/afulay_alljobs.zip

Note how the status column reports success or failure for each job being exported. With the file exported, we now move on to the import verb.

Import Job Definitions

In the previous section, we exported a file with all job definitions in it. Now lets say we share this file with bunch of other admins and ask them to import whatever job definitions that make sense or are relevant to their environment.

1. To understand the capabilities of the import verb, we take a look at the help content.

>> ./emcli help import_jobs

emcli import_jobs
[-name=”job name1;job name2;…”]
[-type=”job type1;job type2;…”]
[-targets=”tname1:ttype1;tname2:ttype2;…”]
[-owner=”owner1;owner2;…”]
[-preview]
[-force]
[-stoponerror]
-import_file=<Zip filename that contains job definitions>”

Most of the options are quite similar to export_jobs barring a few. The -force flag allows the admin to update an existing job definition. Typically, you will run into these situations when a conflicting job is found in the new environment, and you want to either update its definition with the new version in the import file or overwrite the localized changes. The -stoponerror flag, when specified, will stop the import process on first encountered failure, and then rollback all jobs imported in the session. We will likely change this label to rollbackonerror to correctly represent its behavior. The default behavior is to skip failed jobs and continue importing others.

2. Before our admins import job definitions, they first need to view the contents of the exported file. This again can be done using the -preview option.

>> ./emcli import_jobs -preview -import_file=/tmp/afulay_alljobs.zip

The -preview option in the import verb is very special. It not only just lists the contents of the exported file, but also connects to the new EM environment and looks for potential conflicts during import. So this is a deep validation test. As seen in the above screenshot, there are two sections in the output, first is just a listing of all job definitions from the import file, while the second is a list of all conflicts. Note: for demo sake, i am exporting and importing to the same EM site, and hence every job shows up as a conflict. To address this issue, i will eventually delete the ‘Library Job’ from my job library, and import it from the import file.

Disclaimer: In the interest of full disclosure, i should mention that there are few known bugs for the import verb, hence the rationale for not releasing these verbs formally with EM12c R4. Some bugs i ran into when writing this blog were: you cannot export an active job, delete it, and import it back to the same EM environment, this currently is only possible with library jobs.  This is an obscure case though. The -force flag is a little flaky, so sometimes it wouldn’t force import even if you want it to The -owner flag does not work on the import file, it instead will throw an exception That said, when the job does get imported, it does so properly, so there is never any risk of metadata corruption.

3. If i try to import the ‘Library Job’, the verb will fail and give me an error message.

>> ./emcli import_jobs -name=’LIBRARY JOB’ -import_file=/tmp/afulay_alljobs.zip

The Status column reports Error, while the Notes column gives the reason as ‘job already exists’.

4. Now lets delete the library job and try to import it.

>> ./emcli delete_library_job -name=’LIBRARY JOB’

>> ./emcli import_jobs -name=’LIBRARY JOB’ -import_file=/tmp/afulay_alljobs.zip

Success!!We were able to delete the library job, and import it back from the import file.

In summary, there two very useful enhancements made in EM12c R4. Unfortunately, due to time constraints and our inability to meet the set quality standards, we decided to ship these features in disabled state. This ensures that production sites and users are not impacted, while still giving the few brave souls the opportunity to test these features. In my assessment, i have found the new UI to be fairly robust as i have been using this exclusively for a while. On the other hand, there are few known bugs with the import and export emcli verbs, so use these with caution.

EM12c Release 4: Job System Easter Eggs – Part 1: New Progress Tracking UI

Tags

Automation, em12c, job system

So you just installed a new EM12c R4 environment or upgraded your existing EM environment to EM12c R4. Post upgrade you go to the Job System activity page (via Enterprise->Job->Activity menu) and view the progress details  of a job. Well nothing seems to have changed, its the same UI, the same multi-page drill down to view step output, same no. of clicks, etc. Wrong! In this two part blog post, i talk about two Job System Easter Eggs (hidden features) that most of you will find interesting. These are:

1. New Job progress tracking UI
2. Import/Export of job definitions
   

So before i go any further, let me address the issue of why are these features hidden? As we were building these features, we realized that we would not be ready to ship the desired quality of code by the set dates. Hence, instead of removing the code, it was decided to ship it in a disabled state so as not to impact customers, but still allowing a brave few to experiment with it and provide valuable feedback.

1.  New Job Progress Tracking UI

The job system UI hasn’t changed much since its introduction almost 10 years ago. It is a daunting task to update all the job system related UIs in a single release, and hence we decided to take a piecemeal approach instead. In the first installment, we have revamped the job progress tracking page.

The current UI, as shown above, while being very functional, is also very laborious. Multiple clicks and drill downs are required to view the step output for a particular target. Also, any click leads to complete page refresh, which leads to wastage of time and resources. The new UI tries to address all these concerns. It is a single page UI, which means no matter where you click, you never leave the page and thus never lose context of the target or step you where in. It also significantly reduces the no. of clicks required to complete the same task as in the current UI. So lets take a look at this new UI.

First, as i mentioned earlier, you need to enable this UI. To do this, you need to run the following emctl command from any of the OMS:

./emctl set property -name oracle.sysman.core.jobs.ui.useAdfExecutionUi -value true

This command will prompt for the sysman password, and then will enable the new UI.

NOTE: This command does not require a restart of the OMS. Once run, the new UI will be enabled for all user across all OMSes.

Now revisit the job progress tracking page from before. You will be directed to the new UI.

There are in all 6 key regions on this new single page job progress tracking UI. Starting from top left, these are:

1. Job Run Actions – These are actions that can be performed on the job run like suspend resume, retry, stop, edit, etc.
2. Executions – This region displays all the executions in the job run. An execution, in most cases, represents a single target and hence runs independently from other executions. This region thus shows the progress and status of all executions in a single view. The best part of this region is the column titled ‘Execution Time’. The cigar chart in this column represents two things – one, the duration of the execution, and two, the difference in start times. The visual representation helps in identifying runaway executions, or just compare execution times across different targets. The Actions menu allows various options like start, stop, debug, delete, etc.
3. Execution Summary – Clicking on an execution in the above region, paints the area on the right. This specific region shows execution summary with information like status, start & end date, execution id, command, etc
4. Execution Steps – This region lists the steps that make up the execution.
5. Step Output – Clicking on a step from the above region, paints this region. This shows the details of the step. This includes the step output and the ability to download it to a text file.
6. Page Options – We imagine that learning any new UI takes time, and hence this final region provides the option to switch between the new and the classic view. Additionally, this also allows you to set the auto refresh rate for the page.

Essentially, considering that jobs have two levels – executions and steps, we have experimented with a multi-master style layout. EM has never used such a layout and hence there were concerns raised when we chose to do so.

Master 1 (region 2) -> Detail 1 (regions 3, 4, & 5)

Master 2 (region 4) -> Detail 2 (region 5)

In summary, with this new UI, we have been able to significantly reduce the no. of clicks required to track job progress and drill into details. We have also been able to show all relevant information in a single page, thus avoiding unnecessary page redirection and reloads. I would love to hear from you if this experiment has paid off and if you find this new UI useful.

In the next part of this blog i talk about the new emcli verbs to import and export job definitions across EM environments. This has been a long standing enhancement request, and we are quite excited about our efforts.

Editing EM12c Jobs in Bulk

Tags

em12c, emcli, job system

I recently received requests for suggestions on how to edit EM12c jobs in bulk. The two examples that were presented to me were:

1. Change the start time of 70+ jobs
2. Change the oracle home path in 80+ RMAN script jobs

So how do we do this?

Option 1: Log into EM12c, navigate to each and every job, click edit and make the desired changes. This of course gets boring after the 4th job, not to mention the potential for developing carpel tunnel syndrome.

Option 2: The other choice is to scan the EM repository tables and try to update the correct columns. Unfortunately, the EM schema is not that simple to follow and your changes to the repository may not take effect as they haven’t updated all relevant references or have not been called from the correct set of APIs. BTW, this also violates our recommendation to never make changes to the EM schema without specific instructions from Oracle support or development, and has the potential of messing up your repository.

Option 3: This makes use of the job system emcli verbs, and is the only viable solution to the problem. The process would be as follows:

1. Use emcli verb “describe_job” to dump all required jobs into separate properties files (I am assuming you have list of job names that needs to be edited)
2. Mass-edit the property files to change the start date or any other parameters you like
3. Create new jobs using these property files with slightly different names using the emcli verb ”create_job”
4. Once satisfied with newly created jobs, remove older jobs using “delete_job” emcli verb

Documentation for these verbs can be found here.

Now you could argue that option 3 is not perfect either, and i would give you credit for making that argument. Hence an ER has been filed to provide a formal ‘edit_jobs’ verb which will allow you to edit jobs via emcli either in bulk or in a loop. Until then, hopefully this serves as a good work around.