Sunday, February 27, 2011

Why no-data-found message on JVM Diagnostics Page?

Assuming that JVM Dignostics monitoring has been started.
If you are seeing No-Data-Found message on JVM Diagnostics Pages, there can be one of the following reasons:
1. JVMD (AD4J) manager is not deployed or is inactive. If JVMD manager is not yet deployed then deploy the manager. If it is inactive then, follow instructions given on http://neelimabawa.blogspot.com/2011/02/how-do-i-know-if-jvm-diagnostics-ad4j.html

2. JVMD(AD4J) Manager is deployed and is active but is not able to connect to its repository (EM database). May be at some point EM database was restarted but JVMD(AD4J) manager was not bounced at that time. Hence all previous db connections were closed. Solution is bounce the JVMD(AD4J) manager application or may be bounce the managed server that is running the JVMD(AD4J) manager
application. It should resolve DB connection issue.

3. JVMD (ad4j/jam agent) is not deployed on target JVM or may be inactive for some reason. Check agent logs, they should give some pointer.

4. JVMD(ad4j/jam agent) is deployed on target JVM and is also active (confirmed with agent logs) but for some reason, agent is not able to connect to manager.
For more details verify agent logs.

5. JVMD(ad4j/jam agent) is deployed on target JVM and is also active (confirmed with agent logs) also connected to manager. But further data is not getting exchanged betweeb agent and manager. For more details verify agent and manager logs.

6. If you are able to Real Time analysis Thread and Heap pages but no monitoring data. Then most likely that monitoring has not started for that JVM Pool/Domain. To start monitoring for that JVM Pool/Domain, goto Targets->Middleware->JVM Diagnostics->Setup->JVM Pools Page. Edit JVM Pool properties and select 'Poll Enabled' property for that JVM Pool/Domain.

7. If agent was deployed manually. Then listing of agent depends on JVM pool mentioned in its web.xml. If web.xml was not modified then by default, manually deployed agent is reported under 'Default Pool'. By default, monitoring for 'Default Pool' is not started. So one needs to edit JVM Pool properties for
'Default Pool' and select 'Poll Enabled'.

8. Even after doing all the above, if UI shows no data found then one needs to dig into jam agent and manager logs to see what's going on.

Note: a. Ensure that JVMD monitoring is started. Verify it from Targets->Middleware->JVM Diagnostics->Setup->Monitoring Page.
b. JVM Pool/Domain for which you looking for JVMD data, should always have 'Poll
Enabled' proprty selected as mentioned in step 6. .

Wednesday, February 23, 2011

How to compare heaps using JVM Diagnostics?

When you suspect there might be some memory leak might be happening in Java/J2EE app. Heap comparision feature would require you to dump two heaps for same JVM over a period of time.  One needs to follow given steps to compare two heaps:
1. Goto Targets->Middleware->JVM Diagnostics page
2. Select target JVM on left Panel
3. Select JVM Menu->Heap->Take Snapshot Now as given in screen shot.

This command would generate a heapdump file on target m/c where JVM is running.
Heapdump filename would be given in next screen and it would be like
/tmp/heapdump<N>.txt .(Default directory for a JVM is /tmp or C:\tmp)

4. Similar way, take another heap snapshot.
5. One you have two heapdump files on target JVM machine. ftp these files to or EM repository machine.
6. Download loadheap.zip from Targets->Middleware->JVM Diagnostics->Setup->Download page. Unzip loadheap.zip, it contains various scripts including :
-loadheap.sh/loadheap.bat,
-create_jvm_diagnostic_db_user.sh/create_jvm_diagnostic_db_user.bat scripts etc.
%create_jvm_diagnostic_db_user.sh
7. Create JVMDIAG user using create_jvm_diagnostic_db_user.sh/bat script.
Note: Use http://download.oracle.com/docs/cd/E11857_01/install.111/e17558/ad4j_setup.htm#CEGGIIBB
8. Load these heapdump files to EM repository as JVMDIAG user using loadheap.sh/bat.
%loadheap.sh <heapdump_filename>
Once both the files are loaded successfully.
9. Goto Targets->Middleware->JVM Diagnostics. Select Pool/JVM->Heap->Saved Heap Snapshots on right panel.


It will give list of loaded heaps in EM repository.


10. Select one heap using % column.
11. Click on 'Compare Heaps' on blue ribbon
12. Select second heap using same % column
13. You can see heap comparison as given screen shots.
14. Click on View Summary to get summary details.


 

Friday, February 18, 2011

How do I know if JVM Diagnostics (AD4J) Manager is active?

1. To check the Manager status,
Goto Targets->Middleware->JVM Diagnostics->Setup->JVMs&Managers
as given in screen shot

or

2. Goto weblogic server where ad4j manager is deployed
and look at the EMAD4JMANAGER.out file. It should not contain
any errors.

Thursday, February 17, 2011

EM11g - Steps to Deploy AD4J (JVMD) manager on a Remote Machine

Step 1. Install EM agent on the remote host (host2) and point it to the OMS.

Step 2. Install weblogic server bits on host2 using EM installer. (Software only option).  This is to ensure same version and patch level for weblogic server on host2.
e.g. ./runInstaller -noconfig b_softwareonly=true

Step 3. Create a new server using the weblogic console of the emgc domain, name it as 'AD4JRemoteServer'. Make sure the listen address as the one of the remote host and a port number.

Step 4. Create a new machine from the weblogic console. Associate this machine with the nodemanager on remote host.
Enter the correct listen address ( the remote machine ), and the nodemanager port number( default 5556).
Add the managed server created in step 3 (AD4JRemoteServer) to the machine. ( Select the machine created and click on servers )

Step 5 . Now we will create a weblogic template based on EM GC domain running on local host using pack command. And unpack it on the remote host.
./pack.sh -domain=/u01/app/oracle/work/gc_inst/user_projects/domains/GCDomain -template=/u01/app/testing/emgcdomain.jar -template_name="My Weblogic Domain" -managed={true}
Where domain=/u01/app/oracle/work/gc_inst/user_projects/domains/GCDomain is the location of domain on local host. 
And ‘/u01/app/testing/emgcdomain.jar’ is the location where we want to create the template .

Step 6 . Now we will copy emgcdomain.jar to remote host .

Step 7 . Now we will unpack this template on remote host using the unpack command
./unpack.sh -domain=/u01/app/oracle/work/gc_inst -template=/net/myhost1/u01/app/testing/emgcdomain.jar
Where domain=/u01/app/oracle/work/gc_inst is the domain location of the remote machine and ‘/net/myhost1/u01/app/testing/emgcdomain.jar’ is the location of template where you copied emgcdomain.jar in step 6 on remote host.

Step 8. We need to enroll the weblogic domain with node manager using wlst command nmEnroll().
On the remote host, run
a. <Weblogic home>/common/bin/wlst.sh
b. Connect to the admin url of the weblogic console
    connect('weblogic','welcome1','t3://myhost1.us.oracle.com:7001')
    Note: the admin url should be precisely given. No '/' at the end, otherwise it gives weird MbeanServer exceptions
c. Register your domain with the following command
    nmEnroll(<Domain home>) 
    e.g. wls:/GCDomain/serverConfig> nmEnroll('/u01/app/oracle/work/gc_inst')
d. Exit Wlst script using exit()

Step 9. On the remote host, the nodemanager startup file will be located at  <Weblogic Home>server/bin/startNodeManager.sh
Before we run it however, we must set the property startScriptEnabled=true in the nodemanager.properties file.
This is to work around a weblogic JRF bug.
We can easily set it by running this script
<middleware home>/oracle_common/common/bin/setNMProps.sh
Now start the nodemanager using
<Weblogic Home>server/bin/startNodeManager.sh
Note down the nodemanager port number and mode. By default, it will run in ssl mode on port 5556

Step 10. Start up the server AD4JRemoteServer from the weblogic console. It will send a request to the nodemanager to start it and in a short while show up as running.
Weblogic server setup is done. Now discover the EM domain in GC. If you have discovered it already do a “Farm Refresh” so that new managed server is discovered in EM.
Following jars are required by AD4J Manager in 11g. Add these in CLASSPATH of AD4JRemoteManager
“/u01/app/oracle/work/middleware1/oms11g/sysman/jlib/emCORE.jar:/u01/app/oracle/work/middleware1/oms11g/sysman/jlib/emagentSDK.jar:/u01/app/oracle/work/middleware1/oms11g/sysman/jlib/log4j-core.jar:/u01/app/oracle/work/middleware1/oms11g/sysman/jlib/emcoreAgent.jar”
Save and then restart the server for it to take effect.

Step 11. If you are using EM 11g, proceed to Weblogic Console to deploy AD4J Manager i.e. jammanager.ear
Download jammanager.ear from OMS i.e. Targets->Middleware->JVM Diagnostics->Setup->Download Page and deploy it on target AD4JRemoteManager. You need to ensure that port no. 3600 is available on remote host. This is the listen port that ad4j manager needs.

Thursday, February 10, 2011

Middleware mgmt: How to generate URL availability report for web applications?

1. Goto Targets->System, Add System, Provide System Name say ad4j,
2. Add Components to system. In my case, I added two weblogic Servers as Target components.
wls1 -> that has webapp1 URL
wls2 -> that has webapp2 URL
3. Goto Target->Services,
Add Generic Service ,
Provide Service Name e.g. ad4j_service
Select System i.e. ad4j Define Key components of system (Select needed web applications, in my case it's ad4j_http_test and petstore_http_test)
Define availability based on Service Tests
Select Test Type -> Http Ping->Give url_test_name and specify complete webapp1_url
Create Beacon->select agent which will be executing the http url test->Create beacon for that
Click on 'Verify Service Test', then move to next step
Select Performance Metrics (I went for default values for now)->next->..
Click on Finish
4. Till now we have defined the  system and Service based one web application url. Let's add another webapp_url.
Again goto Targets->Service
Select service web_app_service and click on 'Configure' button
Goto 'Monitoring Configuration'->Select Service Tests and Beacons
Add another 'HTTP ping' test -> Provide test name and url->Verify Service test and click on 'ok'
Enable newly added test
When check box appears under 'Key Service Test', select all tests and click on 'Change Key tests' button below this table
click ok
5.Goto 'Monitoring Configuration'->Availability Definition
Select 'At least one key service test is successful'. Click OK.
Select Yes when asked for confirmation.
6. You can get the URL availability report under Targets->Services
Click availability status icon for selected service.
It will give detailed report for entire service.

Given is example of URL availability if anyone of the URL is available and user wants to show status of web app as Up.

If one wants all URLs to be up and then show status as up, one needs to modify the availability Definition for Service and select service test rule as 'All key service tests are successful'.

Monday, February 07, 2011

Target Weblogic Server shows status down in OEM even though server is Up.

Weblogic Server has been discovered and added. It's up but in Grid Control UI, status shows down and emagentfetchlet.log shows following error:
Error encountered after accessing the mbean(java.lang:type=Runtime) for metric weblogic_j2eeserver:jvm_runtime from server:service:jmx:t3://:/jndi/weblogic.management.mbeanservers.runtime:: java.lang:type=Runtime Cause: null

Then :
1. Verify that Target Weblogic Server Domain->Advance seetings should contain Platform MBean Server Enabled.
2. Restart Weblogic Server
3. Try refreshing WLS Domain
4. If nothing works try deleting and adding Weblogic target.

Thursday, February 03, 2011

What is acsera agent?

acsera agent is ADP (Application Dependency and Performace) agent. Also known as CAMM agent. It's a j2ee application that is deployed on an app server to diagnose composite apps and get ADP metrics.