Encrypting DataSource password in JBoss EAP 7.1

 

It’s always a risk if you are storing plain-text passwords on the file system. By default, JBoss EAP6/7 data source passwords are stored in plaintext inside standalone.xml/domain.xml.

JBoss EAP6 uses picketbox security implementation for encrypting data source passwords.

Redhat Link -  https://access.redhat.com/solutions/184963

https://middlewaretechadmin.blogspot.com/2017/10/how-to-use-encrypted-passwords-in-eap67.html

 

JBoss EAP7.1 uses “elytron” subsystem allows using credential stores as secure storage for your credentials

Redhat Link -  https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.1/html/how_to_configure_server_security/securely_storing_credentials#credential_store

 

High level Steps to Encrypting DataSource Passwords

Step-1: Take the backup of standalone.xml file.

Step-2: Select the application to perform the DB password encryption and keep the DB  password handy from standalone.xml  file.

Step-3: Check Test connection for the particular DataSource (By Using CLI)

Step-4: Create a Credential Store  (By Using CLI).

Step-5 : Add a DB  password  to a Credential Store (By Using CLI).

Step-6:  Now disable the Data source, remove the password and use a Stored Credential  in DataSource configuration (By Using CLI)

Step-7: Now enable the DataSource and restart the application server. Make sure application is started without any errors.

Step-8: Check test connection for the DataSources. (By Using CLI)

 

Detailed Steps

1. Check Test connection for the particular DataSource:

 /subsystem=datasources/data-source=eCore-DataSource:test-connection-in-pool

2. Create a Credential Store

/subsystem=elytron/credential-store=my_store:add(location="credentials/csstore.jceks", relative-to=jboss.server.data.dir,  credential-reference={clear-text=mypassword},create=true)

        The above command has created a Credential Store named "csstore.jceks" in the "jboss.server.data.dir/credentials" using a clear text password named "mypassword".

        3. Add a DB  password  to a Credential Store.

/subsystem=elytron/credential-store=my_store:add-alias(alias=database-pw, secret-value="secret")

The above command will add an alias into the credential store to reference the password ("secret") of DB user

4. Now disable the Data source

/subsystem=datasources/data-source=eCore-DataSource:disable

5. Remove the current plain text password from Data source configuration

  /subsystem=datasources/data-source=eCore-DataSource:undefine-attribute(name=password)

6. Add "credential-reference“ to Data source configuration  which points to your alias (which we created from step 3)

/subsystem=datasources/data-source=eCore-DataSource:write-attribute(name=credential-reference,value={store=my_store, alias=database-pw})

 we managed to replace correctly the password with a Credential Store reference.

7. Enable Datasource

/subsystem=datasources/data-source=eCore-DataSource:enable

8. Restart the app server &  Make sure application is started without any errors.

9. Check test connection for the DataSources, After app server started.

 /subsystem=datasources/data-source=eCore-DataSource:test-connection-in-pool

 Repeat  the above steps for other datasources.

 

 

 

 

Creating Alias for JBOSS CLI command in Linux env..

In this tutorial I'll share  creating alias for your JBOSS CLI command in Linux env.. 

Normally you can connect to a running standalone server or managed domain by using the connect command as below.

/jboss/jboss-eap-6.4/bin/jboss-cli.sh --connect --controller=127.0.0.1:9999  --user=admin --password=password

In JBOSS environment, often need to use JBOSS CLI commands repeatedly. Typing or copying the same command again and again reduces your productivity and distracts you from what you are doing.

You can save yourself some time by creating alias for your JBOSS CLI command.  Alias is like a shortcut command which will have same functionality as if we are writing the whole command.

First, I am creating CLI properties file as below, where Host, Port, User and Password are present.

[jboss@middleware jboss]$ cat cli.props  host="127.0.0.1" port=9999 user="jbossas" password="jboss@456" [jboss@middleware jboss]$

Next, I am adding below line into ~/.bash_profile and Save the same file.  

alias cli='_(){  (. /jboss/jboss-eap-6.4/cli.props ;  /jboss/jboss-eap-6.4/bin/jboss-cli.sh --controller=$host:$port --connect --user=$user --password=$password ; ) }; _'

The file will be automatically loaded in your next session. If you want to use the newly defined alias in the current session, issue the following command:

[jboss@middleware jboss]$ source ~/.bash_profile [jboss@middleware jboss]$

Test our alias command (cli) whether it's working or not. 

[jboss@middleware jboss]$ cli [domain@127.0.0.1:9999 /]

It's  working successfully....

Rolling garbage collector logs in JBOSS

In this post, I am going to explain  GC Logs Rotation configuration in JBoss.

Please check my previous blog post Enable GC logs in JBoss  to enable GC logs in Jboss.

Garbage Collection logs are essential artifacts to optimize application’s performance and trouble shoot complex memory problems. Garbage Collection logs can be generated in a particular file path by passing the “-Xloggc” JVM argument.

–Xloggc:/path/to/log/dir/gc.log

whenever the application is restarted, old GC log file will be over-ridden by the new GC log file as the file path is same.

Thus you wouldn’t be able to analyze the old GC logs that existed before restarting the application. Especially if the application has crashed or had certain performance problems then, you need old GC Logs for analysis.

Because of the heat of the production problem, most of the time, IT/DevOps team forgets to back up the old GC log file; A classic problem that happens again & again, that we all are familiar. Most human errors can be mitigated through automation and this problem is no exception to it.

A simple strategy to mitigate this challenge is to write new GC log contents in a different file location.   In this article 2 different strategies to do that are shared. 

1. Suffix timestamp to GC Log file :

If you can suffix the GC log file with the time stamp at which the JVM was restarted then, GC Log file locations will become unique. Then new GC logs will not over-ride the old GC logs. It can be achieved as shown below:

-Xloggc:/path/to/log/dir/gc.log-"`date +%Y-%m-%d-%H-%M`

This strategy has one minor drawback:

a. Growing file size

Suppose if you don’t restart your JVMs, then GC log file size can be growing to huge size.  Because in this strategy new GC log files are created only when you restart the JVM. But this is not a major concern in my opinion, because one GC event only occupies few bytes. So typically GC log file size will not grow beyond a manageable point.

2. Use -XX:+UseGCLogFileRotation

Another approach is to use the below JVM properties:

-Xloggc:/path/to/log/dir/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

When ‘-XX:-UseGCLogFileRotation’ is passed GC log rotation is enabled by the JVM itself.

‘-XX:NumberOfGClogFiles’ sets the number of files to use when rotating logs, must be >= 1. The rotated log files will use the following naming scheme, <filename>.0, <filename>.1, …, <filename>.n-1.

‘-XX:GCLogFileSize’ defines the size of the log file at which point the log will be rotated, must be >= 8K

But this strategy has few challenges:

a. Losing old GC Logs

Suppose if you had configured -XX:NumberOfGCLogFiles=5 then, over a period of time, 5 GC log files will be created:

gc.log.0 — oldest GC Log content gc.log.1 gc.log.2 gc.log.3 gc.log.4 — latest GC Log content

Most recent GC log contents will be written to ‘gc.log.4’ and old GC log content will be present in ‘gc.log.0’.

When the application starts to generate more GC logs than the configured ‘-XX:NumberOfGCLogFiles’ in this case 5, then old GC log contents in gc.log.0 will be deleted. New GC events will be written to gc.log.0. It means you will end up not having all the generated GC logs. You will loose the full visibility of the events.

b. Mixed-up GC Logs

Suppose application has created 5 gc log files i.e.

gc.log.0 gc.log.1 gc.log.2 gc.log.3 gc.log.4

Then, let’s say you are restarting the application. Now new GC logs will be written to gc.log.0 file and old GC log content will be present in gc.log.1, gc.log.2, gc.log.3, gc.log.4 i.e.

gc.log.0 — GC log file content after restart gc.log.1 — GC log file content before restart gc.log.2 — GC log file content before restart gc.log.3 — GC log file content before restart gc.log.4 — GC log file content before restart

So your new GC log contents get mixed up with old GC logs. Thus to mitigate this problem you might have to move all the old GC logs to a different folder before you restart the application.

Jboss AJP thread pool not released idle threads

In this post, I am going to explain the problem and the solution related to AJP connector executor leaking threads.

We may notice and detect that AJP thread pool with the thread counts will start increase continuously on application servers.  It looks like that the idle threads never recycle or reclaim even after keepalive-time. This behavior will happen continuously even if the system will remain idle for a long period of time (without any user request). The application leak connections because it is not closing the AJP connection and returning them to the pool.

In the case of unbounded-queue-thread-pool executor configured and the default maximum-connections value being used, a new thread will create for every new request and will remain idle forever. As the number of AJP requests increase, the response from Jboss EAP may get slow eventually.

The reason for this behavior is that the AJP thread-pool connections by default have no timeout (default value of "DEFAULT_CONNECTION_TIMEOUT" is -1 for AJP, which means it never recycle) and therefore, the thread connection will persist permanently once established. The same behavior detectable if we use httpd/mod_cluster in front of the JBoss which will invoke periodic pings to JBoss and fill its connection pool.

The solution is to add the following System property in standalone.xml or domain.xml to limit the keepalive timeout value for AJP:

<property name="org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT" value="600000" boot-time="true"/>

Here value 600000 menace 10 minutes . After 10 minutes of idle time, JBoss kills the thread.

However, if there are no strong reasons to use "DEFAULT_CONNECTION_TIMEOUT" option, keeping AJP connections open between apache and EAP is a good idea. The default value of "DEFAULT_CONNECTION_TIMEOUT" is -1 for AJP, which means it never disconnect.

Which Java Thread Consumes High CPU.. ?

 In this blog post, I am going to explain How to Find which Java Thread Consumes high CPU..

Frequently we are facing a strange issue with one of our JBOSS server in Production. The monitors that we configured on these always showed degraded performance. We started to analyze the performance of the server by using the basic top command which gave us the highest CPU and Memory. High CPU may cause System Crash or Server Crash or Slow Responsiveness of Server.

High CPU utilization is a very very common thing for an Application Server. It is very important for an Administrator to find out what is causing the High CPU.

Fallow the below steps to find out The High CPU consumed thread.  

1. We use top to find out the process id (PID) of the java process consuming most of the CPU.

top -n1 | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' '

2.  Next step is we started to find out the Thread ID that is causing the HIGH CPU usage in this process which gave us this.

printf "%x" $(top -n1 -H | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ')

3.  So we have a process ID of our busy JVM and Linux thread ID (most likely from that process) consuming our CPU. By using jstack we can be obtained Thread dump. Theses dumps/ Thread stack traces re very useful for find out root causes of issues.

$JAVA_HOME/bin/jstack <PID> | grep -A500 <ThreadID> | grep -m1 "^$" -B 500

The above command simply dumps the JVM stack trace of the given PID and filters out (using grep) the thread which has matching Thread ID.

Shell script :

I have written shell scripts as below with all above commands.  This script make easy to find out High CPU consumed JAVA PID & Thread ID and simply dumps the JVM stack trace.  

#!/bin/bash JAVA_HOME=/usr/java/jdk1.8.0_121/bin PID=$(top -n1 | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ') ThreadID=$(printf "%x" $(top -n1 -H | grep -m1 java | perl -pe 's/\e\[?.*?[\@-~] ?//g' | cut -f1 -d' ')) EAP=`ps -ef | grep java |grep -v grep | awk '{print $2 "," $9}' | grep $PID | awk -F, '{ print $2}' | sed -e 's/^-D//' | tr -d '[]' | sed -e 's/Server://'`   if [ -z "$PID" ] || [ -z "$NID" ] || [ -z "$EAP" ]; then        echo ""        echo "No High CPU consumption Threads at this moment"        exit 1   else        echo ""        echo -e "****************************************************************"        echo " High CPU consumed PID       :- " $PID        echo " High CPU consumed EAP Name  :- " $EAP        echo " High CPU consumed Thread ID :- " $ThreadID        echo -e "****************************************************************"        echo ""        echo " High CPU consumption Thread Stack Trace" echo"########################################################################"        echo ""        $JAVA_HOME/bin/jstack $PID | grep -A500 $ThreadID | grep -m1 "^$" -B 500        echo "" echo"########################################################################"   fi

Copy the above Code and Save this with .sh extension and run it with sufficient permissions.

Let see how this script will work

[jboss@middleware ~]$ ./High_CPU_Consumption_Threads.sh **************************************************************** High CPU consumed PID       :-  72520 High CPU consumed EAP Name  :-  Server-One High CPU consumed Thread ID :-  6d08 **************************************************************** High CPU consumption Thread Stack Trace ################################################################ "HttpManagementService-threads - 2288" #10447 prio=5 os_prio=0 tid=0x0000000000891000 nid=0x6d08 waiting on condition [0x00007f80ba35a000]    java.lang.Thread.State: TIMED_WAITING (parking)         at sun.misc.Unsafe.park(Native Method)         - parking to wait for  <0x00000000e0461778> (a java.util.concurrent.SynchronousQueue$TransferStack)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)         at org.jboss.threads.JBossThread.run(JBossThread.java:122) ################################################################

Running the script multiple times (or with watch, see below) will capture Busy thread in different places..

Hope this helps

Monitor JVM memory usage in WAS

Now, here is the Jython script designed to help monitor the “HeapSize”  which can let us able to proactively measure and take action for Outofmemoryerror issue.

Copy the Below Code and  Save this with .py extension and run it with wsadmin.

appServer=raw_input("Enter application server name: ") JVMName = AdminControl.completeObjectName('type=JVM,process='+ appServer +',*') JVMObject = AdminControl.makeObjectName(JVMName) perf = AdminControl.completeObjectName('type=Perf,process='+appServer+',*') perfObject = AdminControl.makeObjectName(perf) Obj = AdminControl.invoke_jmx(perfObject,"getStatsObject",[JVMObject,java.lang.Boolean('false')],['javax.management.ObjectName','java.lang.Boolean']) current = Obj.getStatistic('HeapSize').getCurrent() used = Obj.getStatistic('UsedMemory').getCount() usage = float(used)/float(current)*100 uptime = float(Obj.getStatistic('UpTime').getCount())/60/60/24 print "--------------------------------------------" print "ServerName      :", appServer print "uptime(in days) :",  int(uptime) print "--------------------------------------------" print "CurrentUsage    :", current print "Usedmemory      :", used print "Usage in Percent:", int(usage) print "--------------------------------------------"

Let see how this script will work

Here i saved above code as getHeapSize.py and run as below.

Syntax :   $WAS_HOME/bin/wsadmin.sh  -lang jython  -f getHeapSize.py

Remember that WebSphere allows you to set an Initial as well as a Maximum heap size - if they're not the same then the heap size could be anywhere between these two values.  The output above will show you the percentage use of the *current* heapsize, which may not be the maximum.  Some recommend setting these to the same value to remove the performance overhead of the JVM needed to allocate more memory as it grows the heap

How to rotate the proccess-controller.log, host-controller.log log files in EAP 6?

In this blog post, I am going to explain the log rotation of host-controller.log and process-controller.log

Issue: Every time the host controller starts is overwritten the logs:  host-controller.log, process-controller.log.

Resolution:

Define the PeriodicRotatingFileHandler in $JBOSS_HOME/domain/configuration/logging.properties as following:

# File handler configuration handler.BOOT_FILE=org.jboss.logmanager.handlers.PeriodicRotatingFileHandler handler.BOOT_FILE.level=DEBUG handler.BOOT_FILE.properties=autoFlush,append,fileName,suffix handler.BOOT_FILE.autoFlush=true handler.BOOT_FILE.append=true handler.BOOT_FILE.suffix=.yyyy-MM-dd handler.BOOT_FILE.fileName=${org.jboss.boot.log.file:boot.log} handler.BOOT_FILE.formatter=PATTERN

The suffix property is responsible for rotating the log files based on a time interval that is determined by the content of the string which is passed in to this property:

handler.BOOT_FILE.suffix=.yyyy-MM-dd

Also note that you need to add append and suffix to handler.BOOT_FILE.properties. (handler.BOOT_FILE.properties=autoFlush,fileName by default.)

Validation:

After a day, verify the $JBOSS_HOME/domain/log path and the logs should be rotated with suffix of date.

[jboss@middleware log]$ ls -ltr  $JBOSS_HOME/domain/log -rw-rw-r--. 1 jboss jboss   7542 Dec 27 23:50 process-controller.log.2017-12-23 -rw-rw-r--. 1 jboss jboss  14329 Dec 27 23:50 host-controller.log.2017-12-23 -rw-rw-r--. 1 jboss jboss   7572 Dec 28 21:11 process-controller.log -rw-rw-r--. 1 jboss jboss  19940 Dec 28 21:14 host-controller.log [jboss@middleware log]$ ls -ltr

EAP - logging.properties

You may have noticed that there is a logging.properties file in the  below directory

$JBOSS_HOME/standalone/configuration     --> standalone mode

$JBOSS_HOME/domain/servers/<SERVER_NAME>/data    --> Domain Mode

This logging configuration is used when the server boots up until the logging subsystem kicks in. If the logging subsystem is not included in your configuration, then this would act as the logging configuration for the entire server.

In the future this file may go away or be automatically generated from the logging subsystem. For most users this file should not be modified.

Note: If the logging.properties is not available during start there will be no logging until the logging subsystems kicks in, this is also the case if the configuration is damaged, the server might exit without any further message.

JBoss Deployment EAR Error on start-up - No content hash available

In this post, we are going to discuss the deployment problem that prevents JBoss Server to start properly.

 The below defined error usually occurs when we delete directory standalone\data or domain\data and then try to redeploy the same application war. The JBoss server unable to delete entries defined in standalone.xml/domain.xml with the same hash for the deployed application.

06:02:24,774 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("add") failed - address: ([("deployment" => "abc.ear")]) - failure description: "JBAS018717: No deployment content with hash 545ac26ed6a31ef41d749d5228669eb6858bb345 is available in the deployment content repository for deployment 'abc.ear'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart." 06:02:24,785 FATAL [org.jboss.as.server] (Controller Boot Thread) JBAS015957: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.

 The domain.xml looks like below:

In Deployments definition:

<deployment name="abc.ear" runtime-name="abc.ear">                <content sha1="545ac26ed6a31ef41d749d5228669eb6858bb345"/>         </deployment>

Server –group Definition:

<server-group name="abc-group" profile="abc">             <socket-binding-group ref="ha-sockets"/>             <deployments>                 <deployment name="abc.ear" runtime-name="abc.ear"/>                 <deployment name="ojdbc6.jar" runtime-name="ojdbc6.jar"/>             </deployments>        </server-group>

Removing the entries manually for the application having defined hash from domain.xml and then restart the DC and application server. 

mod_proxy || mod_jk || mod_cluster

mod_proxy

This proxy can either use HTTP or AJP for routing request from Apache to JBoss. The advantage of mod_proxy is that it is available as a built in module in Apache 2.x versions.

Advantage:

•          No need to compile and maintain an opposite module. Mod_proxy, mod_proxy_http, mod_proxy_ajp, mod_proxy_balancer is Apache 2.2+ standard integrated part,
•          You can use the HTTP, HTTPS and AJP protocols, even in the same balancer.

Inferiority:

•          Mod_proxy_ajp does not support more than 8K data packet,
•          Only the most basic load balancer,
•          Does not support domain model cluster(domain model clustering)

mod_jk

Mod_jk is based on the AJP protocol, mod_jk connector is an Apache module specifically written by Tomcat team. This module routes the Apache to JBoss request via the AJP (Apache Jserv Protocol).

Advantage:

•  The load balancer advanced,
•   Advanced node failure detection function,
•   Support for large-scale AJP packet

Inferiority:

•          The need for a separate maintenance of an independent module,

mod_cluster

mod_cluster is an httpd-based load balancer. Like mod_jk and mod_proxy, mod_cluster uses a communication channel to forward requests from httpd to one of a set of application server nodes. Unlike mod_jk and mod_proxy, mod_cluster leverages an additional connection between the application server nodes and httpd. The application server nodes use this connection to transmit server-side load balance factors and lifecycle events back to httpd via a custom set of HTTP methods, affectionately called the Mod-Cluster Management Protocol (MCMP). This additional feedback channel allows mod_cluster to offer a level of intelligence and granularity not found in other load balancing solutions.

Advantages:

•          Dynamic configuration of httpd workers

Traditional httpd-based load balancers require explicit configuration of the workers available to a proxy. In mod_cluster, the bulk of the proxy's configuration resides on the application servers. The set of proxies to which an application server will communicate is determined either by a static list or using dynamic discovery via the advertise mechanism. The application server relays lifecycle events (e.g. server startup/shutdown) to the proxies allowing them to effectively auto-configure themselves. Notably, the graceful shutdown of a server will not result in a failover response by a proxy, as is the case with traditional httpd-based load balancers.

•          Server-side load balance factor calculation

In contrast with traditional httpd-based load balancers, mod_cluster uses load balance factors calculated and provided by the application servers, rather than computing these in the proxy. Consequently, mod_cluster offers a more robust and accurate set of load metrics than is available from the proxy. (see Load Metrics for more)

•          Fine grained web-app lifecycle control

Traditional httpd-based load balancers do not handle web application undeployments particularly well. From the proxy's perspective requests to an undeployed web application are indistinguishable from a request for an non-existent resource, and will result in 404 errors. In mod_cluster, each server forwards any web application context lifecycle events (e.g. web-app deploy/undeploy) to the proxy informing it to start/stop routing requests for a given context to that server.

• AJP is optional

Unlike mod_jk, mod_cluster does not require AJP. httpd connections to application server nodes can use HTTP, HTTPS, or AJP.

Monitor DataSource connection pool statistics in JBoss EAP

In this post we focus on DataSource connection pool statistics for your application with

The Command Line Interface (CLI).

In EAP 6.3 and later, DataSource statistics need to be explicitly enabled as they are disabled by default to minimize performance impact.

By using below command you can enable DataSource connection pool statistics. after that restart the servers to take effect.

Syntax: /profile=<Profilename>/subsystem=datasources/data-source=< data-source Name>:write-attribute (name=statistics-enabled,value=true)

[domain@192.168.1.12:9999 /] /profile=better/subsystem=datasources/data-source=BetterDS:write-attribute(name=statistics-enabled, value=true) {     "outcome" => "success",     "result" => undefined,     "server-groups" => undefined }

Then check the DataSource statistics:

To view the statistics via management console, migrate to the Runtime Tab. Go to Subsystems --> Datasources. Click on the data-source whose statistics you need to view.

CLI example in Domain mode:

Core Statistics :

Syntax: host=<HC name>/server=<AS name>/subsystem=datasources/data-data=<data source name>/statistics=pool:read-resource(recursive=true, include-runtime=true)

[domain@192.168.1.12:9999 /]/host=slave01/server=better-as1/subsystem=datasources/data-source=BetterDS/statistics=pool:read-resource(recursive=true, include-runtime=true) {     "outcome" => "success",     "result" => {         "ActiveCount" => "1",         "AvailableCount" => "20",         "AverageBlockingTime" => "0",         "AverageCreationTime" => "2583",         "CreatedCount" => "2",         "DestroyedCount" => "1",         "InUseCount" => "0",         "MaxCreationTime" => "5092",         "MaxUsedCount" => "1",         "MaxWaitCount" => "0",         "MaxWaitTime" => "0",         "TimedOut" => "1",         "TotalBlockingTime" => "0",         "TotalCreationTime" => "5166",         "statistics-enabled" => true     } }

 The following table contains a list of the supported datasource core statistics:

Name	Description
ActiveCount	The number of active connections. Each of the connections is either in use by an application or available in the pool
AvailableCount	The number of available connections in the pool.
AverageBlockingTime	The average time spent blocking on obtaining an exclusive lock on the pool. The value is in milliseconds.
AverageCreationTime	The average time spent creating a connection. The value is in milliseconds.
CreatedCount	The number of connections created.
DestroyedCount	The number of connections destroyed.
InUseCount	The number of connections currently in use.
MaxCreationTime	The maximum time it took to create a connection. The value is in milliseconds.
MaxUsedCount	The maximum number of connections used.
MaxWaitCount	The maximum number of requests waiting for a connection at the same time.
MaxWaitTime	The maximum time spent waiting for an exclusive lock on the pool.
TimedOut	The number of timed out connections.
TotalBlockingTime	The total time spent waiting for an exclusive lock on the pool. The value is in milliseconds.
TotalCreationTime	The total time spent creating connections. The value is in milliseconds.
WaitCount	The number of requests that had to wait for a connection.

JDBC Statistics

Syntax: host=<HC name>/server=<AS name>/subsystem=datasources/data-data=<data source name>/statistics=jdbc:read-resource(recursive=true, include-runtime=true)

[domain@192.168.1.12:9999 /]/host=slave01/server=better-as1/subsystem=datasources/data-source=BetterDS/statistics=jdbc:read-resource(recursive=true, include-runtime=true) {     "outcome" => "success",     "result" => {         "PreparedStatementCacheAccessCount" => "0",         "PreparedStatementCacheAddCount" => "0",         "PreparedStatementCacheCurrentSize" => "0",         "PreparedStatementCacheDeleteCount" => "0",         "PreparedStatementCacheHitCount" => "0",         "PreparedStatementCacheMissCount" => "0",         "statistics-enabled" => true     } }

The following table contains a list of the supported datasource JDBC statistics:

Name	Description
PreparedStatementCacheAccessCount	The number of times that the statement cache was accessed.
PreparedStatementCacheAddCount	The number of statements added to the statement cache.
PreparedStatementCacheCurrentSize	The number of prepared and callable statements currently cached in the statement cache.
PreparedStatementCacheDeleteCount	The number of statements discarded from the cache.
PreparedStatementCacheHitCount	The number of times that statements from the cache were used.
PreparedStatementCacheMissCount	The number of times that a statement request could not be satisfied with a statement from the cache.

please subscribe my blog  and provide the feedback on this article/blog to improve the articles.