Minion problem. Need to restart it from time to time

Hello hello!

Found out that we have problems with one of our minions.
Every-time it hangs we need to kill the Java-process and start it again with “/usr/share/minion/bin/start”

4GB RAM on minion-host
from the minion: ps -ef | grep java : show that it starts with the following:
/usr/bin/java -Xms128M -Xmx512M -XX:+UnlockDiagnosticVMOptions





In the minion log i see a lot of these INFO.
| INFO | pool-14-thread-4570 | SinglePingResponseCallback | 353 - org.opennms.opennms-icmp-api - 25.2.1 | waiting for ping to /x.x.x.x
| INFO | pool-14-thread-4554 | SinglePingResponseCallback | 353 - org.opennms.opennms-icmp-api - 25.2.1 | finished waiting for ping to /x.x.x.x

I changed everything from INFO to WARN. See if that helps.

Should i increase the mem on the host first and then tweak to this?
from:

# minimum amount of memory for the Karaf container to allocate
# export JAVA_MIN_MEM=128M

# maximum amount of memory for the Karaf container to allocate
# export JAVA_MAX_MEM=512M

to:

# minimum amount of memory for the Karaf container to allocate
export JAVA_MIN_MEM=1024M

# maximum amount of memory for the Karaf container to allocate
export JAVA_MAX_MEM=2048M

opennms - ver 26.1.0

The default Xmx value is not suitable for a production use case.

1 Like

Hi!

Just to be clear. You suggest that i increase the mem on the host and also change to the following:
This is not on Master. Its on the minion.

> # minimum amount of memory for the Karaf container to allocate
>  export JAVA_MIN_MEM=1024M
> 
> # maximum amount of memory for the Karaf container to allocate
>  export JAVA_MAX_MEM=2048M
> 
> 
> # other options to pass to Java on startup
>  export JAVA_OPTS="-Xms${JAVA_MIN_MEM} -Xmx${JAVA_MAX_MEM} -XX:+UnlockDiagnosticVMOptions"

Yes, my suggestion is to increase JAVA_MAX_MEM . 2G is a good place to start, depending on the workload, more may be necessary.

1 Like

Now it died again this morning. Here are the logs:
Will do the change ASAP. :slight_smile:

 Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618965]
2021-06-17T06:38:00,598 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:38:00,598 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:38:00,597 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:38:00,597 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-618999 on ExchangeId: ID-xxxx01-41067-1623844649067-0-618927). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618927]
2021-06-17T06:38:28,739 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619003 on ExchangeId: ID-xxxx01-41067-1623844649067-0-618946). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618946]
2021-06-17T06:38:28,241 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:38:30,728 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619004 on ExchangeId: ID-xxxx01-41067-1623844649067-0-618945). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618945]
2021-06-17T06:38:18,304 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619001 on ExchangeId: ID-xxxx01-41067-1623844649067-0-618950). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618950]
2021-06-17T06:38:13,766 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:38:30,717 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:38:31,233 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619005 on ExchangeId: ID-xxxx01-41067-1623844649067-0-618940). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-618940]
2021-06-17T06:39:02,755 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:39:05,209 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:39:07,702 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:39:36,327 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:39:22,914 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:39:22,916 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:39:27,353 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:40:27,486 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:40:18,153 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:40:29,917 | ERROR | Timer-379095     | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:40:32,358 | ERROR | Timer-379105     | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:40:36,262 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2005 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2006). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2006]
2021-06-17T06:40:54,016 | ERROR | Timer-379095     | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619125 on ExchangeId: ID-xxxx01-41067-1623844649067-0-619098). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-619098]
2021-06-17T06:40:54,016 | ERROR | Timer-379105     | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619126 on ExchangeId: ID-xxxx01-41067-1623844649067-0-619109). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-619109]
2021-06-17T06:39:56,610 | ERROR | AggregatorFlush-Syslog | Aggregator                       | 274 - org.opennms.core.ipc.sink.common - 25.2.1 | An error occurred while flushing one or more aggregates in module 'Syslog'.
2021-06-17T06:39:23,415 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:39:25,403 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619122 on ExchangeId: ID-xxxx01-41067-1623844649067-0-619077). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-619077]
2021-06-17T06:41:10,427 | ERROR | pool-10-thread-28356 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.poller.client.rpc.PollerClientRpcModule@2b3a31fa failed.
2021-06-17T06:41:54,673 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2007 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2008). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2008]
2021-06-17T06:41:54,671 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:41:54,673 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:41:52,145 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
2021-06-17T06:41:50,695 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:41:17,725 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:41:39,752 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:43:35,742 | ERROR | AggregatorFlush-Syslog | Aggregator                       | 274 - org.opennms.core.ipc.sink.common - 25.2.1 | An error occurred while flushing one or more aggregates in module 'Syslog'.
2021-06-17T06:43:23,885 | ERROR | fileinstall-/etc/minion | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | In main loop, we have serious trouble
2021-06-17T06:42:39,833 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:42:37,394 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:42:00,180 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:45:30,985 | ERROR | ActiveMQ Task-8  | TaskRunnerFactory                | 171 - org.apache.activemq.activemq-osgi - 5.14.5 | Error in thread 'ActiveMQ Task-8'
2021-06-17T06:45:18,366 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2009 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2010). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2010]
2021-06-17T06:45:08,544 | ERROR | ActiveMQ Task-7  | TaskRunnerFactory                | 171 - org.apache.activemq.activemq-osgi - 5.14.5 | Error in thread 'ActiveMQ Task-7'
2021-06-17T06:45:01,465 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:44:43,217 | ERROR | AggregatorFlush-Syslog | Aggregator                       | 274 - org.opennms.core.ipc.sink.common - 25.2.1 | An error occurred while flushing one or more aggregates in module 'Syslog'.
2021-06-17T06:44:28,206 | ERROR | JNA-ICMP-6817-Timeout-Processor | RequestTracker                   | 336 - org.opennms.lib.tracker - 0.7.0 | Unexpected exception on Thread Thread[JNA-ICMP-6817-Timeout-Processor,5,main]!
2021-06-17T06:44:15,056 | ERROR | fileinstall-/usr/share/minion/deploy | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | In main loop, we have serious trouble
2021-06-17T06:44:15,544 | ERROR | Timer-379130     | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:44:14,054 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:45:51,211 | ERROR | Timer-379130     | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-0-619194 on ExchangeId: ID-xxxx01-41067-1623844649067-0-619140). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-0-619140]
2021-06-17T06:44:15,548 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:45:52,786 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2011 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2012). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2012]
2021-06-17T06:45:55,235 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2013 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2014). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2014]
2021-06-17T06:45:57,810 | ERROR | ActiveMQ Task-1  | TaskRunnerFactory                | 171 - org.apache.activemq.activemq-osgi - 5.14.5 | Error in thread 'ActiveMQ Task-1'
2021-06-17T06:46:05,623 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2017 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2018). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2018]
2021-06-17T06:49:12,522 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:49:12,522 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | Marshalling a response in RPC module org.opennms.netmgt.snmp.proxy.common.SnmpProxyRpcModule@1aa147f1 failed.
2021-06-17T06:49:34,392 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:49:32,007 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:49:36,767 | ERROR | Camel (trapdListenerContext) thread #90 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-xxxx01-41067-1623844649067-2-2025 on ExchangeId: ID-xxxx01-41067-1623844649067-2-2026). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-xxxx01-41067-1623844649067-2-2026]
2021-06-17T06:49:39,161 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:49:58,907 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:49:54,121 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:50:21,690 | ERROR | fileinstall-/etc/minion | fileinstall                      | 10 - org.apache.felix.fileinstall - 3.6.4 | In main loop, we have serious trouble
2021-06-17T06:50:12,962 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:50:10,467 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-17T06:50:15,446 | ERROR | DefaultUDPTransportMapping_0.0.0.0/0 | MessageDispatcherImpl            | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | java.lang.OutOfMemoryError: GC overhead limit exceeded

Looks like this now:
ps -ef | grep java
root 14984 14931 7 12:48 ? 00:00:51 /usr/lib/jvm/java-8-oracle/bin/java -Xms1024M -Xmx2048M -

But in the log below it complans about the change. IGNORING???

minion.service: Ignoring invalid environment assignment 
minion.service: Ignoring invalid environment assignment 
minion.service: Ignoring invalid environment assignment

And i had to start it with “systemctl start minion”. When using the script /usr/share/minion/bin/start - it doesen’t take the config. only using the default.

minion.service - The OpenNMS Minion distributed client.
   Loaded: loaded (/lib/systemd/system/minion.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-06-17 12:48:48 CEST; 7s ago
  Process: 14693 ExecStart=/etc/init.d/minion start (code=exited, status=0/SUCCESS)
 Main PID: 14984 (java)
    Tasks: 121 (limit: 4915)
   CGroup: /system.slice/minion.service
           ├─14931 /bin/sh /usr/share/minion/bin/karaf server
           └─14984 /usr/lib/jvm/java-8-oracle/bin/java -Xms1024M -Xmx2048M -XX:+UnlockDiagnosticVMOptions -Dcom.sun.management.jmxremote -Xms1024M -Xmx2048M -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemor

Jun 17 12:48:43 min systemd[1]: minion.service: Ignoring invalid environment assignment 'export JAVA_MIN_MEM=1024M': /etc/default/minion
Jun 17 12:48:43 min systemd[1]: minion.service: Ignoring invalid environment assignment 'export JAVA_MAX_MEM=2048M': /etc/default/minion
Jun 17 12:48:43 min systemd[1]: minion.service: Ignoring invalid environment assignment 'export JAVA_OPTS=-Xms${JAVA_MIN_MEM} -Xmx${JAVA_MAX_MEM} -XX:+UnlockDiagnosticVMOptions': /etc/default/minion
Jun 17 12:48:43 min systemd[1]: Starting The OpenNMS Minion distributed client....
Jun 17 12:48:48 min systemd[1]: minion.service: Supervising process 14984 which is not our child. We'll most likely not notice when it exits.
Jun 17 12:48:48 min systemd[1]: Started The OpenNMS Minion distributed client..
~

Hi!

We have increased the memory. But we still recieve this ERROR:

I will now try to change it to -Xms1024M -Xmx8192M

2021-06-24T08:56:02,385 | ERROR | Camel (trapdListenerContext) thread #92 - timer://restTrapConfigurationTimer | DefaultErrorHandler              | 174 - org.apache.camel.camel-core - 2.19.1 | Failed delivery for (MessageId: ID-imonmin01-45085-1623926181915-3-19719 on ExchangeId: ID-imonmin01-45085-1623926181915-3-19720). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-imonmin01-45085-1623926181915-3-19720]

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[restClientToTrapdC] [restClientToTrapdC] [timer://restTrapConfigurationTimer?fixedRate=true&period=60000                ] [     35543]
[restClientToTrapdC] [bean1             ] [bean[ref:restClient method:getSnmpV3Users]                                    ] [     35543]

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-imonmin01-45085-1623926181915-3-19720]
        at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1790) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:350) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:266) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:178) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [174:org.apache.camel.camel-core:2.19.1]
        at java.util.TimerThread.mainLoop(Timer.java:555) [?:?]
        at java.util.TimerThread.run(Timer.java:505) [?:?]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-24T08:56:02,385 | WARN  | Camel (trapdListenerContext) thread #92 - timer://restTrapConfigurationTimer | TimerConsumer                    | 174 - org.apache.camel.camel-core - 2.19.1 | Error processing exchange. Exchange[ID-imonmin01-45085-1623926181915-3-19720]. Caused by: [org.apache.camel.CamelExecutionException - Exception occurred during execution on the exchange: Exchange[ID-imonmin01-45085-1623926181915-3-19720]]
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-imonmin01-45085-1623926181915-3-19720]
        at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1790) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:350) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:266) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:178) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) ~[174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197) [174:org.apache.camel.camel-core:2.19.1]
        at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79) [174:org.apache.camel.camel-core:2.19.1]
        at java.util.TimerThread.mainLoop(Timer.java:555) [?:?]
        at java.util.TimerThread.run(Timer.java:505) [?:?]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded



2021-06-24T08:57:33,857 | FATAL | Timer-3829911    | Snmp                             | 282 - org.opennms.core.snmp.implementations.snmp4j - 25.2.1 | Failed to process pending request PduHandle[984574185] because GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-06-24T08:57:33,862 | WARN  | Timer-3829876    | CamelRpcServerProcessor          | 266 - org.opennms.core.ipc.rpc.camel - 25.2.1 | An error occured while executing a call in SNMP.

Now the process looks like this:
root 909 855 9 09:14 ? 00:02:18 /usr/lib/jvm/java-8-oracle/bin/java -Xms1024M -Xmx8192M -XX:+UnlockDiagnosticVMOptions -Dcom.sun.management.jmxremote -Xms1024M -Xmx8192M -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/usr/lib/jvm/java-8-oracle/jre/lib/endorsed:/usr/lib/jvm/java-8-oracle/lib/endorsed:/usr/share/minion/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/java-8-oracle/jre/lib/ext:/usr/lib/jvm/java-8-oracle/lib/ext:/usr/share/minion/lib/ext -Dkaraf.instances=/usr/share/minion/instances -Dkaraf.home=/usr/share/minion -Dkaraf.base=/usr/share/minion -Dkaraf.data=/usr/share/minion/data -Dkaraf.etc=/usr/share/minion/etc -Dkaraf.log=/usr/share/minion/data/log -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir=/usr/share/minion/data/tmp -Djava.util.logging.config.file=/usr/share/minion/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /usr/share/minion/lib/boot/jna-4.4.0.jar:/usr/share/minion/lib/boot/jna-platform-4.4.0.jar:/usr/share/minion/lib/boot/mina-core-2.0.16.jar:/usr/share/minion/lib/boot/org.apache.karaf.diagnostic.boot-4.2.6.jar:/usr/share/minion/lib/boot/org.apache.karaf.jaas.boot-4.2.6.jar:/usr/share/minion/lib/boot/org.apache.karaf.main-4.2.6.jar:/usr/share/minion/lib/boot/org.apache.karaf.specs.activator-4.2.6.jar:/usr/share/minion/lib/boot/org.osgi.core-6.0.0.jar org.apache.karaf.main.Main