Hardware inventory fails (after upgrade?)


#1

Hi,

On adding some Cisco environment checks and re-discovering that node, I noticed the hardware info was gone.
The periodical hardware discovery event reports success but there’s nothing on the webpage.

On browsing the provision log I see the hardware inventory scanning of that node but the subsequent insert into the database failing with a
null value in column "hwentityid" violates not-null constraint”.

Needless to say there is probably something rotten in the database.

Now I recently did an -unintended- upgrade from 18.x (I think) to 23.x and my guess is something went haywire there.

So my questions are:

  1. could somebody hazard a guess as to what caused this/how to resolve this?
  2. shouldn’t this insert-failure lead to a failed hardware discovery event?

Please find below an excerpt from provisiond.log of the error.

TIA,
Tom

2019-03-08 03:06:22,450 DEBUG [DefaultUDPTransportMapping_0.0.0.0/0] o.s.Snmp: Running pending async request with handle PduHandle[1213990186] and retry count left 2
2019-03-08 03:06:22,451 WARN  [pool-16-thread-1] o.h.u.JDBCExceptionReporter: SQL Error: 0, SQLState: 23502
2019-03-08 03:06:22,451 ERROR [pool-16-thread-1] o.h.u.JDBCExceptionReporter: ERROR: null value in column "hwentityid" violates not-null constraint
  Detail: Failing row contains (4629133, null, 0, .1.3.6.1.2.1.2.2.1.1.2).
2019-03-08 03:06:22,452 ERROR [pool-16-thread-1] o.o.n.p.SimpleQueuedProvisioningAdapter: Unexpected exception during node operation: could not insert: [org.opennms.netmgt.model.OnmsHwEntityAlias]; SQL [insert into hwEntityAlias (hwEnt
ityId, index, oid, id) values (?, ?, ?, ?)]; constraint [hwentityid]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [org.opennms.netmgt.model.OnmsHwEntityAlias]
org.springframework.dao.DataIntegrityViolationException: could not insert: [org.opennms.netmgt.model.OnmsHwEntityAlias]; SQL [insert into hwEntityAlias (hwEntityId, index, oid, id) values (?, ?, ?, ?)]; constraint [hwentityid]; nested 
exception is org.hibernate.exception.ConstraintViolationException: could not insert: [org.opennms.netmgt.model.OnmsHwEntityAlias]
        at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:659) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
        at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:800) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:670) ~[org.apache.servicemix.bundles.spring-orm-4.2.9.RELEASE_1.jar:?]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150) ~[org.apache.servicemix.bundles.spring-tx-4.2.9.RELEASE_1.jar:?]
        at org.opennms.netmgt.provision.SimplerQueuedProvisioningAdapter.processPendingOperationForNode(SimplerQueuedProvisioningAdapter.java:125) ~[opennms-dns-provisioning-adapter-23.0.3.jar:?]
        at org.opennms.netmgt.provision.SimpleQueuedProvisioningAdapter$AdapterOperation.run(SimpleQueuedProvisioningAdapter.java:423) [opennms-dns-provisioning-adapter-23.0.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

#2

This sounds to me like you still need to run $OPENNMS_HOME/bin/install -dis to update the database schema to match the running version.


#3

In rel 23, scanning of .1.3.6.1.2.1.47.1.3.2.1.2 MIB was added.
It appears that the scan in your environment is being parsed to contain no Index value.

Can you confirm this by checking for log entries like the one below:
‘result from entAliasMappingTable: found entry {} index: {} oid: {}’

If you can retrieve that mib from one of the devices we can look at what might be going wrong.
Thanks


#4

I’m also experiencing this in my test environment. I’ve set up a virtual Cisco 72xx and tried running the hardware discovery over it - after enabling debug in the logs, I get the following:

2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 14 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.1
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 15 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.2
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 29 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.3
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 31 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.4
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 32 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.5
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 33 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.6
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 34 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.7
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 36 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.8
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 37 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.9
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 38 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.10
2019-03-06 18:48:11,958 DEBUG [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: result from entAliasMappingTable: found entry 39 index: 0 oid: .1.3.6.1.2.1.2.2.1.1.11
2019-03-06 18:48:11,961 INFO  [pool-16-thread-1] o.o.n.p.SnmpHardwareInventoryProvisioningAdapter: Saving hardware inventory for nodeId 13
2019-03-06 18:48:11,996 DEBUG [pool-16-thread-1] o.o.n.e.EventIpcManagerDefaultImpl: sending: {"_events":{"_eventList":[{"_creationTime":Wed Mar 06 18:48:11 CST 2019,"_uei":"uei.opennms.org/inter
nal/discovery/hardwareInventorySuccessful","_source":"Provisiond.SnmpHardwareInventoryProvisioningAdapter","_nodeid":13,"_time":Wed Mar 06 18:48:11 CST 2019,"_interfaceAddress":/192.168.1.238}]}}
2019-03-06 18:48:12,020 WARN  [pool-16-thread-1] o.h.u.JDBCExceptionReporter: SQL Error: 0, SQLState: 23502
2019-03-06 18:48:12,020 ERROR [pool-16-thread-1] o.h.u.JDBCExceptionReporter: ERROR: null value in column "hwentityid" violates not-null constraint
 Detail: Failing row contains (1452, null, 0, .1.3.6.1.2.1.2.2.1.1.1).
2019-03-06 18:48:12,022 ERROR [pool-16-thread-1] o.o.n.p.SimpleQueuedProvisioningAdapter: Unexpected exception during node operation: could not insert: [org.opennms.netmgt.model.OnmsHwEntityAlias
]; SQL [insert into hwEntityAlias (hwEntityId, index, oid, id) values (?, ?, ?, ?)]; constraint [hwentityid]; nested exception is org.hibernate.exception.ConstraintViolationException: could not i
nsert: [org.opennms.netmgt.model.OnmsHwEntityAlias]

#5

Also, although I am pretty sure that I already had, I did re-run the install script, to make sure that all the updates to the database had been made, then re-ran the scan - the same error occurred in the logs


#6

Hi guys,

Thanks for some excellent suggestions, but alas to no avail.

Stopped opennms, ran install -dis, started opennms, deleted node and started rediscovery again, results did not change. :unamused:

Grepping for the entAliasMappingTable in the logs only gave me entries that contained sensible values (albeit that index values were 0, but that should be normal).

So, we need to search elsewhere for a possible cause.
Again, any suggestions on other avenues to explore will be most welcome.

Cheers,
Tom.


#7

Yeah, it looks like a problem in the codepath - I’m looking into it and will get back to this thread with something soon.


#8

Tracking this with the following JIRA: https://issues.opennms.org/browse/NMS-10606


#9

Hi,

Thanks for the swift action.
Given that the JIRA issue is solved and closed am I correct in assuming that this fix will only appear in V.24 of OpenNMS?

TIA,
Tom.