Do not display symptoms on published web page
When IBM Cognos BI is configured to authenticate against a Windows based OpenLDAP server of version 2.4.x browsing the LDAP namespace or searching it will lead to CAM-AAA-0045 errors:
CAM-AAA-0045 The call to the LDAP function 'doLDAPSearch_s/LdapAPIWrapper_ldap_search_ext_s' returned with the error code: '18'.Inappropriate matching.
It's impossible to work around and thus this is preventing browsing/searching. Authentication is working fine though.
The issue is likely caused by a OpenLDAP issue.
OpenLDAP.org the creators of OpenLDAP don't provide binary builds of their product, they only publish the source code. Several 3rd parties provide pre-compiled binaries for various platforms which contain the OpenLDAP and some database back-end though. When a binary package is built by some 3rd party optional "OpenLDAP overlays" which are modules (LDAP controls) adding enhanced functionality to the OpenLDAP server can be added. Thus binary packages may differ depending on the creator.
The most common recent Windows binaries (2.4.x) which come up in internet searches first originate from a 3rd party which has added a "server side sorting" overlay. This feature allows OpenLDAP to sort the result set of a search on the server rather than the client.
The IBM Cognos BI LDAP provider detects the presence of this LDAP control upon initialization of the LDAP namespace and uses it to improve search performance. Some Windows binaries of OpenLDAP show an issue when requested to sort ou attributes. The LDAP control returns an error code to the client which causes IBM Cognos BI to cancel the request.
This only occurs with Windows binary builds of OpenLDAP which contain the LDAP server side control (OID 1.2.840.1135188.8.131.523).
The environment of IBM Cognos 10 is not relevant.
Diagnosing the problem
If unsure whether an instance of OpenLDAP offers the server side LDAP control, attach to the RootDSE of the OpenLDAP instance using some LDAP browser and look at the supportedControl attribute. If one of it's values matches OID 1.2.840.1135184.108.40.2063 it does offer that control.
Use a binary of OpenLDAP which does not have the server side sort control or use a different LDAP V3 compliant LDAP