Friday, October 5, 2012

SBL-SCM-00018: Could not open connection to Siebel Gateway



Applies to:


Siebel System Software - Version: 7.5.3.11 [16199] and later   [Release: V7 and later ]
z*OBSOLETE: Microsoft Windows 2000

Product Release: V7 (Enterprise)

Version: 7.5.3.11 [16199]

Database: Microsoft SQL Server 2000 SP 2

Application Server OS: Microsoft Windows 2000 Server SP 2

Database Server OS: Microsoft Windows 2000 Server SP 2



This document was previously published as Siebel SR 38-2846624411.



Symptoms


SBL-DAT-00222, SBL-SDM-00109, SBL-SCM-00018

We moved our Siebel Test Environment Server machines from one domain
to another. Consequently, the server machine names were changed.

When
the servers had been commissioned on the new domain we uninstalled
Siebel from the Siebel Application Server and reinstalled it using the
new hostnames where needed.

Now when we access the Server Admin screens either through the Web Client or Dedicated Client we get the following error.

An
error has occurred creating business component 'Server Server' used by
business object 'Server Admin'. Please ask your systems administrator to
check your application configuration.(SBL-DAT-00222)<p>Error from
Admin Server, SBL-SCM-00018: Could not open connection to Siebel
Gateway configuration store (LNGWOKAPPT030:2320)(SBL-SDM-00109)<p>

Other screens in the application work as expected when using either client type.

When we use srvrmgr utility on the command line we get the error

Fatal error (3900018): Could not open connection to Siebel Gateway configuration store (%1:%2), exiting...

We used the same Siebel Enterprise Server and Siebel Server Names as before only the server machine names have changed.

We can ping the app server lngwokappt030 from the web server lngwokiist030 and desktop PCs running the clients.

Please help us diagnose.



Cause


EnableFQDN parameter in the Siebel application object manager cfg file had not been set.


Solution





For the benefit of other users:

The customer found that the EnableFQDN parameter in the Siebel application object manager cfg file had not been set.

After
setting this parameter and restarting the siebel server and gateway
services, it is now possible to access srvrmgr and server administration
screens.

Siebel Technical Support recommends the customer uninstall the Report Server from this machine based on Bookshelf > Siebel
Reports Administration Guide > Chapter 1 “Installing the Siebel
Reports Server for Microsoft Windows” > Section “Siebel Reports
Server Installation
” > Oracle also recommends that Actuate
e.Reporting Server not be installed on a machine where other Siebel
products reside (except for Siebel Report Server Access)

Customer is recommended to install the Report Server in a separate machine following the instructions in Bookshelf.












Applies to:


Siebel eCommunications - Version: 8.0 [20405] to 8.1.1 [21112] - Release: V8 to V8
Information in this document applies to any platform.



Symptoms




A new Siebel server installation does not start up. The services gets
started and stop within two minutes. Siebel gateway services can be
started fine.



ERROR:

------

No error is displayed for the Administration/user



Following errors are found in the log files:

NameSrvr.log

SBL-SEC-10018: [DataDirect][ODBC Oracle driver][Oracle]ORA-12154: TNS:could not resolve the connect identifier specified

SBL-SEC-10007: The password you have entered is not correct. Please enter your password again



SiebSrvr.log



SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store

SBL-SVR-00005: Stale or invalid Task handleScfEventLog

SCFMessageFacility::s_pSCFMsgFacLock is null and hence the SCFMessageFacility cannot be initializedIPCLog

SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store



ENVIRONMENT:

------------

Siebel application version 8.1.1 on Windows 2003, Oracle Database



STEPS:

------

starts the gateway service

starts the siebel service




Cause




The issue is caused by the following setup:

- there was two Oracle clients installed on the server machine : Oracle 10.2.0 and Oracle 10.1.10

- tns entries was incorrect setup




Solution


For the benefit of other readers:


To implement the solution follow the steps:



1) if you have more than one Oracle client installed, check that the
Oracle environment variables is pointing to the Oracle client that you
want to use

2) for this Oracle client, check on tnsnames.ora for the tns alias

3) ensure that you can connect fine using sql*plus through this tns alias

4) if not, work with your dba to be able to connect fine

5) once you can connect from tns alias, check that Siebel ODBC also
works fine. For this go to ODBC Data Source, System DSN tab, double
click on the odbc created by the Siebel installation



In this case, after adjusting the tns alias from tnsnames.ora Siebel Server came up fine.
 










Applies to:


Siebel Partner Portal - Version: 8.1 [21039] and later   [Release: V8 and later ]
Siebel System Software - Version: 8.1 [21039] and later    [Release: V8 and later]
Information in this document applies to any platform.



Symptoms


Unable to connect to Siebel Server Manager version 8.1 or any Siebel
application on Oracle Enterprise Linux 4.0 after attempting to delete
archived logfiles and thereafter regenerating the Gateway service
definition file (SVC.GTWYNS.) because the Siebel server services failed
to respond (start/stop). The following error message is displayed:




SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (%1).


where %1 is of the form Host:Port


Cause


The Gateway service definition file (SVC.GTWYNS.) had been generated with an incorrect set of parameters using the utility SIEBCTL. This has been proven by comparing the definition file from a working environment.


Solution


If a working backup of the service definition file can not be identified then run the utility SIEBCTL (specific to UNIX implementations in this example) given a root installation directory of say /siebel as follows:



  1. Source the Gateway installation


    cd /siebel/gtwysrvr
    . ./siebenv.sh


  2. Stop the Gateway service


    stop_ns


  3. Backup existing SVC.GTWYNS. service definition file


    cp sys/svc.gtwyns. sys/svc.gtwyns.backup


  4. Run the following command to delete the existing service definition file


    siebctl -S gtwyns -d


  5. Run the following command to add a new service definition file


    siebctl -S gtwyns -a -g "/f /siebel/gtwysrvr/sys/siebns.dat /t 2320 /c /siebel/gtwysrvr/bin/gateway.cfg"


  6. Start the Gateway service


    start_ns



 




Applies to:


Siebel Finance Service - Version: 8.1.1.6 [21233] and later   [Release: V8 and later ]
Information in this document applies to any platform.



Symptoms



On : 8.1.1.6 [21233] version, Installation



When attempting to start 6 newly added Siebel Servers to an existing environment, the following errors occur.



SiebSrvr.log
SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (cvnfw8ga:2320)

NameSrvr.log

[DataDirect][ODBC Oracle driver][Oracle]ORA-12516: TNS:listener could not find available handler with matching protocol stack
SBL-SEC-10018:
[DataDirect][ODBC Oracle driver][Oracle]ORA-12516: TNS:listener could
not find available handler with matching protocol stack
SBL-SEC-10007: The password you have entered is not correct. Please enter your password again.


Cause


Number of Processes parameter too small at the Oracle db server settings






Solution


Increase the pfile or spfile setting for PROCESSES


References


NOTE:240710.1 - Intermittent TNS-12516 or TNS-12519 Errors When Connecting Via Oracle Net










Applies to:


Siebel CRM - Version 8.1.1 [21112] and later
Information in this document applies to any platform.

***Checked for relevance on 10-SEP-2012***


Description


With Siebel CRM version 8.1.1 it can happen that the gateway service
is unresponsive under high load. In such a scenario, no more client
connections from server manager can be established or an entire siebel
server node and additional application object manager (AOM) processes
can not be started.



Occurrence


 You might run into this behavior when you run Siebel CRM version
8.1.1 or above and run an enterprise with more than 10 defined
application servers or when enterprise configuration updates to the
siebns.dat file are made. You are observing slowness in enterprise
startup in general.



Symptoms


Once it has been verified that the gateway service is up and running,
the database can be accessed and the correct passwords are provided, if
you then get one of the below error messages you might be facing the
issue discussed here:



 a)   The srvrmgr command needs a very
long time to connect to a single server using the /s switch or returns
the following error after a while in srvrmgr.log file:


NameServerLayerLog      Error   1    000000024f5f0132:0      2012-03-13 10:12:21     Unable to connect to the gateway server.

GenericLog      GenericError    1       000000024f5f0132:0      2012-03-13 10:12:21     NSC - ErrCode 5009  SysErr 0 

GenericLog      GenericError    1       000000024f5f0132:0     
2012-03-13 10:12:21     (scmnsclnt.cpp (135) err=2555922 sys=2)
SBL-SCM-00018: Could not open connection to Siebel Gateway configuration
store (localhost:2320).


b)   An Object manager process is
terminating during startup. When running the srvrmgr command "list procs
for comp <compname>", processes are stuck in the "Starting" state
and never transition to a state of "Running" or "Online"


c)   The siebsvc server  scheduler process
service is exiting, no processes are started and in the Siebsrvr.log
file the following error can be found:


NameServerLayerLog      Error   1       0000fb2e4f7100de:0      2012-03-27 08:56:09     Unable to connect to the gateway server.

GenericLog      GenericError    1       0000fb2e4f7100de:0      2012-03-27 08:56:09     NSS - ErrCode 5009  SysErr 0

GenericLog      GenericError    1       0000fb2e4f7100de:0     
2012-03-27 08:56:09     (scisvc.cpp (1391) err=1311765 sys=0)
SBL-SVR-01045: No components are configured.

GenericLog      GenericError    1       0000fb2e4f7100de:0     
2012-03-27 08:56:19     (siebsvc.cpp (221) err=2555922 sys=0)
SBL-SCM-00018: Could not open connection to Siebel Gateway configuration
store ((null):(null)).


d) The siebns.dat file is recreated
multiple times per minute. You can observe a temporary file called
siebns.dat.new frequently listed in the folder where siebns.dat is
located.


e) You are also running into this issue if
none of the above errors occur but you notice increased startup times
when adding servers to the enterprise.

   






Workaround


The following three bugs have been created to address this behavior:



  • Bug 12561457: GATEWAY SERVER CONSUMING HIGH CPU.



This has been fixed in Fixpack 8.1.1.8


  • Bug 13827461: SIEBNS.DAT FILE COMPACTION AND OPTIMISATION



Thas been fixed in Fixpack 8.1.1.6.


  • Bug 13827574: EXTRA SRVRMGR CONNECTIONS CAUSING PERFORMANCE ISSUES



This has been fixed in Fixpack 8.1.1.6.




Currently there exist Quick Fixes for the following Fixpacks to address these issues:






Customers that are looking to accelerate the enterprise startup
process should apply these Quick Fixes regardless if one of the above
errors is encountered or not. They should benefit from the optimized
siebns.dat write performance in any case.





To mitigate the issue on versions where currently no quickfix does
exist, the following workarounds have helped to reduce the load put on
the gateways siebsvc process:


a) do not start many siebel servers concurrently. Starting each
server with a delay of 3-5 minutes will give gateway sufficient time to
handle all incoming requests.



To monitor when a server has completely
started up, you can either run the srvrmgr "list procs for server
<servername>" command and verify all processes are in running
state or enable auditing by setting the gateway.cfg parameter
EnableAuditTrail = TRUE. Then monitor the file nameserver_audit.log. The
server has started up when no more lines are added to the audit file.

b) try to run as little srvrmgr commands as possible; use /i command
file switch switch rather than to launch multiple srvrmgr in a row.



In case a system monitoring tool is used that makes frequent use of srvrmgr, it should be disabled or poll frequency reduced.

c) if you are starting lots of additional batch component processes after initial enterprise startup:



define a dedicated admin user that only
has the Siebel administrator responsibility assigned and use this new
account to start recurring batch components like Workflow, Assignment,
EIM etc. The standard sadmin account has 228 responsibilities and all
these are retrieved from the database for each gateway login. Reducing
this to 1 makes the database login part of a gateway connection faster.

d) customers running Oracle 11g database might deploy the DRCP protocol to connect the gateways odbc source.



To setup a DRCP connection string see
note How To Setup and Trace Database Resident Connection Pooling (DRCP)
(Doc ID 567854.1). Then the gateways odbc source definition needs to be
changed to use the new drcp tnsnames connection string. Tests have shown
that gateways database login can be accelerated  with this DRCP 
approach by up to 30% compared to the standard dedicated  database mode.

e) Configuration changes to the enterprise should only be done in a maintenance window.





To avoid the risk of frequent siebns.dat rewrites, configuration
changes that require updates to the siebns.dat like for example adding
new servers to the entrprise, assigning or unassigning components,
changes in component definition etc should only be done when the
enterprise is down or there is very little load during low volume
business hours. In 24x7 deployments special care needs to be taken to
find a period of very low activity.






Patches


Fixpack: 8.1.1.3 QF03CK


Fixpack: 8.1.1.4 QF0475


Fixpack: 8.1.1.5 QF0591


Fixpack: 8.1.1.7 QF0723



History


 .


SREGHEZZ [31-05-2012]: Added bug and doc refs, plus added patch download links under the 'Patches' section












Applies to:


Siebel Financial Services Call Center - Version: 8.1.1.1 [21211] and later   [Release: V8 and later ]
Microsoft Windows (32-bit)



Symptoms


On a clustered environment Gateway server, the environment was working
fine. But, suddenly all attempts to connect to the Gateway through
srvrmgr commands resulted in error message:
"SBL-SCM-00018: Could not open connection to Siebel Gateway configuration store (<GatewayServer>:2320)".

When it happened, the NameSrvr.log showed:



SBL-SEC-10018: Specified driver could not be loaded due to system error 8 (Siebel Oracle90 J:/sba81/siebsrvr).
SBL-SEC-10007: The password you have entered is not correct. Please enter your password again.
[TCPIP-server] recv() failed for sd=-1 (err=10038 | Socket operation on non-socket (invalid socket handle).)





Cause


-The message "system error 8" is a Windows Operating System message
displayed when a server service fails to start. It stands for:
"ERROR_NOT_ENOUGH_MEMORY 8 (0x8)  Not enough storage is available to process this command".
Basically,
it suggests an application is using a lot of Windows resources
(memory). Hence, there was a possibility this error was related to a
memory leak issue.

- This behavior was not related to the Gateway server implemented on a clustered environment.

-
The siebsvc process of the Gateway was monitored and it was verified it
grew from approximately 200 MB to almost 2 GB: there was a memory leak.

-
Further investigation determined there were srvrmgr commands fired over
time,  which caused a small memory leak with each new DB connection
until it became a memory issue. This behavior is described in Bug 10572441 "Gateway memory leak due to DataDirect odbc driver".


Solution


Siebel 8.1.1.1 QF0125 addresses Bug 10572441.

In order to solve the behavior, follow the bellow steps:

1. Download Siebel 8.1.1.1 QF0125, patch number 9146659, from My Oracle Support site.

2.
Review the readme file of the quick fix and follow carefully its
configuration instructions, which include a parameter that should be
added in DBSecAdpt Section of gateway.cfg/namesrvr.cfg:



[DBSecAdpt]
Parameter "GlobalConnections = TRUE"


3.
Retest the issue. Notice the size of the gateway process (siebsvc.exe)
should be monitored in order to verify if the behavior continues.

4. Migrate the solution as appropriate to other environments.

Notice Bug
10572441 has also been fixed in Siebel 8.1.1.3 fix pack. However, the
parameter "GlobalConnections = TRUE" should still be added manually in
DBSecAdpt Section of gateway.cfg/namesrvr.cfg










Applies to:


Siebel Life Sciences CRM - Version: 8.1.1.3 SIA[21219] and later   [Release: V8 and later ]
Information in this document applies to any platform.



Symptoms


The customer is encountering the error below when trying to navigate to Server Administration views:

SBL-SDM-00109:
Error from Admin Server, SBL-SCM-00018: Could not open connection to
Siebel Gateway configuration store (gtwyhost:2320).


Cause


The issue was caused by multiple enterprises configured for a single
Gateway Server with different SADMIN password set for each enterprise.

Since
Siebel 8.1.x, it has been implemented a security feature to avoid
access to the Gateway Server information without being properly
authenticated as a Siebel Administrator, and this is verified by
connecting to the database using the password for UserName parameter and
querying for the 'Siebel Administrator' responsibility.


Solution


The customer was recommended to create a dedicated Gateway Server for
each Enterprise to avoid further conflicts, but finally decided to
rollback the password changes to have the same SADMIN password used
across the Enterprises.

Regards,


No comments:

Post a Comment