In the guide below, you can find useful information on applications included in Hydra Billing. To learn more about the application interaction please see Application Interaction (API).

• Web Applications
• Agents
• SNMP data collector (hsnmp)
• The Core (Oracle Database 11g)

Web Applications

• The Service Provider Console (hoper or arm_isp)
• The Customer Self-Care Portal (hupo or arm_private_office)
• «Migration» (arm_migration)
• Payment Portal (arm_payments)

Web applications that interact with Hydra are based on the Ruby on Rails framework (the Ruby programming language). They function under the rails OS user account with the help of the Apache web server together with the passenger module (or with the help of nginx + unicorn). New versions of web applications are set up on the server by HBS engineers at their own discretion.

Each application is set up inside a separate directory (for example, The Service Provider Console — inside /opt/hydra/rails/arm_isp, The Customer Self-Care Portal — inside /opt/hydra/rails/arm_private_office, and so on), which has the following contents:

• releases — a directory that contains releases (versions) of applications. Such a directory is used for storing all versions of a corresponding application to be set up on the server. In case of an unsuccessful update, it gives an opportunity to roll back to a previous version.

• current — a symbolic link to the directory that contains the version of the application that is currently in use. You should use this link to set up a path to the application in the settings of a virtual host inside the settings of a web server.

• shared — a directory that contains all the general settings for the application that are used regardless of releases. This directory, for example, is used to store temporary files and logs generated during the application run. 

Production

1. Rotating and archiving logs
   
   During its run, an application logs performed requests in the shared/log/production.log file, so that over a long production period it may grow up to several GB. To ensure having available disk space at all times, you should regularly rotate and archive this log, for example, with the help of logrotate.
    
   
2. Deleting obsolete releases
   
   Over a long production period, the number of obsolete application versions that are incompatible with the current version may grow enormously inside the releases directory. To ensure having available disk space at all times, you should regularly delete obsolete releases, for example, one month after their last use.
   
   

3. Cache flushing
   
   To speed up processing requests, web applications generate internal cache in RAM during their runs. In some cases, for example, when a link to the current release is changed, you should flush the cache to ensure stable application running. You can flush the cache by executing the following command from the application's directory:
   

   
   

   rails@server:~/arm_isp$ touch current/tmp/restart.txt

4. Launch, stop and restart
   
   In most cases, you manage web applications with the help of the /etc/init.d/apache Apache init script. For example, the Apache restart command can be as follows:

   
   

   root@server:~# /etc/init.d/apache2 restart

   
   

   You may need to restart a separate application, for example, when it becomes unavailable for all users.

Agents

• Traffic Collector (hcd)
• Autonomous RADIUS Daemon (hard)
• Active Equipment Manager (hamd)
• Payment Gateway (hpd)
• Interface Daemon (hid)

All agents use the Python programming language and have similar ways of setting up and managing their execution. In production mode, all of them run in a daemon mode and are managed with the help of individual init scripts (for example, /etc/init.d/hcd — for hcd). Each application has a separate config file with certain launch and logging parameters. See an example for hard (the /etc/hard/hard.conf file):

Production

1. Rotating and archiving logs

   During its run, the application logs data within an individual file, and the path to it is specified in config (for example, in /var/log/hard/hard.log  — for hard). All agents have built-in functionality for rotating a log file. When setting up parameters responsible for rotating  (log rotate size и log rotate count), you should pay attention to the rate at which a log file is filled in while in production mode. Use it to adjust necessary settings in config. A log file should contain data collected at least over one last week of the application run.

   
   

2. Rotating the MongoDB log
   
   Agent hard uses the MongoDB object database which, during its run, logs data in the /var/log/mongodb/mongodb.log file. Over a long production period, the file may grow up to several GB. To ensure having enough available disk space at all times, you should regularly rotate and archive this log, for example, with the help of logrotate.

   /etc/logrotate.d/mongodb-server

   
   

   /var/log/mongodb/mongodb.log {    daily    missingok    rotate 7    compress    delaycompress    notifempty    create 640 mongodb mongodb    sharedscripts # find в скриптах удаляет log files, которые mongodb ротирует встроенным механизмом.    postrotate       killall -SIGUSR1 mongod       find /var/log/mongodb/ -type f -regex ".*\.\(log.[0-9].*-[0-9].*\)" -exec rm {} \;    endscript }

3. Flushing cache for hard

   In some cases, agent hard (Autonomous RADIUS daemon) requires flushing cache of the internal MongoDB object database, which is used for data caching, You can flush the cache with the help of the following commands:

   
   

   root@server:~# mongo MongoDB shell version: 1.4.4 url: test connecting to: test type "help" for help > use hard_cache switched to db hard_cache > db.dropDatabase() { "dropped" : "hard_cache.$cmd", "ok" : 1 }

4. Launch, stop and restart
   

   You can manage agent applications with the help of individual init scripts (for example, /etc/init.d/hpd — for hpd). Restart command is as follows:

   
   

   root@server:~# /etc/init.d/hpd restart

SNMP Data Collector (hsnmp)

SNMP data collector, similar to agents, has its own config file /etc/hsnmp/hsnmp.conf where its operation and logging parameters are set up.

Production

1. Rotating and archiving logs
   

   During its run, the collector logs data within an individual file, and the path to it is specified in config. Similar to agents, hsnmp has built-in functionality for rotating a log file. When setting up parameters for rotating (log rotate size and log rotate count) you should pay attention to the rate at which a log file is filled in while in production mode. Use this rate to adjust necessary settings in config. A log file should contain data collected at least over one last week of the application run.  

   
   

2. Archiving data files
   
   During its run, the collector generates data files which are deleted when they are loaded into the system (i.e. the load mode, see Collector Operation Description). However, in some cases loaded data files may be required in order to analyze the reasons for the presence of traffic for some customers. In such cases, before loading, we recommend copying data files that are ready for loading into a separate storage where they can be stored for a long time.

The Core (Oracle Database 11g)

Files containing the Database data are typically located within the /var/oradata directory on the server. This directory stores tablespace files (the main ones are HYDRA and HYDRA_INDEX), redo logs, and auxiliary data files. During its run, the DBMS database creates database log files and tracing files inside the /opt/oracle/admin and /opt/oracle/diag/rdbms directories.

Production

1. Monitoring available tablespace
   
   To ensure having enough available tablespace at all times when operating in the production mode, you should set up monitoring available tablespace. For more information please refer to this section. 
   
   
2. Monitoring system tasks execution.
   
   To ensure a stable and logical system operation you should set up monitoring the execution of integrated tasks and adapt it to the current task installation.
   
   

3. Rotating and archiving database log files and tracing files
   
   Over the production period, the DB creates a large number of active log files and tracing files inside the /opt/oracle/admin и /opt/oracle/diag/rdbms directories, so that they take up a large part of the disk space. To avoid it, you should regularly delete obsolete files, for example, one month after their creation. In case you have a standby server please make sure to rotate the /opt/hydra/oracle/logs/update.log file. 

    

4. Rotating and archiving logs for working with external tables логов работы с внешними таблицами
   
   When working with external tables, Oracle logs the results for each table into a separate file. External tables are used, for example, for loading accounting details into the main DB by Agent HARD (starting from version 4.0). We recommend setting up rotating for these logs and periodically deleting obsolete files. For example, rotating can be performed daily, and you should consider deleting files that were changed more than one month ago.
   
   You can get the path to the file directory with the help of the following request:
   

   
   

   SELECT DIRECTORY_PATH FROM   DBA_DIRECTORIES WHERE  DIRECTORY_NAME = 'HYDRA_TMP_DIR'

   
   

    

   When working with the HYDRA_TMP_DIR directory please note that logs for working with external tables are stored only within files of the *.log type, so that deleting or changing other types of files will lead to the software bugs.

    

5. Launch, stop, restart

    
   You can manage the operation of databases set up on the server with the help of the /etc/init.d/ora.database init script. For example, the restart command is as follows:

   
   

   root@server:~# /etc/init.d/ora.database restart

6. Deleting unaccounted traffic data
   

   When the amount of unaccounted traffic data within the system reaches large numbers (see the EX_TRAFFIC_COLLECT_C table, and check if the number of lines is over 2 mln.), you should clear the data with the help of the following SQL request:

   
   

   SQL> TRUNCATE TABLE EX_TRAFFIC_COLLECT_C;

   
   

   Make sure to perform this request under the AIS_NET user.
   
   

   A large number of records inside this table means there is a large amount of unaccounted traffic. You should sort out this issue (see the Hydra Billing User Guide for the corresponding manual), or increase the threshold for unaccounted traffic registered over one session for collecting the data off the collector.

Possible issues

1. The «ORA-01555: snapshot too old» error in the task for a DB schema analysis
   Over the production period, there can be errors while running the task for the DB analysis. Typically, such errors are caused by the insufficient size of the rollback DB segment. For example:

   
   

   ORA-01555: snapshot too old: rollback segment number 8 with name "_SYSSMU8_2372141723$" too small

   
   

   To resolve this issue, you need to check the current size of the rollback DB segment with the help of the following request:

   
   

   select p.value                            undo_retention,        max(maxquerylen)                   maxquerylen,        (max(maxquerylen) * 1.2)           undo_retention_optimal,        (p.value - max(maxquerylen) * 1.2) delta from   v$undostat s,        v$parameter p where  p.name = 'undo_retention' group by p.value;

   
   

   If the undo_retention_optimal value is much greater than the undo_retention value (i.e. the delta parameter is negative), you should execute the following command:

   
   

   SQL> alter system set undo_retention=$undo_retention_optimal scope=both;

   
   

   where instead of the $undo_retention_optimal variable you should insert the figures as per the corresponding segment value you have previously defined.

   This request is to be executed under the AIS_NET user.