Example System / System Integrator's Manual

Note: This technical article is a work-in-progress. More content will be added, if needed.

Concept

We'd like welcome you to the community of Genius Vision NVR users and are thrilled that you would want to put our software into a larger system integration in order to create more mutually benefitial value.

Before you actually dive into IP video system integration, there are some important information that we'd like you to review. That's why we wrote this article that collects many frequently asked questions or related issues into one place, thus as a system integrator you can make informed decisions before you proceed.

IP video surveillance is not for everyone

IP video surveillance is the combination of video surveillance and IP technologies, which in turns, are closely tied with last decades of evolution of almost all information technologies (IT). For people who do not have basic IT training, it could be difficult to do the most basic tasks. Nevertheless, the combination with IT makes IP video extremely versatile to be applied into various projects.

For traditional CCTV system integrators, IP video is like a double-edged sword, that it has the advantage of powerfulness but comes with the cost of difficult-to-learn. IT industry is an eco-system with multiple layers of supplier. Integrators are frequently asked to take the responsibility to combine various components (computer, OS, network switch/cables, hard disk, camera, software, etc) and deal with their compatibility and maintenance issues. In this eco-system, no component supplier will guarantee that their product works with all combinations of other components. A successful integrator will need to learn most of their experiences from trial and error. These facts often cause tremendous confusion to traditional CCTV integrators.

Try to ask youself following questions. If you can answer these questions with high confidence, then IP video might just work very good for you. If not, you need to take some network or computer technology lessions before really put IP video to work.

Do you know how to configure a newly purchased wireless router properly with office equipments, or mobile phones, so they can all connect to the Internet?
Can you isolate and fix duplicated IP devices in the network?
When a previously good computer won't boot, can you isolate the problem by examing each component?
Can you tell roughly when a fan, hard disk, or a power supply, is about to break?
Do you know what are the following terms: DHCP, LAN, WAN, TCP/IP, HTTP, DNS, SMTP, FTP?
Do you have basic understanding to IP technologies that involves following terms: IP address, gateway, IP subet mask, NAT, ipconfig, ping, Ethernet, CAT5, port forwarding, link aggregration, TCP connection, UDP, DoS, and DDoS ?

System integration is a specialized expertise

Some users are under the impression that system integration tasks of IP video is simply to purchase a bunch of components and connect them altogether, and then they should all work. We cannot stress enough that this is wrong. It can be that simple only under very trivial cases, but for most of the cases, long term reliability with high system loading is required in order to be cost effective.

The three requirements we must emphasize, which are (1) long term reliability (2) high system loading (3) cost effectiveness, make the system integration tasks of IP video a very specialized area of expertise. What it really means is that a system integrator must carefully balance every aspect of a integrated system to reduce the potential bottlenecks and risks.

For example, if you want to build a video recording system with 400 cameras, you should not put all the 400 camera into a single recording server. Doing so will violate the balancing principle and introduce multiple single points of failure, e.g. the recording server or the network switch the server connects to.

We cannot tell you how to properly design or integrate a system, as it's completely dependent on the requirements from the user. It can be focused on either one of security, cost, reliability, or quality. It can also be a combined requirements with different priorities on each item. The role of Genius Vision is to provide a versatile software that comes with many functions, but you have to decide how to use the software on the system you designed. Improper system design could lead to problems that manifest on software level, and they're simply beyond the expertise of a software developer to solve them, or even to identify them.

Testing, testing, and more testing before going to the field

Testing can be a very important task because most of the time when system is gone to the field, it's difficult to diagnose any problems when something goes wrong. You need to establish your confidence level by simulating similar testing envirnoment inside your own lab.

It's not wise to put only trivial loading to lab system, while using ten times of loading in the field. You could never know what happened, if something ever go wrong in the field.

How can Genius Vision help you on technical issues?

Genius Vision is the software developer who developed Genius Vision NVR. We can potentially solve almost any problems on the software we design, but this work only in a particular and limited way, as we will elaborate as follows.

Reproducibility

For our R&D technical staff to solve any issue, the first step is that we can see and recreate the issue reliably inside our lab. To achieve this “reproducibility” goal, often we need following information:

Software product version and edition.
Operating system type. (including OS version, service packs, and 32-bit/64-bit)
InfoCollector data or crash dump diagnostic data.
Screenshots or configuration screenshots.
Is the system ever running normally?

If yes, then when did the problem occur? Has anyone done anything to the system right before the problem occur? Did such change affect software behavior?
If not, then in what environment would the software run correctly? What factors could differentiate the correctness and incorrectness of the software behavior? Please try to (1) replace or isolate system components, or (2) change a different computer system in order to figure out the point of conflict.

What is the probability this problem occurs? Can it be reproduced in different environment? To solve a non-repeatable random problem is extremely difficult in practice. To solve a problem that can only occur in specific hardware/software combination could also be potentially very challenging.
Can this problem be solved by changing to another computer or system environment? If it can be solved this way, then it means the point of conflict might be hidden in the environmental factors. Please try to utilize scientific method and control variables to determine the possible point of conflict.

External factors

If the problem cannot be reliably recreated inside our lab, then it's conceivable that the root cause of the problem does not lie inside the software, but somewhere else in the integated system. For example, it can be:

Improper user operation.
Overload of the system.
Obsolete or partially damaged equipments, components, or cables.
Bugs in camera firmware, hardware, or OS.
Computer virus, anti-virus software, firewalls, or other interfering factors.

The possible causes listed above is not an exhaustive list, but it shows the fact that any of the causes in the list can produce a problem in software that is almost impossible to be diagnosed by just examing the software programming. We need help to acquire more information about possible root cause.

Scarcity of resource

Lastly, let's not forget the scarcity of any resources, particularly the time of an R&D technical staff. When requesting a technical support, please also provide urgency and priority through proper sales channel. These additional information would also provide important hints about how we schedule each request.

I need to report a software problem. How to do this effectively?

If you encounter problems using our software, and that you require technical assistance from us, please don't just use plain description to contact us.

Plain description usually offer insufficient or partial information. It's not efficient for us to respond to such inquiry. The back-and-forth Emailing would introduce significant management cost that we simply cannot afford. I mean, we all have to work cost-effectively, don't we?

For any possible technical assistance, detailed operation steps, screenshots and/or screen video recording should be provided and organized in a way that is easy-to-understand. The more accurate data provided, the more likely we will dispatch human resource to investigate the problem.

If you think there is something wrong with the software (that you have operated correctly but did not get the expected result), please read this article to understand how to properly report a software problem, and please make sure you encountered the same problem on at least two or more different environments.

Thank you for your understanding.

If I pay for the software, how much more support can I get in the field?

We have written a specialized FAQ article around this question:

Fields problem FAQ for paid customer

Can you remotely assist me through TeamViewer to fix my problem?

TeamViewer or remote assistance is a topic covered by paid consultant service, so it's not offered freely or accompanied by the purchase of software license.

We have written a specialized FAQ article around this question:

Why can't you use TeamViewer to assist me remotely?

Does Genius Vision provide mandatory service regarding fixing software issues or bugs?

No.

As our standard licensing term states, Genius Vision excludes the implied warranty of the fitness of a particular purpose. In practical sense, no one can guarantee any software runs in all combination of all system components and operations.

We will, nevertheless, do what is commercially reasonable in good faith regarding technical support requests.

If you have urgent requests that require us to dispatch engineering force to provide some kind of service, you can refer to our consultant services, which are a paid services and are available only through our distributors.

What are the terms & conditions using Genius Vision software?

Standard licensing terms

1. All software licenses, unless otherwise specified, is per-camera-channel permanent license and includes one-year free upgrade.

2. For software license of Commercial Edition, Genius Vision employees "online license authorization" mechanism. You will receive electronic license code without physical goods. The target computer where you intend to use the software on must be equipped with Internet access at the time of installation. The granted license will be fused into target computer configuration and cannot be migrated to another computer. In the case of computer maintenance, such as OS re-installation, you are responsible to properly backup/restore license information in order for continued use.

3. For software license of Subscription Edition, license are stored on Subscription Cloud Server and ready to be downloaded to NVR. Please refer to Introduction to Software Subscription Licensing.

4. The effectiveness of the software depends on many factors, including but not limited to: hardware, network, OS versions/patches, other software, and user operations. You have already tested the evaluation version of the software and found the software is suitable for your usage. By purchasing you agree to indemnify Genius Vision from any and all direct and/or indirect damages as a result of using the software that is caused by non-malicious intention. Therefore before you put the "Software" into use, it is your sole responsibility to evaluate and ensure its fitness.

5. Genius Vision excludes the implied warranty of (1)the fitness of a particular purpose (2)merchantability and (3)non-infringement.

6. The "Software" is a copyrighted work and proprietary properties of "Genius Vision Inc." It is a violation of international copyright law to (1)redistribute without proper authorization (2)reverse-engineer the software (3)circumvent license protection mechanism

Notice to System Integrators

We do not recommend changing anything on a well-established server in operational state, including upgrading software or adding license. The user must be aware and be solely responsible that any change to such system could result in data loss that Genius Vision is not responsible of.
Our standard support policy does not cover systems that have operation history over a year, including adding new license or upgrading new software to such systems. Support request over systems that have operation history over a year is subject to paid consultant service.
Due to the complexity introduced by IT industry eco-system, integrators are required to take the responsibility to combine various system components and deal with their compatibility and maintenance issues. The sophistication of professional software such as Genius Vision NVR needs experts to properly install and adjust configuration carefully. You could risk losing our standard tech support for improper user operation or configuration.

Topics

Dual Stream - Reduce CPU requirement

What is Dual Stream

Dual Stream is the ability of camera to simultaneously stream two videos with different encoding/resolution/framerate/bitrate. And in some cameras each stream are limited to only certain encoding and resolution combinations. The camera must support Dual Stream for NVR software to use it. It can't be done with NVR software alone. In Genius Vision NVR, primary stream is for recording and secondary stream is for live view and IVS analyzing. Keep in mind that streams are not profiles. The terms are sometimes used interchangeably by camera manufacturers. But a camera with multiple profiles are not always able to stream them simultaneously.

What Dual Stream can do

Save storage. Set the primary stream to low resolution, low framerate, or high compression (H. 264).
Decrease CPU loading. Set the secondary stream to low resolution, low framerate, or simple encoding (JPEG, MPEG).

What Dual Stream can NOT do

Decrease bandwidth loading. On the contrary, it increases bandwidth loading. NVR requests two streams from camera.
Stream partial area of a video. This feature is usually labelled as sub-window or ROI (region of interest).

Requirements to use Dual Stream in Genius Vision NVR

Camera must support Dual Stream.
Camera must be connected using ONVIF driver. To check if your cameras support ONVIF, visit http://goo.gl/U9eGn3

How to use Dual Stream

To use Dual Stream, sub-streams usually needed to be enabled in camera. They are disabled by default because it adds additional loading to the camera. Because the various design of API, Genius Vision NVR does not (and will not) integrate this setting in software. To enable sub-streams, please use IE to log into camera and consult the manual shipped with camera or downloaded from camera website.

Figure 1. There are 4 sub-streams for this camera. They don't have the option to enable/disable.

Now go to the Genius Vision NVR, open Camera Setup dialog. Note that on the right-top corner ONVIF driver is displayed. If it's not ONVIF, you may see different tabs and options, and Dual Stream is probably not supported. In Video and Stream 2 tabs, select the stream you want to use. The options are different from camera to camera.

Figure 2. Settings for primary stream. This is the stream for recording.

Figure 3. Settings for secondary stream. This is the stream for live view or IVS analyzing.

If the camera is streaming 2 videos, they can be selected in right-click menu of player.

Figure 4. There are 2 streams: Main and Secondary.

Figure 5. Main stream is 1080p H. 264 and Secondary is CIF JPEG. They are zoomed to see the quality difference.

There are also some automatic mechanisms for NVR to use the stream best fit the use.

Figure 6. NVR automatically selects 2nd stream in 3x3 layout or layout with smaller players. This also applies to Matrix View.

Figure 7. Stream for IVS can be selected in Video Analytics Setup. IVS usually don't benefit from high resolution. Keep in mind that only server-side dectection have the option because streams for client players are controlled by right-click menu and layout.

Storage (Disk) Requirement

Recording space estimation

We provide storage space calculator function inside our software to can assist you to compute the required disk space.

Memory requirements for playback

To prepare the archiver subsystem for fast playback, the memory required is proportional to the number of existing recorded files. The memory is used simply for file indexing. One should keep in mind that there is a limit to everything, and in this case:

The number of total recording files for playback is limited by available memory.

For example, the indexing memory required for 200MB file size is 5 times than the memory required for 1GB file size. For very large disk arrays, such as 20TB, 5 times of file count could be a major diaster.

Currently we don't pose a hard limit on the total number of playback files, so it's the administrator's responsibility to make sure the system is properly configured so the memory will operate within acceptable range.

There are several memory topics related to this issue:

32-bit application memory limitation

Debunking the myth of RAID storage

RAID storage systems are often advertised and perceived as the ultimate high-end solution for fail-over and backup mechanism that can overcome disk data loss problems all at once.

However, this is not the whole truth. The only truth about RAID systems is probably that being a high-end solution, it's much more expensive. Like all system add-ons, using RAID comes with pros and cons, and it's all about trades-offs. So if you chose to use RAID, be sure you understand all the goods and bads in depth. If you are not well-informed about all the goods and bads, it's probably a bad idea to adopt RAID because whether or not a video recording system should adopt RAID is a highly controversial issue (meaning it's not all good), so you should decide carefully.

Here we can help you debunk the myth by giving you hints about certain concepts around RAID. You can look it up on the Internet for more accurate information:

RAID is not intended to be a data backup system, it merely increases overall system availability.
Data stored in RAID can still be damaged (or even completely destroyed). The probability of such damage can be reduced, if RAID is properly configured, but a chance is still there.
Improper configuration of RAID could increase the risk of data loss, comparing to that without RAID.
RAID can reduce overall system performance, sometimes drastically.
The average lifespan for disks mounted on RAID is much shorter than that without RAID.
RAID, like any disk systems, requires periodic maintenance. Without proper maintenance, data loss could be catastrophic.

Please note that because Genius Vision is a software developer, we are not responsible to provide you all the information about RAID systems. These hints are only here for informative purpose and you should consult your RAID solution provider for more detailed information.

Disk Damage Detection (Slot Blacklisting)

Overview

Starting from version 915, an experimental feature is added that when storage encounters a critical error (be it hardware malfunction, driver error, or OS error), NVR storage subsystem may fail to mount, making NVR unable to record any video. Storage monitor add corrupted file to a blacklist and avoid it to make sure storage subsystem works.

Limitation

Storage monitor only prevents NVR from accessing corrupted data that stalls storage subsystem. It does not…

1. Fix hard drive malfunction.

2. Recover corrupted data.

3. Detect corrupted video and recover it.

Disk space will be lost for any corrupted file blacklisted.

Usage

On NVR start, storage is busy scanning files...

If it takes longer than usual to finish, warning message pop-ups and shows scanning progress.

If any file takes too much time to be opend, NVR blacklists that file and restart. After restart, the file will be skipped to avoid critical error.

On NVR startup, if there's any blacklisted file, Critical Disk Error dialog pops up and in system logs all affected drives are listed.

NVR probably already blacklisted all affected files and storage may come online. But this message usually indicates that there's a critical error in hard drive, possibly caused by hardware malfunction. There's a high risk that recording will fail or recorded video will be lost. Users are suggested to do the following procedure…

Find which hard disk drives the affected logical drives are located on
Replace all affected hard disk drives except system one (usually C:)
Run NVR and click "Reset blacklist" in Critical Disk Error dialog
Restart NVR
If system drive is affected, avoid using it for recording

The dialog can also be opened in system logs.

Replacing problematic hard disk drives is the only way to fix critical disk error and keep NVR stable and healthy.

RemoteChannel - Simulate Required Bandwidth

What is Remote Channel

Remote Channel is a special feature in Genius Vision NVR to request video from other NVR just like from a camera. It can be used to duplicate video sources. Connecting to a single camera from multiple NVR usually makes the camera unstable or even stops working. NVR uses PC hardware to duplicate videos and can handle more connections.

What Remote Channel can do

Remote Channel can be used for two purposes:

Testing, especially for performance benchmark of NVR hardware (PC). It's common that a NVR connects to 16+ or even 32+ cameras and this puts lot of stress on the system. The only way to find out best cost/performance combination, is to actually put the loading on the PC and run for days or even weeks. In the early stage of system design it's really not practical to buy a lot of IP cameras. Using Remote Channel to duplicate cameras can easily test the PC to the limit.
Allow more users to view a camera. There's always a limit of TCP connections a NVR can handle. The limit is determined by hardware, software, and OS. To break the limit, use Remote Channel to duplicate video and balance user requests among multiple NVRs.

How to use Remote Channel

To create a remote channel, add channel just like common cameras and select driver "Remote channel".

Channel is the channel name of the source channel in NVR, not the channel index of a video server.
Fill the IP address of the source NVR.
Default port is 3557, not 80. Fill the proper value if it's changed in NVR config.
Login information is for NVR, not for camera. The user in the source NVR must have access to the source channel.

Figure 1. Remote channel is created just like any other channels.

Figure 2. Nine remote channels are created to duplicate a single video source.

Memory Requirement

Incoming Network Connections Limitation & Requirements

There is often a question about how many users can remotely connect to a NVR server. There is no direct answer to this question, but you can extrapolate an idea using information provided in this section.

To control system security, we set an upper limit to the number of incoming TCP connection. Sometimes this prevents improper user operations crash the server. This limit can be changed, but is highly unrecommended because changing it can compromise system security.

The default limit of incoming TCP connection is 150. A user connected to an Genius Vision NVR will need 3 TCP connections, one to maintain user authorization, and the other two for metadata query. With each new video viewing (live or playback) is requested, one more TCP connection is required. Video connection is only needed when video is transmitting. If you close a window, or stop a playback, that connection is no longer needed.

In summary, the number of TCP connections required is calculated as follows: If N is the number of active user sessions, and M is the total number of video connection requested, then:

<required-number-of-TCP-connection> = N*3 + M

Local connections are also accounted. Using connections exceeding this number is not recommended, as connection request will simply rejected by server.

Changing connection limit

This TCP-connection limit can be changed, however changing it could compromising system security and therefore is not recommended. Please be noted that unrecommended settings changing can disqualify your from requesting support, so you are on your own risk changing those system settings.

Below is a link to the technical article illustrating how to change this TCP connection limit.

Advanced System Configurations: TCP Connections

Frequently Asked Questions (Technical FAQ)

Can your software support large projects (10,000+ cameras) or not?

Our official policy is not to respond to this type of inquiries directly, but let us try our best to explain as follows:

For obvious reasons we cannot just answer "Yes!", because if we did, and something goes wrong in the field caused by unknown bottleneck in network, hard disk, or memory, we became the one to blame.

The fact is that if anything goes wrong in the field, even a "bug" in computer OS/hardware, network, or camera can manifest as a software error. Thus it's not easy for us to answer this kind of questions.

We knew some of our customers used our software on more than two thousands cameras project and succeeded. We also knew some customers used our software on two hundreds cameras and encounter many issues. We don't have the knowledge about why they succeed or fail, but we think whether or not a software can be applied to large project heavily depends on system design decisions and careful testing before-hands.

For example, we know at least one way to do a 1,000,000,000 cameras project. That you can use one PC to handle only 10 low-resolution cameras, then install 100,000,000 such an enclosed system independently. However, most projects are probably a bit more complicated, and these projects tend to connect everything together. Upon this scenario, many architectural decisions become vital as to the question "whether or not such a large system will work". These design decisions will come in various flavors and each is associated with different pros & cons. For example, if you purchase more computers, the cost goes up, but reliability also increases. If you want to save cost, you can connect as many cameras as possible to one computer and risking it to overflow. It's always a trade-off in these design decisions, and it's simply beyond our scope of expertise to make such design decisions or judgement.

We've met some customers who "believed in" our software, that they tried to put incredibly high loading to one computer while using unreasonably cheap hardware. When the system crashed, they try to blame us, but we knew the root cause is their inappropriate cost-down design strategy.

As one can surely understand that a software so complicated like ours, it's not possible to guarantee that it will work in all combinations of various system components, but we believe among all competitors of ours, we offer outstanding reliability because of our prudent software engineering methodology.

To understand how a software developer, like us, handles such doubts, one need to understand there is a limit to our problem solvability. To solve a problem, we need to establish a reproducible lab environment and conduct repeated tests by adjusting program parameters. This is basically what all software developers do. As we do not offer "field programming/fixing service", we often use simulation techniques that can create something a bit more than what we actually have. Even with these techniques, as limited by our intrinsic (small start-up) resources, there are still questions that are simply beyond our ability to answer, but in most cases, if one is able to find a way to reproduce a scalability issue in our (limited) lab, we can solve it.

We want to be clear that the "testing service" to address a particular issue mentioned above comes with engineering efforts, thus it's not automatically included in license fee. But if you have such a project, you can convince us (by providing concrete offerings) to help to solve some potential issues (if any), in order for you to enter the large project world.

We've been frequently asked similar questions, so we wrote a bunch of related FAQs for your reference:

What if I need some kind of software modification to win a project? Can you help me?

To further understand the standing of Genius Vision on this issue, please firstly read following FAQ entries:

Generally speaking, people don't quite understand the cost to develop a feature that will be combined to a continuously maintained product, such as Genius Vision NVR software. In short, it's extremely expensive, and yes, most likely we can do it given reasonable funding, but we don't recommend customers to engage any modifications. Our reasons can be illustrated by the following:

Dispute of requirement understanding: Often a customer made a set of requirements and would regret that he or she didn't add more requirements later when modification is done. At this time, dispute will arise and it takes a lot of time to sort out these disputes, even when both parties are in good faith. In our experience, the cost of time to engage any dispute resolution is simply not worthwhile.
Our software is already very versatile in function: There are already some users complaining the software being too complicated due to too many functions. It's not wise to add arbitrary function just because of a specific project needs.
Most project-specific modifications can't be utilized repeatedly: Due to the high cost of software modification, customer will want to balance the cost by doing many similar projects with the same modification. However, in our experience, this idea is simply impractical. Either because (1) those "similar projects" turn out to be not-so-similar and requires additional modification for each new project, or that (2) the customer fails to win the subsequent projects. In the end, the cost is still not justified and dispute will arise.

While our standing about software modification can be seen as "not recommended", it does not prevent you from proposing your offerings. Considering equivalent market pricing of an experienced R&D team with matured product, the price to do any modification will not be a small number in our opinion (and no, we cannot provide quotation for any software modification because we don't recommend it).