Private: implementing esg

Debug tools

We have the following software analyze tool:

1) Divicatch + USB Analyzer

2) DVBSam + WinTV DVB-T USB

We have the following hardware:

1) Nokia N92

2) Nokia N96

We have the following 2 "working" transport stream:

1) N92.ts (only works on N92, this stream is created in Aug 7, 2006, comes from Nokia)

2) N96.ts (only works on N96, this stream is created in Oct 16, 2007, comes from Nokia)

These streams are under \\students\yliu\DVBH_debug\TS

Analyze of N92 DVB-H Transport Stream

For N92.ts, its transmission parameters are as follows:

 Frequency1: 562 MHz
 Modulation QPSK
 DVB parameters: 8MHz, QPSK, 2/3, 1/8, 8k
 Code Rate 2/3
 Guard Interval 1/8
 Transmission-mode 8k

It includes two ESGs, one is Nokia OAI, the other is early verion of CBMS

[Trial with cell phone device]

N92 cell phone can load Nokia OAI ESG very well and play the corresponding program. and can not load CBMS ESG. Since in this transport stream, it has two int tables, and each one correspond to one type of ESG. So it is easy for me to identify which one works.

N96 failed to load any of the above ESGs.

[Trial with software analyzer]

1) DiviCatch can not detect Nokia OAI ESG, can find CBMS ESG running on port 3937 but failed on bootstrap steps.

2) DVBSam can see Nokia OAI ESG Bootstrap running on FF05:0000:0000:0000:0000:0000:0000:012D with port 9214 , can pass bootstrap steps but failed on ESG receive; can see CBMS ESG running on 224.0.23.14 with port 3937, can pass bootstrap steps but failed on ESG receive

[Conclusion]

It seems like early version of ESG are not supported by current analyzer and latest cell phones.

Analyze of N96 DVB-H Transport Stream

For N96.ts, its transmission parameters are as follows:

 Frequency 690 MHz
 Modulation QPSK
 Code Rate 2/3
 Guard Interval 1/8
 Transmission-mode 8k

Inside this TS, there are three channels:

• Channel1 is used for ESG delivery
• Channel2 contains three programs, all free to air
• Channel3 contains two encrypted programs

Our cell phone and analyzer can see the free programs from Channel2.

Channel 2 contains 3 video programs:

• Program1 use 102143.sdp
• Program2 use 102317.sdp
• Program3 use 102491.sdp

It includes three types of ESGs, 1st is OMA BCAST, 2nd is DVB IPDC, and the 3rd is Prestandard OMA BCAST and they all use port 9214.

N92 cell phone can NOT load any of the above ESGs

N96 can load ESG and play the video.

But since this TS file only has one int table, so three types of ESG are mixed up together. So the goal is to find which is the actual working ESG on N96.

• Try to load IPDC ESG by divicatch : this can pass bootstrap step and then will crash the Divicatch software during ESG reception.

• Try to load OMA ESG/Prestandard OMA BCAST by divicatch : this can pass bootstrap step and then will crash the Divicatch software during ESG reception.

[Conclusion]

Our USB analyzer software can not even read the ESG that works on Nokia N96 cell phone!

• DVBSam failed to receive Prestandard OMA BCAST

• DVBSam can receive IPDC ESG and OMA BCAST ESG successfully.

[Conclusion]

Either IPDC ESG or OMA BCAST ESG in N96.ts is valid for Nokia N96 cell phone.

I have put ESG Bootstrap data, IPDC ESG, OMA BCAST ESG for this TS in \\students\yliu\DVBH_debug\ESG

Compare IPDC ESG form N96.ts with our current implementation

One obvious error of our current IPDC ESG implementation is in urn:dvb:ipdc:cid:2

The length of ESG data repository is wrong, should be 00 0B 53 instead of the original 00 07 5E , such that the whole SDP can be transferred.

The IPDC ESG used insider N96.ts:

• Originally sent in gzip format

• After unzip, self contains 17 outline SDP files, 5 image files (one for each video program) and 7 XML files

 19_Acquisition.xml 
 19_Content.xml  
 19_PurchaseChannel.xml
 19_Purchase.xml
 19_Schedule.xml 
 19_ServiceBundle.xml
 19_Service.xml

Our current IPDC ESG implementation:

• Not sent in gzip format, no need to unzip

• Self-contain 1 inline SDP file, 4 XML files

The main difference is how we deal with SDP files, ours are created by VLC which may not works fine on N96 cell phone.

Possible failure reason of our current implementation:

• Might be inline SDP content has problems?

Actually we have not reached to parsing SDP step. So I use the word might be. As if only SDP fail, we should be able to still detect and select the ESG, but when we try to play the corresponding channel, the ESG will tell you no stream received (due to bad SDP)

• Maybe due to get too many ESG bootstrap

Using DVBSam, when I check ESG status, our testbed will refresh ESG very quickly (less than 10 seconds) That is to do a ESG Bootstrap again and again.

The bad thing is during a Boorstrap, it will delete all the existing ESG. So maybe the cell phone is trying to receive the actual ESG, but out testbed sends a ESG bootstrap message and delete all the ESG the cell phone is trying to receive. (which equals to a refresh in DVBSam ESG monitor)

And in real DVB-H TS, we can easily see there is no such kind of refresh for ESG. After bootstrap ,we get the actual ESG and they are always sitting there!

Obviously, the real world DVB-H TS is using an ESG server to populate the ESGs , and we are using flute tool, I do not know how to get rid of more than necessary ESG Bootstrap based on our current implementation.

And do remember to use DVBSam which is very powerful. Divicatch in most case can not help much on ESG part!

Compare OMA BCAST ESG used in N96.ts with our current implementation

The OMA BCAST ESG in N96.ts is much more complex than our current. It has 9 containers, self contain 93 SDP files , 29 xml files and 19 DU files.

Ours only has 1 DU file, 1 SDP, 4 xml files.

I did not do much on this direction due to time limit.

DVB-H related info found on N92

C:\dvbh.ini

 ScheduleItemRemovalInterval=7
 AutoScanNwTimeRetryDelay=5
 ESGRootPort=9214
 IpdcNumberOfLogFiles=3
 IpdcLogFile=ipdc.txt
 IpdcLogDir=ipdc
 IpdcLogLevel=1
 IpdcLogLevelRDebug=9
 IpdcLoggingEnabled=1 
 NifFwdToStackInterval=500
 NifReadInterval=500
 SocketRecvBufSize=800000
 TerminalMaximumBandwidth=500000
 RequiredDiskFreeSpacePercentage=5
 AllowUsageTracking=1
 SignalScanFreqLow=472000000
 SignalScanFreqHigh=702000000
 SignalScanBW=8000000
 SignalScanType=1
 AutoScanEnabled=1
 AutoScanStartDelayAfterBoot=5
 AutoScanInterval=10800
 AutoScanPreferredPlatformId=-1
 AutoScanPlatformSetRetryDelay=120
 PlatformSetUniversalTimeout=60
 ScanUniversalTimeout=600
 NetworkTimeUpdateUniversalTimeout=35
 BootUniversalTimeout=20
 InactivityPowerOffTime=300
 ER4BootImageFile=z:\spi_boot.img
 ER4SWImageFile=z:\enginer4.img
 ER4AntennaSetup=z:\ER4AntennaSetup.dat
 ER4SetPlatformRetryTimeout=25000
 ER4SetPlatformRetryCount=10
 ESGMgrUpdTime=120
 ESGMgrUpdTimeAfterErr=10
 ESGMgrUpdInterval=180
 ESGMgrUpdRestartCount=3
 ESGMgrRxInitTime=30
 ESGMgrRxPeriodChkTime=20
 ESGMgrUpdateEnabled=1
 ESGMgrFCastRestartCount=3
 ESGMgrUpkeepCleaningDelay=20160
 ESGMgrCarouselDeltaStartTimeInHours=48
 UsageTrackerRetryInterval=1
 UsageTrackerRetryCounts=5
 ECRenewalTime=1800000000
 ECRenewalOffset=1
 ECRemovalTimeInHours=48
 AutoScanPlatformSetRetries=1
 NwTimeUpdateRetries=2
 PowerOffDelayAfterNwDiscovery=3
 ReplayBuffer=30
 UsageTrackerDbUpperLimit=1000000
 DRMClockUpdateInterval=30

Error message found on N96

Based on N92 dvbh.ini config file, I guess on N96, DVBH.img is either ER4BootImageFile=z:\spi_boot.img or ER4SWImageFile=z:\enginer4.img on N92. And it sounds more like the complete image, not just the boot image.

And even though N96's dvbh.ini file is short, it seems like some parameters still follows the N92's example, for example the log files on N96 matched with N92's config file

 IpdcNumberOfLogFiles=3
 IpdcLogFile=ipdc.txt
 IpdcLogDir=ipdc
 IpdcLogLevel=1
 IpdcLogLevelRDebug=9
 IpdcLoggingEnabled=1 

In DVBH.img we find the following readable error message:

 PSI_Processor   PSI_patpmt  PSI_nitint

 ThreadX API error: The event group is already created. Threadx API error: Invalid event group pointer. The pointer is NULL. System timer thread obj\dvbh\dtv1000_dbg
   

As the above error message happens after the PSI/SI info section, It is the next step--ESG Bootstrap on Cell phone that cause the above error lines. The cell phone tried to receive something and the application crashed. So on N96 the Subsystem ESG Manage did not get anything and leads to timeout.

TS files for various ESG versions available on-line (Mon Oct 13 09:30:11 PDT 2008)

There is a recent DVB-H trial that adopts multiple ESG versions. A final report can be found at here. In this trial, two types of Nokia ESG platforms are used: OAI and OMA BCAST. They also made some ts files available on an ftp site (see pp.18 in the above report).

To reverse engineer the requirements of Nokia's OMA BCAST ESG implementation, we can download and analyze the ts files of Nokia BCAST platform. We can use the channel parameters listed in the report to identify the right ts stream.

To debug buffer overflow problems (Sun Oct 5 14:57:03 PDT 2008)

There have been quite a few nasty buffer overflow bugs in dataaggregator and timeslicer, which cannot be easily found by reviewing the code. To trace the problem, we may use the memory checking feature of the valgrind tool. Valgrind can be installed through apt-get.

To use it, compile your code with CC flags: -g and -O0. Then run the timeslicer (for example) as follows:

valgrind --leak-check=yes ./timeslicer -n 1024 -g 8 -b 8 -d 400 -c 3 -o qpsk -p esgbt:/tmp/fatcapsesgbt:0x555 -p esg:/tmp/fatcapsesg:0x666 -p prog1:/tmp/fatcapsinput1:0x771 -p prog2:/tmp/fatcapsinput2:0x772 -p prog3:/tmp/fatcapsinput3:0x773 -p prog4:/tmp/fatcapsinput4:0x774 -p prog5:/tmp/fatcapsinput5:0x775 -p prog6:/tmp/fatcapsinput6:0x776 -p prog7:/tmp/fatcapsinput7:0x777 -p prog8:/tmp/fatcapsinput8:0x778 -f /tmp/fatcapsfifo

The output of valgrind is descriptive, see their webpage for details.

DVBH related files on N96 (Wed Oct 1 19:39:06 PDT 2008)

There are a few DVB-H related files in the N96 ROM (z:\) and flash (c:\):

z:\dvbhsetup.dat
z:\DVBH.img
z:\System\data\dvbh\dvbh.ini   <--- readable
c:\System\data\esg
c:\System\data\ipdc
c:\logs\ipdc                             <--- readable 

Some error files from N96 (Wed Oct 1 17:26:50 PDT 2008)

The log indicates that the esg receiver fails to receive some file. Unfortunately, we cannot be sure which stage did it fail. Little can be done beyond this point without Nokia's supports, e.g., putting the logger into development/verbose mode.

Therefore, I decided to put N96 aside until we get some supports from Nokia.

2008/10/01	2:15:01	
2008/10/01	2:15:01	
2008/10/01	2:15:01	------- Subsystem DVBH Manager error -------
2008/10/01	2:15:01	Emitting module: c\cdvbhmgrinactivityobserver.cpp line: 231
2008/10/01	2:15:01	Error code:      -2497
2008/10/01	2:15:01	Description:     Cutting_power_in_NoPlatform_state_due_to_inactivity
2008/10/01	2:15:01	Additional info: (none)
2008/10/01	2:20:30	
2008/10/01	2:20:30	
2008/10/01	2:20:30	------- Subsystem ESG Manager error -------
2008/10/01	2:20:30	Emitting module: \common\src\cesgfilereceiver.cpp line: 548
2008/10/01	2:20:30	Error code:      0
2008/10/01	2:20:30	Description:     TimerTimeout NoBlocksReceived
2008/10/01	2:20:30	Additional info: (none)
2008/10/01	2:25:30	

Comments from Mr. Hanel, who is an engineer at Decontis (the company who makes dvbSAM)

Nokia is a special case. It is correct that Nokia moves from its own proprietary OAI ESG toward OMA BCAST ESG, but the transport streams created with Nokia ESG server I saw so far in the near past were not fully compatible yet with OMA BCAST. So handhelds from vendors != Nokia have problems with broadcasts where Nokia ESG is inside. And the other way is often also blocked - Nokia handhelds mostly could not receive ESGs provided by ESG servers from other vendors. Mostly only the combination Nokia ESG server + Nokia handheld worked properly. Of course, this behavior may change with every new software version Nokia releases for its handhelds. Since nobody except Nokia itself knows what's going on within their handhelds, finding the problem could be difficult. Could be that your ESG is fully compliant to the standard and nevertheless the Nokia handheld cannot handle it. I'd recommend to update the N96 firmware to the latest version and to try at least one more DVB-H handheld, e.g. a recent one from LG or Samsung.