Private: implementing esg
Debug tools
We have the following software analyze tool:
1) Divicatch + USB Analyzer
2) DVBSam + WinTV DVB-T USB
We have the following hardware:
1) Nokia N92
2) Nokia N96
We have the following 2 "working" transport stream:
1) N92.ts (only works on N92, this stream is created in Aug 7, 2006, comes from Nokia)
2) N96.ts (only works on N96, this stream is created in Oct 16, 2007, comes from Nokia)
These streams are under \\students\yliu\DVBH_debug\TS
Analyze of N92 DVB-H Transport Stream
For N92.ts, its transmission parameters are as follows:
Frequency1: 562 MHz Modulation QPSK DVB parameters: 8MHz, QPSK, 2/3, 1/8, 8k Code Rate 2/3 Guard Interval 1/8 Transmission-mode 8k
It includes two ESGs, one is Nokia OAI, the other is early verion of CBMS
[Trial with cell phone device]
N92 cell phone can load Nokia OAI ESG very well and play the corresponding program. and can not load CBMS ESG. Since in this transport stream, it has two int tables, and each one correspond to one type of ESG. So it is easy for me to identify which one works.
N96 failed to load any of the above ESGs.
[Trial with software analyzer]
1) DiviCatch can not detect Nokia OAI ESG, can find CBMS ESG running on port 3937 but failed on bootstrap steps.
2) DVBSam can see Nokia OAI ESG Bootstrap running on FF05:0000:0000:0000:0000:0000:0000:012D with port 9214 , can pass bootstrap steps but failed on ESG receive; can see CBMS ESG running on 224.0.23.14 with port 3937, can pass bootstrap steps but failed on ESG receive
[Conclusion]
It seems like early version of ESG are not supported by current analyzer and latest cell phones.
Analyze of N96 DVB-H Transport Stream
For N96.ts, its transmission parameters are as follows:
Frequency 690 MHz Modulation QPSK Code Rate 2/3 Guard Interval 1/8 Transmission-mode 8k
Inside this TS, there are three channels:
- Channel1 is used for ESG delivery
- Channel2 contains three programs, all free to air
- Channel3 contains two encrypted programs
Our cell phone and analyzer can see the free programs from Channel2.
Channel 2 contains 3 video programs:
- Program1 use 102143.sdp
- Program2 use 102317.sdp
- Program3 use 102491.sdp
It includes three types of ESGs, 1st is OMA BCAST, 2nd is DVB IPDC, and the 3rd is Prestandard OMA BCAST and they all use port 9214.
N92 cell phone can NOT load any of the above ESGs
N96 can load ESG and play the video.
But since this TS file only has one int table, so three types of ESG are mixed up together. So the goal is to find which is the actual working ESG on N96.
- Try to load IPDC ESG by divicatch : this can pass bootstrap step and then will crash the Divicatch software during ESG reception.
- Try to load OMA ESG/Prestandard OMA BCAST by divicatch : this can pass bootstrap step and then will crash the Divicatch software during ESG reception.
[Conclusion]
Our USB analyzer software can not even read the ESG that works on Nokia N96 cell phone!
- DVBSam failed to receive Prestandard OMA BCAST
- DVBSam can receive IPDC ESG and OMA BCAST ESG successfully.
[Conclusion]
Either IPDC ESG or OMA BCAST ESG in N96.ts is valid for Nokia N96 cell phone.
I have put ESG Bootstrap data, IPDC ESG, OMA BCAST ESG for this TS in \\students\yliu\DVBH_debug\ESG
Compare IPDC ESG form N96.ts with our current implementation
One obvious error of our current IPDC ESG implementation is in urn:dvb:ipdc:cid:2
The length of ESG data repository is wrong, should be 00 0B 53 instead of the original 00 07 5E , such that the whole SDP can be transferred.
The IPDC ESG used insider N96.ts:
- Originally sent in gzip format
- After unzip, self contains 17 outline SDP files, 5 image files (one for each video program) and 7 XML files
19_Acquisition.xml 19_Content.xml 19_PurchaseChannel.xml 19_Purchase.xml 19_Schedule.xml 19_ServiceBundle.xml 19_Service.xml
Our current IPDC ESG implementation:
- Not sent in gzip format, no need to unzip
- Self-contain 1 inline SDP file, 4 XML files
The main difference is how we deal with SDP files, ours are created by VLC which may not works fine on N96 cell phone.
Possible failure reason of our current implementation:
- Might be inline SDP content has problems?
Actually we have not reached to parsing SDP step. So I use the word might be. As if only SDP fail, we should be able to still detect and select the ESG, but when we try to play the corresponding channel, the ESG will tell you no stream received (due to bad SDP)
- Maybe due to get too many ESG bootstrap
Using DVBSam, when I check ESG status, our testbed will refresh ESG very quickly (less than 10 seconds) That is to do a ESG Bootstrap again and again.
The bad thing is during a Boorstrap, it will delete all the existing ESG. So maybe the cell phone is trying to receive the actual ESG, but out testbed sends a ESG bootstrap message and delete all the ESG the cell phone is trying to receive. (which equals to a refresh in DVBSam ESG monitor)
And in real DVB-H TS, we can easily see there is no such kind of refresh for ESG. After bootstrap ,we get the actual ESG and they are always sitting there!
Obviously, the real world DVB-H TS is using an ESG server to populate the ESGs , and we are using flute tool, I do not know how to get rid of more than necessary ESG Bootstrap based on our current implementation.
And do remember to use DVBSam which is very powerful. Divicatch in most case can not help much on ESG part!
Compare OMA BCAST ESG used in N96.ts with our current implementation
The OMA BCAST ESG in N96.ts is much more complex than our current. It has 9 containers, self contain 93 SDP files , 29 xml files and 19 DU files.
Ours only has 1 DU file, 1 SDP, 4 xml files.
I did not do much on this direction due to time limit.
C:\dvbh.ini
ScheduleItemRemovalInterval=7 AutoScanNwTimeRetryDelay=5 ESGRootPort=9214 IpdcNumberOfLogFiles=3 IpdcLogFile=ipdc.txt IpdcLogDir=ipdc IpdcLogLevel=1 IpdcLogLevelRDebug=9 IpdcLoggingEnabled=1 NifFwdToStackInterval=500 NifReadInterval=500 SocketRecvBufSize=800000 TerminalMaximumBandwidth=500000 RequiredDiskFreeSpacePercentage=5 AllowUsageTracking=1 SignalScanFreqLow=472000000 SignalScanFreqHigh=702000000 SignalScanBW=8000000 SignalScanType=1 AutoScanEnabled=1 AutoScanStartDelayAfterBoot=5 AutoScanInterval=10800 AutoScanPreferredPlatformId=-1 AutoScanPlatformSetRetryDelay=120 PlatformSetUniversalTimeout=60 ScanUniversalTimeout=600 NetworkTimeUpdateUniversalTimeout=35 BootUniversalTimeout=20 InactivityPowerOffTime=300 ER4BootImageFile=z:\spi_boot.img ER4SWImageFile=z:\enginer4.img ER4AntennaSetup=z:\ER4AntennaSetup.dat ER4SetPlatformRetryTimeout=25000 ER4SetPlatformRetryCount=10 ESGMgrUpdTime=120 ESGMgrUpdTimeAfterErr=10 ESGMgrUpdInterval=180 ESGMgrUpdRestartCount=3 ESGMgrRxInitTime=30 ESGMgrRxPeriodChkTime=20 ESGMgrUpdateEnabled=1 ESGMgrFCastRestartCount=3 ESGMgrUpkeepCleaningDelay=20160 ESGMgrCarouselDeltaStartTimeInHours=48 UsageTrackerRetryInterval=1 UsageTrackerRetryCounts=5 ECRenewalTime=1800000000 ECRenewalOffset=1 ECRemovalTimeInHours=48 AutoScanPlatformSetRetries=1 NwTimeUpdateRetries=2 PowerOffDelayAfterNwDiscovery=3 ReplayBuffer=30 UsageTrackerDbUpperLimit=1000000 DRMClockUpdateInterval=30
Error message found on N96
Based on N92 dvbh.ini config file, I guess on N96, DVBH.img is either ER4BootImageFile=z:\spi_boot.img or ER4SWImageFile=z:\enginer4.img on N92. And it sounds more like the complete image, not just the boot image.
And even though N96's dvbh.ini file is short, it seems like some parameters still follows the N92's example, for example the log files on N96 matched with N92's config file
IpdcNumberOfLogFiles=3 IpdcLogFile=ipdc.txt IpdcLogDir=ipdc IpdcLogLevel=1 IpdcLogLevelRDebug=9 IpdcLoggingEnabled=1
In DVBH.img we find the following readable error message:
PSI_Processor PSI_patpmt PSI_nitint
ThreadX API error: The event group is already created. Threadx API error: Invalid event group pointer. The pointer is NULL. System timer thread obj\dvbh\dtv1000_dbg
As the above error message happens after the PSI/SI info section, It is the next step--ESG Bootstrap on Cell phone that cause the above error lines. The cell phone tried to receive something and the application crashed. So on N96 the Subsystem ESG Manage did not get anything and leads to timeout.
TS files for various ESG versions available on-line (Mon Oct 13 09:30:11 PDT 2008)
There is a recent DVB-H trial that adopts multiple ESG versions. A final report can be found at here. In this trial, two types of Nokia ESG platforms are used: OAI and OMA BCAST. They also made some ts files available on an ftp site (see pp.18 in the above report).
To reverse engineer the requirements of Nokia's OMA BCAST ESG implementation, we can download and analyze the ts files of Nokia BCAST platform. We can use the channel parameters listed in the report to identify the right ts stream.
To debug buffer overflow problems (Sun Oct 5 14:57:03 PDT 2008)
There have been quite a few nasty buffer overflow bugs in dataaggregator and timeslicer, which cannot be easily found by reviewing the code. To trace the problem, we may use the memory checking feature of the valgrind tool. Valgrind can be installed through apt-get.
To use it, compile your code with CC flags: -g and -O0. Then run the timeslicer (for example) as follows:
valgrind --leak-check=yes ./timeslicer -n 1024 -g 8 -b 8 -d 400 -c 3 -o qpsk -p esgbt:/tmp/fatcapsesgbt:0x555 -p esg:/tmp/fatcapsesg:0x666 -p prog1:/tmp/fatcapsinput1:0x771 -p prog2:/tmp/fatcapsinput2:0x772 -p prog3:/tmp/fatcapsinput3:0x773 -p prog4:/tmp/fatcapsinput4:0x774 -p prog5:/tmp/fatcapsinput5:0x775 -p prog6:/tmp/fatcapsinput6:0x776 -p prog7:/tmp/fatcapsinput7:0x777 -p prog8:/tmp/fatcapsinput8:0x778 -f /tmp/fatcapsfifo
The output of valgrind is descriptive, see their webpage for details.
There are a few DVB-H related files in the N96 ROM (z:\) and flash (c:\):
z:\dvbhsetup.dat z:\DVBH.img z:\System\data\dvbh\dvbh.ini <--- readable c:\System\data\esg c:\System\data\ipdc c:\logs\ipdc <--- readable
Some error files from N96 (Wed Oct 1 17:26:50 PDT 2008)
The log indicates that the esg receiver fails to receive some file. Unfortunately, we cannot be sure which stage did it fail. Little can be done beyond this point without Nokia's supports, e.g., putting the logger into development/verbose mode.
Therefore, I decided to put N96 aside until we get some supports from Nokia.
2008/10/01 2:15:01 2008/10/01 2:15:01 2008/10/01 2:15:01 ------- Subsystem DVBH Manager error ------- 2008/10/01 2:15:01 Emitting module: c\cdvbhmgrinactivityobserver.cpp line: 231 2008/10/01 2:15:01 Error code: -2497 2008/10/01 2:15:01 Description: Cutting_power_in_NoPlatform_state_due_to_inactivity 2008/10/01 2:15:01 Additional info: (none) 2008/10/01 2:20:30 2008/10/01 2:20:30 2008/10/01 2:20:30 ------- Subsystem ESG Manager error ------- 2008/10/01 2:20:30 Emitting module: \common\src\cesgfilereceiver.cpp line: 548 2008/10/01 2:20:30 Error code: 0 2008/10/01 2:20:30 Description: TimerTimeout NoBlocksReceived 2008/10/01 2:20:30 Additional info: (none) 2008/10/01 2:25:30
Comments from Mr. Hanel, who is an engineer at Decontis (the company who makes dvbSAM)
Nokia is a special case. It is correct that Nokia moves from its own proprietary OAI ESG toward OMA BCAST ESG, but the transport streams created with Nokia ESG server I saw so far in the near past were not fully compatible yet with OMA BCAST. So handhelds from vendors != Nokia have problems with broadcasts where Nokia ESG is inside. And the other way is often also blocked - Nokia handhelds mostly could not receive ESGs provided by ESG servers from other vendors. Mostly only the combination Nokia ESG server + Nokia handheld worked properly. Of course, this behavior may change with every new software version Nokia releases for its handhelds. Since nobody except Nokia itself knows what's going on within their handhelds, finding the problem could be difficult. Could be that your ESG is fully compliant to the standard and nevertheless the Nokia handheld cannot handle it. I'd recommend to update the N96 firmware to the latest version and to try at least one more DVB-H handheld, e.g. a recent one from LG or Samsung.
