$1,999

ETRM Data Engineering & Analytics

Course
283 Lessons

The only program that teaches modern data engineering on a complete, real ETRM trading platform - from canonical trade model to a signed-off P&L, built end to end by you. Disclaimer: Educational use only, Excludes vendor screenshots, licensed software and live demonstrations. Support, mentoring and consulting not included. Digital access, usage or downloads make the purchase non-refundable

Buy now Add to cart

Brochure and Program Preview

Brochure - ETRM Data Engineering & Analytics_DurgaAnalytics.pdf

Preview

Sample_ch01_why_data_modeling_DurgaAnalytics.pdf

Preview

ETRM-Demo App.png

Preview

ETRM_Demo_App_video.mp4

Preview

ETRM-Demo App (Module Labs & Full Source Code)

Guide, Labs & Full Source Code.zip

ETRM-Demo-Complete_DurgaAnalytics.pdf

Contents-ETRM-Demo App.png

Preview

01 Data Modeling for ETRM Systems

1. Why data modeling is the foundation of every ETRM system.m4a

Preview

2. Entities, attributes, and relationships in trading data.m4a

3. Primary keys, surrogate keys, and natural keys for trades.m4a

4. Normalization to third normal form, and when to stop.m4a

5. The canonical trade entity and its core attributes.m4a

6. Instruments, products, and contract templates.m4a

7. Counterparties, legal entities, and hierarchies.m4a

8. Books, portfolios, strategies, and desks.m4a

9. Locations, delivery points, and geography.m4a

10. Currencies, units of measure, and conversions.m4a

11. Temporal modeling_ business date vs system date.m4a

12. Slowly changing dimensions for reference data.m4a

13. Bitemporal modeling_ valid time and transaction time.m4a

14. The dimensional model_ facts and dimensions for ETRM.m4a

15. Star schema vs snowflake for trading marts.m4a

16. Grain, additivity, and semi-additive P&L facts.m4a

17. Data types, precision, and the cost of rounding.m4a

18. Constraints, referential integrity, and data contracts.m4a

19. Modeling for change_ extensibility without migrations.m4a

20. From logical model to physical schema and DDL.m4a

Module_01_data_modeling_for_etrm_systems_DurgaAnalytics.pdf

Module 01 - Data Modeling for ETRM Systems.mp4

Preview

Module 01 - Data Modeling for ETRM Systems.chapters.txt

02 Reference & Master Data Management

21. Reference, master, and transactional data distinguished.m4a

22. Why bad static data is the root of most ETRM incidents.m4a

23. The golden record and single source of truth.m4a

24. Instrument master_ identifiers and cross-references.m4a

25. Counterparty master and the legal-entity hierarchy.m4a

26. Product and commodity taxonomies.m4a

27. Holiday calendars and business-day conventions.m4a

28. Location, hub, and delivery-point reference data.m4a

29. Curve definitions and market-data reference.m4a

30. Survivorship rules and match-merge logic.m4a

31. Data stewardship, ownership, and governance.m4a

32. Versioning static data and effective dating.m4a

33. Change management and four-eyes approval.m4a

34. Data-quality rules, scorecards, and thresholds.m4a

35. Validation gates before data reaches trading.m4a

36. Distributing reference data_ APIs vs replication.m4a

37. Caching, consistency, and staleness trade-offs.m4a

38. Auditability and lineage for static data.m4a

39. Integrating external vendors and feeds.m4a

40. Designing a reference-data service end to end.m4a

Module_02_reference_and_master_data_management_DurgaAnalytics.pdf

Module 02 - Reference & Master Data Management.mp4

Module 02 - Reference & Master Data Management.chapters.txt

03 Trade Lifecycle & Instrument Modeling

41. The trade lifecycle as a sequence of data events.m4a

42. Deal capture_ what is recorded and when.m4a

43. Trade states and the lifecycle state machine.m4a

44. Instrument modeling_ forwards, futures, and swaps.m4a

45. Options, structures, and optionality in data.m4a

46. Physical vs financial trade representation.m4a

47. Legs, schedules, and cash-flow generation.m4a

48. Fixings, resets, and floating-price mechanics.m4a

49. Amendments and the audit of what changed.m4a

50. Novation and assignment as data operations.m4a

51. Termination, unwind, and early settlement.m4a

52. Rollovers and their lifecycle impact.m4a

53. Give-ups, allocations, and block trades.m4a

54. Exercise, expiry, and delivery events.m4a

55. Corporate actions and contract adjustments.m4a

56. Lifecycle events and downstream recalculation.m4a

57. Idempotency and replaying lifecycle events.m4a

58. Modeling exotic and structured trades.m4a

59. Lifecycle integrity checks and reconciliation.m4a

60. A queryable trade-history data model.m4a

Module_03_trade_lifecycle_and_instrument_modeling_DurgaAnalytics.pdf

Module 03 - Trade Lifecycle & Instrument Modeling.mp4

Module 03 - Trade Lifecycle & Instrument Modeling.chapters.txt

04 FpML & Trade Representation

61. Why a standard trade representation exists.m4a

62. FpML in the trade-messaging landscape.m4a

63. XML, namespaces, and schema fundamentals.m4a

64. The FpML document structure and root elements.m4a

65. Parties, trade headers, and identifiers.m4a

66. Product architecture in FpML.m4a

67. Representing swaps and swap streams.m4a

68. Representing options and optionality.m4a

69. Commodity products in FpML.m4a

70. Schedules, calculation periods, and adjustments.m4a

71. Business centers and day-count conventions.m4a

72. Validating FpML against the XSD.m4a

73. XPath and querying trade documents.m4a

74. Mapping FpML to your canonical model.m4a

75. Round-tripping_ canonical back to FpML.m4a

76. Versioning and schema evolution.m4a

77. Confirmation, matching, and messaging flows.m4a

78. Common FpML pitfalls and malformed messages.m4a

79. Performance_ streaming large FpML batches.m4a

80. Building an FpML ingestion pipeline.m4a

Module_04_fpml_and_trade_representation_DurgaAnalytics.pdf

Module 04 - FpML & Trade Representation.mp4

Module 04 - FpML & Trade Representation.chapters.txt

05 Market Data, Curves & Time-Series

81. Market data as the fuel for valuation and risk.m4a

82. Quotes, fixings, and observation data.m4a

83. Time-series data models and storage.m4a

84. Bitemporal market data_ as-of and observation date.m4a

85. Forward curves_ concepts and construction.m4a

86. Bootstrapping a curve from instruments.m4a

87. Interpolation methods and their trade-offs.m4a

88. Seasonality in energy and commodity curves.m4a

89. Volatility surfaces and their representation.m4a

90. Correlation matrices and their data model.m4a

91. Interest-rate and discount curves.m4a

92. FX curves and cross-currency mechanics.m4a

93. Curve versioning and point-in-time retrieval.m4a

94. Handling gaps, spikes, and bad ticks.m4a

95. Market-data validation and quality control.m4a

96. Snapshotting curves for end-of-day.m4a

97. Intraday vs end-of-day market data.m4a

98. Storage formats_ columnar and compression.m4a

99. Serving curves to valuation at scale.m4a

100. A production curve pipeline end to end.m4a

Module_05_market_data_curves_and_time_series_DurgaAnalytics.pdf

Module 05 - Market Data, Curves & Time-Series.mp4

Module 05 - Market Data, Curves & Time-Series.chapters.txt

06 Position & P&L Data Models

101. From trades to positions_ aggregation logic.m4a

102. Position keeping and the position data model.m4a

103. Netting, offsetting, and gross vs net.m4a

104. Exposure by book, desk, and counterparty.m4a

105. Mark-to-market and revaluation mechanics.m4a

106. The P&L data model and its grain.m4a

107. Realized vs unrealized P&L.m4a

108. Daily P&L and the change explanation.m4a

109. P&L attribution_ price, volume, and time.m4a

110. Greeks and sensitivity data.m4a

111. Delta, gamma, vega storage and aggregation.m4a

112. New-trade and amendment P&L effects.m4a

113. Carry, roll, and theta decomposition.m4a

114. FX effects and base-currency conversion.m4a

115. Reconciliation to the trade blotter.m4a

116. Break analysis and P&L integrity checks.m4a

117. Intraday P&L and estimate vs actual.m4a

118. Hypothetical and clean P&L for risk.m4a

119. Aggregation cubes and OLAP for P&L.m4a

120. A reconciling P&L pipeline end to end.m4a

Module_06_position_and_pandl_data_models_DurgaAnalytics.pdf

Module 06 - Position & P&L Data Models.mp4

Module 06 - Position & P&L Data Models.chapters.txt

07 Counterparty, Credit & Limit Modeling

121. Credit risk in trading, from the data up.m4a

122. The counterparty exposure data model.m4a

123. Current exposure and potential future exposure.m4a

124. Collateral, margin, and netting agreements.m4a

125. The legal-entity and credit hierarchy.m4a

126. Limit types_ notional, tenor, and PFE.m4a

127. The limit data model and hierarchy.m4a

128. Limit utilization and headroom computation.m4a

129. Pre-deal vs post-deal limit checks.m4a

130. Real-time limit checking architecture.m4a

131. Breach detection and escalation data.m4a

132. Wrong-way risk and concentration.m4a

133. Settlement risk and Herstatt exposure.m4a

134. Credit valuation adjustment data needs.m4a

135. Aggregating exposure across products.m4a

136. Graph models for entity relationships.m4a

137. Caching exposures for fast checks.m4a

138. Auditing limit changes and overrides.m4a

139. Stress testing exposure and limits.m4a

140. A limit-management service end to end.m4a

Module_07_counterparty_credit_and_limit_modeling_DurgaAnalytics.pdf

Module 07 - Counterparty, Credit & Limit Modeling.mp4

Module 07 - Counterparty, Credit & Limit Modeling.chapters.txt

08 Data Lake, Warehouse & Lakehouse

141. Why platform architecture decides analytics success.m4a

142. Data lake fundamentals and object storage.m4a

143. Data warehouse fundamentals and MPP.m4a

144. The lakehouse pattern and open table formats.m4a

145. Medallion architecture_ bronze, silver, gold.m4a

146. Batch vs streaming ingestion patterns.m4a

147. Schema-on-read vs schema-on-write.m4a

148. Partitioning, clustering, and file layout.m4a

149. Columnar formats_ Parquet and ORC.m4a

150. Open table formats_ Delta, Iceberg, Hudi.m4a

151. ACID transactions on the lake.m4a

152. Time travel and versioned data.m4a

153. The conformed layer and shared dimensions.m4a

154. Gold marts for position, P&L, and risk.m4a

155. Compute engines and separation of storage.m4a

156. Data governance and catalog integration.m4a

157. Cost, performance, and FinOps trade-offs.m4a

158. Security, access control, and row-level policy.m4a

159. Migration patterns from legacy warehouses.m4a

160. A reference ETRM lakehouse architecture.m4a

Module_08_data_lake_warehouse_and_lakehouse_DurgaAnalytics.pdf

Module 08 - Data Lake, Warehouse & Lakehouse.mp4

Module 08 - Data Lake, Warehouse & Lakehouse.chapters.txt

09 Event Sourcing & Audit Trail Design

Module_09_event_sourcing_and_audit_trail_design_DurgaAnalytics.pdf

Module 09 - Event Sourcing & Audit Trail Design.mp4

Module 09 - Event Sourcing & Audit Trail Design.chapters.txt

175. Change data capture from operational stores.m4a

168. Rebuilding state by replaying events.m4a

170. Idempotency and exactly-once semantics.m4a

169. Point-in-time state and temporal queries.m4a

165. Modeling trade events as immutable facts.m4a

179. Testing and validating an event-sourced system.m4a

161. Why trading systems need a perfect audit trail.m4a

162. State-oriented vs event-oriented persistence.m4a

166. Projections and building read models.m4a

173. Snapshots to bound replay cost.m4a

172. Ordering, partitioning, and consistency.m4a

176. Reconciling event streams and state.m4a

177. Compensating events and corrections.m4a

167. CQRS_ separating writes from reads.m4a

163. Events, commands, and the append-only log.m4a

178. Regulatory reconstruction and lineage.m4a

174. The audit trail as a first-class product.m4a

164. The event store and its data model.m4a

180. An event-sourced ETRM core end to end.m4a

171. Event schemas and evolution with Avro.m4a

10 Streaming & Real-Time Risk

Module_10_streaming_and_real_time_risk_DurgaAnalytics.pdf

Module 10 - Streaming & Real-Time Risk.mp4

Module 10 - Streaming & Real-Time Risk.chapters.txt

186. Windowing_ tumbling, sliding, and session.m4a

181. Why real-time changes what a desk can see.m4a

189. Joining trade and market-data streams.m4a

196. Fault tolerance and checkpointing.m4a

184. Stream processing vs batch processing.m4a

187. Event time vs processing time.m4a

200. A real-time risk pipeline end to end.m4a

199. Reconciling streaming with batch truth.m4a

197. Serving real-time data to dashboards.m4a

194. Backpressure and flow control.m4a

188. Watermarks and handling late data.m4a

198. Latency budgets and performance tuning.m4a

190. Maintaining live positions in a stream.m4a

185. Stateful stream processing and local state.m4a

182. Streaming fundamentals and the log abstraction.m4a

183. Producers, consumers, topics, and partitions.m4a

193. Exactly-once processing guarantees.m4a

192. Streaming limit checks and breach alerts.m4a

191. Real-time revaluation on market moves.m4a

195. Scaling and rebalancing consumers.m4a

11 Real-Time Dashboards & Analytics

Module_11_real_time_dashboards_and_analytics_DurgaAnalytics.pdf

Module 11 - Real-Time Dashboards & Analytics.mp4

Module 11 - Real-Time Dashboards & Analytics.chapters.txt

204. Aggregation for interactive drill-down.m4a

211. Latency, caching, and perceived speed.m4a

213. Time-travel and as-of views in the UI.m4a

203. Semantic models and metric definitions.m4a

205. OLAP cubes and pre-aggregation.m4a

219. Reconciling the screen to the books.m4a

208. Position and exposure view design.m4a

215. Handling scale_ thousands of positions.m4a

217. Embedding analytics and self-service.m4a

210. Alerting, thresholds, and notifications.m4a

214. Visualization choices for risk data.m4a

202. The serving layer between data and screen.m4a

207. WebSockets and live data to the browser.m4a

212. Consistency between dashboard and source.m4a

220. A production trading dashboard end to end.m4a

206. Real-time push vs polling architectures.m4a

201. What makes a trading dashboard trustworthy.m4a

218. Observability of the dashboard pipeline.m4a

209. P&L views and attribution drill-down.m4a

216. Access control and data entitlements.m4a

12 Straight-Through Processing (Capstone)

Module_12_straight_through_processing_capstone_DurgaAnalytics.pdf

Module 12 - Straight-Through Processing (Capstone).mp4

Module 12 - Straight-Through Processing (Capstone).chapters.txt

233. Reconciliation at every handoff.m4a

229. The event-sourced backbone of STP.m4a

239. Governance, controls, and audit across STP.m4a

222. The end-to-end flow from capture to settlement.m4a

235. Orchestration and workflow control.m4a

237. Failure handling and dead-letter design.m4a

224. Integrating trade capture and the lifecycle model.m4a

223. Designing the STP pipeline as connected stages.m4a

231. Exception management and the repair queue.m4a

226. Feeding market data and curves into valuation.m4a

227. Computing position and P&L in the flow.m4a

225. Wiring in reference and master data validation.m4a

236. Observability, tracing, and SLAs.m4a

238. Settlement, confirmation, and downstream feeds.m4a

221. What straight-through processing means and why it matters.m4a

232. Straight-through rates and manual-touch metrics.m4a

240. The complete STP reference implementation.m4a

234. Idempotency and safe reprocessing.m4a

230. Streaming versus batch stages in the pipeline.m4a

228. Embedding credit and limit checks inline.m4a